Why do Agile Initiatives fail?
Trying to do agile in an organisation where the processes and the culture are not conducive to its success can be like shouting into the wind. As I always say:
Change the content and you haven't changed anything. Change the context and you've changed everything.
Most times, it is not that anyone is consciously trying to stop you. It is more cockup than conspiracy. For example, it can be hard to iterate a set of prototypes if your finance team need a multi-year programme business case (which includes all the foreseeable costs up-front). Similarly, if your security team need to approve the design but you can't tell them what all the components are going to be yet. Or if your procurement team want to RFI/RFP a supplier against a written specification, but you don't even know yet what mix of skills you might need and certainly can't specify the solution elements in full.
There is a good reason all these processes and controls exist! And normally it is about managing risk and uncertainty to avoid undesirable outcomes (like suffering security breaches, or wasting money). But trying to apply these controls to an agile initiative can either doom it before it's started or force you into giving up on agile altogether, in favour of a more traditional, sequential method (that better meets the contextual needs of your colleagues).
Suitable Alternative Controls
Speaking as an ex-auditor (sigh I know; not my best ever idea), Suitable Alternative Controls (SACs) are your best friend. As an example of this, say you have a control in your business to approve each and every line item of an expense claim before the expenses are even incurred. Well, one level of an SAC would be to approve the reclaim of those expenses after they have been incurred, but still before the reclaim is paid to the employee. Such a change would still be a "preventative" control, in that it still occurs before the company itself has incurred any costs. A further level of SAC would be to approve all expense claims (on trust) but review a sample of these after they have been paid, to make sure trust is not being abused, then act on any fraud with disciplinary action. This is called a "detective" control (as it occurs after the fact).
Where am I going with this? Well, what I am saying is that there is absolutely no point in asking to be excused your responsibility for proper control, simply because you want to work in an agile way. Rather, you need to help your colleagues to come up with suitable alternative ways in which their control needs can be met, in an agile context. I call this "creating the agile enterprise".
Kobayashi Maru
The Kobayashi Maru was a Starfleet training simulation in the Star Trek franchise, where Academy cadets were placed in a no-win scenario, to test their character. What I love about it is that James Tiberius Kirk became (famously) the only cadet to have ever beaten the test. How? Well he re-programmed the simulation to make it winnable. Ha ha ha. Brilliant! I tend to take a similar approach to Kirk and - like him - I don't believe in no-win scenarios.
So for the rest of this article, I am going to work with the Star Trek Crew, as they reprogram their organisation to create the agile Enterprise (as, yes, even bad puns are not beyond me).
Agile Leadership ◼
We start on the bridge of the enterprise. Command and helm. Where else? Kirk and the folks in the yellow t-shirts. Wouldn't it be great if all your business and technology sponsors had a common understanding of agile and a common language to describe its components? When I was at British Airways, we took all of our Top 250 Leaders out of the business (in the year 2000) to spend a day with their closest role counterpart at Cisco. They learnt what it meant to work in an agile way and to make maximum use of technology in the process. At the time, this 'sheep dipping' was truly transformational (in helping us to modernise the business).
Not every organisation sees business change the way we did at BA at that time. At the organisation, there was a strong tradition of 'people programmes' (as they called it), so we were pushing at an open door with the idea. However, it's a huge investment to 'lose' so many senior people hours to 'networking'. Perhaps a more feasible idea would be to engage an agile coach in your business, to work with Senior Leaders and mentor them in the methods. Either way, I can strongly recommend doing something. True change always starts or ends at the top.
Agile Functions ◼
Now for the most important area; Spock and his colleagues in the blue uniforms handling 'corporate resources' and the management science, processes, and controls that underpin them. Here we are talking about functions like PMO, HR, Procurement, Finance, Legal & Risk Management (to name but a few). I am going to handle each, briefly, in turn.
Programme Management Office ('PMO')
By now, you will doubtless know my views on Agile Project Management (and the fact that there is no such thing). However, it is important to work with PMO colleagues on how the Project Management Method (or 'PMM') defining your 'management deliverables' intercepts with your Agile Solution Delivery Lifecycle (or 'SDLC') that defines the specialist deliverables (on an agile technology project). Unless you are tri-modal on the project, you are likely operating pure agile on a continuous improvement, client tech or innovative NPD piece. In such a circumstance, your specialist deliverables might be limited to burndown charts, features & epics (in a backlog), release notes, and the like (rather than the business requirements documents, specifications, or designs of Waterfall methods). If your PMO are trying to work with traditional 'stage gates' they might struggle to attach these to your agile outputs.
The best recommendation I can make it to encourage them to give up on 'common' gating and instead define the stages of each project (differently) to match the (major points of its product) release cycle. So your quarter 1 and 2 (major point) releases might match, for example, stages 1 and 2 of the project. As a major release should have a 'release theme' with some major associated features or epics, that gives you naturally your gating criteria for the project stage; i.e. have the features been signed off by the business product owner or not?
In some organisations, this 'bonding' is achieved by using 'programme increments' and a method like Scaled Agile Framework (or 'SAFe'). Whilst I understand the thinking behind SAFe (i.e. how do you scale agile to situations where multiple different product teams - operating different backlogs and release cadences - need to come together in a project), I am frankly not a massive fan as it can stymie true agility and end up being Wagile. I would rather align the release cadence of the different product teams involved at the outset and have the PM & PMO act as the common coordinating element (attending each team's client panel to ensure their backlogs contain the right things at the right time and that inter-dependencies are managed).
A word on RAG Reports. Try to encourage your PMO to ditch them altogether, in favour of agile metrics like cycle & lead times, output velocity, and resource efficiency. Traditional RAG is (a) often more subjective than we would like to believe, (b) taken at a point in time, and (c) predicated on a trade-off of quality, time, and cost. In agile, time and cost are usually largely fixed (as we shall see in a moment), so the trade-off becomes somewhat moot. Plus agile metrics are real-time (rather than point-in-time) and objective by nature (if measured correctly).
HR & Talent Acquisition
I'll keep this one brief. It's useful to make sure your HR colleagues have a full understanding of modern product management job families and agile competency frameworks, so they can benchmark new and existing roles properly and help you find the right training and professional qualification frameworks for ongoing people development. I find SFIA to be a very useful primer for that conversation and I favour ICAgile's Agile Product Ownership (APO) qualification (and associated Pluralsight learning pathways) for my product teams.
Procurement & Vendor Management
For commercial colleagues, I have always liked the distinction of pre- and post- signature. Pre-signature is Procurement and post-signature is Vendor Management. It might be called something different in your organisation and might have separate reporting lines, but both these teams need tackling (and taking up a 'learning curve'). In short, there are only three types of commercial construct that work for agile in my humble opinion; (a) time & materials, (b) fixed capacity, and (c) price-per-story-point.
Don't dismiss Time & Materials, because too many people do! After all, if you have a sensible Master Services Agreement (MSA) and Statement of Work (SoW), you will be able to add or remove resources from the supplier with reasonable notice at any time, up-to-and-including full termination for convenience. So where's the risk? And there's lots of flexibility in that, which matches the (naturally emergent) needs of agile well. One of my favourite uses of this is for "annual, rolling (augmentation) SoWs" where you use a partner to provide skills you need but would not choose to hire internally (e.g. because a full FTE wouldn't be fully utilised across any given cadence).
If you must seek increased certainty ahead of time, then a fixed capacity construct works well. In this arrangement you commit to a certain number of resources (N) for a certain period of time (T), so Price (P) x N X T gives you the Contract Value (in the form of a time/resource 'box'). For this, a T of three months (i.e. one quarter) works well. If your commercial team are any good, they will be able to attach a 'Statement of Outcomes' to each quarterly SoW; which binds the partner to a set of deliverables. However, even then you must recognise that this is more a motivational technique, rather than something truly enforceable. The truth - in agile - is that the value you get for that fixed capacity is largely down to how well you manage the work. If you change your mind often and make bad calls along the way, you will get less useable output.
The most interesting - and even today fairly 'exotic' - is price-per-story-point; where the supplier receives a payment which is P X S (where S is the number of story points delivered). This 'piece-rate' can be quite effective in balancing the risks and responsibilities between the parties, whilst mainly pushing the consequences (for poor performance) onto the partner, although at the outset, this obviously requires a "discovery phase" to elucidate at least the first set of user stories and story points (i.e. in my language a tri-modal approach, so not pure agile). There is also a gain-share element, in that the supplier (if super-efficient) can make more money from you than you were expecting in any given period, if their efficiency and output velocity is higher than planned. In case you are doubting the existence of this option as a thing, I can assure you I used this exact construct at Macmillan with our partners there and it worked well. The primary challenge is determining a reasonable value - and definition - for a story point of output. So I would definitely recommend this construct, but only when you have achieved a certain maturity level in agile.
What you do need to do is persuade your commercial colleagues not to insist on fixed price arrangements. This is the world of "build me a toaster" (as I call it) where you can specify up-front you want a knob, two slots, and the precise shade of brown you want for your bread at the end. This is waterfall, not agile and will not work. If they mention 'not-to-exceed price' or 'capped T&M" as an option, say yes! It's really just another way of saying fixed capacity, which does work.
Finance
Business cases are a problem. Usually, your Finance colleagues will want some certainty (within a limited tolerance) of how much it is actually going to cost you to complete the work. But specifying (for example) what software products (and number of licenses) or hosting provider (and capacity) you need can be difficult up-front, when working agile. If possible, try to argue for the use of "quarterly draw-downs" against an agreed budget (at the start of each year). In this model, you have your entire team (and third party costs) re-approved 3-4 times a year in a 'zero-based-budgeting' fashion. This provides the suitable alternative control, as your effort could be stopped at any time if the value is not demonstrably flowing. It does require a certain ongoing commitment to 'financial engineering' however! The price of admission.
As an adjunct to regular draw-downs, I would propose the use of a Benefits Dependency Network ('BDN'). Back in the early noughties, I worked closely with Professor Chris Edwards (and his team at Cranfield Business School) on the development and application of BDNs (at Centrica). At WPP, we've evolved the method (on the WPP Open Programme) to use a Sankey (where you start with the benefits on the left and flow through to the outcomes in the middle and the costs on the right (of the network). This 'value driven' approach means that no activity is undertaken on the project that can't be directly linked to an outcome and benefit. A BDN can be evolved during the agile project, as the value proposition becomes clear and the most important requirements emerge. If you are going to commit to a BDN, then you must commit to (a) the identification of benefit owners, (b) an agreed method for benefit quantification and attribution, and (c) the tracking & evidencing of benefit achievement over time.
Legal & Risk Management
See my section on procurement above for how your legal people should engage on contracts. In general terms, the key point to get across is that risk cannot easily be transferred to a partner / system integrator under agile. This is not a good reason, however, to abandon agile altogether. And innovations like cost-per-story-point can achieve at least some rebalancing of risk back to the supplier, once you are sufficiently mature.
Risk Management is a key area. I would engage with your auditors (both internal and external) and your InfoSec teams early, to review their current frameworks and how suitable alternative controls could be added, specifically to cover agile. Often, their own processes will be tied to traditional design points in a Waterfall SDLC (like the agreement of a final design - for a security review - or the sign-off of UAT). One SAC could be having an assurance specialist attend your client panel sessions (where you scrub your backlog) so they can satisfy themselves of the quality definitions of done. I would also invest in code vulnerability scanning (e.g. SonarQube), regular penetration testing and a secure API layer (like Apigee) where needed. Also have them review your test automation rig and work together to sharpen your IT Policies around third party development. Like I said, SACs are the way to go!
Agile Engineering ◼
So now we are working with Scotty and his chums. The red-shirts who keep the warp core humming. Your engineers. Now I know what you are thinking... nothing to do there, right? They all know this stuff already! Wrong! For tech teams, I focus on three key areas: common language, common methods, and common tools.
Just because a bunch of devs have worked agile in previous lives or places, it does not mean they share a common language! A good place to start is with a glossary of key terms and a set of clear role definitions for all the business and IT people involved in the cadence. A particular area I tend to focus on is the definition of a story-point (and the associated pseudo-science of t-shirt sizing development estimates). One will often find different dev teams arguing for very different measures on story-points, but my argument has always been that it's kinda like the different currencies of the EU in the run-up to the introduction of the Euro: In ERM II, the value of a French Franc, Italian Lire, and Deutsche Mark were broadly fixed against the Euro. In the same way, whilst different teams may define story-points differently, their definitions tend to stay the same over time, so it should be possible to determine an Enterprise Story Point (or ESP) which - similar to the Euro - can be readily adopted by all teams in time.
The area of common methods can also be harder than it appears at first sight. Personally, I am not a massive fan of the standard (Scrum) states of "to do, in-progress, done". I am much more interested in the flow through the lifecycle than I am in what any individual considers her to to-do list. If your teams have lots of items that are marked as "done" but are not finished and sit like that in your backlog for ages, you will perhaps know what I mean! I favour my own custom model of 7 states and 3 statuses (with each ticket carrying both those fields as drop-downs). The statuses are (a) awaiting, (b) in-progress, and (c) blocked. The states are (1) shaping, (2) specification, (3) development, (4) testing, (5) acceptance, and (6) release. So rather than a dev saying he is 'done' with development, he moves the ticket from "in-progress/development" to "awaiting/testing". This 'pushes' items through the lifecycle, as the ticket is now a ticking-clock in the in-tray of the test team. The status of "blocked" can also be very handy for your client panel. I tend to devote an entire agenda item to resolved blocked items and it's a key measure (the team are assessed on) as to what %age are blocked.
> I will cover methods in (much) more detail in an upcoming long-read on Agile SDLC [1]
The easiest to dictate is common tools, although it can be hard to keep teams aligned over time (as there is always some new and interesting DevOps tool emerging) and Devs are like magpies; always chasing the latest shiny thing! For the core DevOps toolset, I would say it's a choice between Atlassian (Jia, Confluence, Bamboo, Bitbucket) and Microsoft (Azure DevOps & GitHub). For Site Reliability Engineering, there are a myriad of ways to go, but I like Atlassian StatusPage, PagerDuty, and Intercom for the different aspects they bring to the table.
Final Thoughts
On reflection, perhaps this post was over-ambitious, in trying to cover a huge topic at the right level of detail without overdoing the word count. Some of these areas I will return to later (as a higher level of granularity). But if there is one message I would love you to take away from this piece, it is this:
Agile Development cannot be delivered in isolation. For Agile to fly, it needs a conducive organisational context. I call this building the "agile enterprise"; which is mainly about introducing suitable alternative controls for agile work (that don't depend on traditional, waterfall-style artefacts)
I really hope this helps you in your adoption efforts. And please do check back here later for further articles! I will be elucidating (at far greater detail) the Agile SDLC and also the measures one can use to manage and monitor your progress. Live long and prosper, friends! 🖖
© David Viney 2023. Licensed for re-use with attribution under CC BY 4.0
Preferred Attribution Code Snippet
Tips on <a href="https://www.david-viney.me/post/creating-the-agile-enterprise">Creating the Agile Enterprise</a>, by <a href="https://www.david-viney.me/">David Viney</a>, licensed under <a href="http://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a>Footnotes
[1] Forthcoming articles in this series; please check back soon
Press release | Article | Wings | More | Telegraph | AngloInfo| Animap | CA | Sly | SMB | Design1 | BlogP | Prx1 | Prx2 | bloggo | blog2 | dviney | David | Another blog | Uzblog | AltDesign | OOV | Bresdem | Article | ddd