You might know by now (from my writing) that I am instinctively suspicious of fads and anything that smells like religious zealotry. So this article sets out to bust some origin myths and hopefully give an honest assessment of the merits of both waterfall and agile for a software development lifecycle (or 'SDLC'). I will also, briefly, debunk both as a project management method and diss the entire modern IT profession for good measure. All in a day's work. Lol.
Waterfall: Origins
There are many competing "origin stories" for the emergence of the Waterfall method. Most attribute its creation to Winston Royce of Lockhead, who laid out a sequential method for systems development in a paper in 1970. However, he never actually used the term Waterfall, which didn't enter common parlance until at least 1976. In fact, I have a very dim view of the 'official' history (as recorded on Wikipedia). As someone who started his career in the late 1980s, I have absolutely no doubt that the real origins of what we now call Waterfall was in the “Cleanroom” Model; developed by IBM and used by NASA on the Space Shuttle Programme (from 1978 onwards).
Most people focus on the sequential nature of Waterfall. But this totally misses the point. The defining characteristic is actually that it is a "correct systems" methodology. If you need to strap 7 people to an expensive rocket, fire them into space, then bring them back safely… over and over again… then the solution must be correct. This means a correct set of business requirements, which leads to a correct set of technical requirements, which leads to a correct design and a correct, tested build – so a sequential, largely non-iterative methodology. Iterations are unnecessary because everything was correct at every stage, right? For example (from the Shuttle Programme), "nose cap of shuttle must resist re-entry temperatures in excess of 1,260°c", which led to the development of the shuttle's famous ceramic tiles. As usual (with computer stuff) it was the UK where this method was fully baked into standards; with the Structured Systems Analysis and Design Method (or 'SSADM') from 1980 onwards.
Agile: Origins
Once again, I find the official history for Agile (now persisting on Wikipedia) to be very flawed. Most people now seem to want to take the view that there was no true Agile before the 2001 "Manifesto for Agile Software Development". But there is absolutely no doubt (in my mind at least) that the Rapid Application Development ('RAD') method of 1991 was the genuine progenitor. It was also developed by (you've guessed it) IBM, who then built upon this with the Dynamic Systems Development Method ('DSDM') of 1994. I had the great honour to work with IBM (whilst at British Airways in the late 90s) on the further development of the latter; contributing the idea of a single 'living document' for documentation throughout the lifecycle.
RAD and DSDM were IBM’s response to the emergence of PCs and Distributed Computing (and the long, slow death of the mainframe). If Waterfall was about "correct" systems, RAD was about “good enough” systems. At the time, the argument was that an average corporate IT system took three years to deliver, the average car model six years, and the average aircraft twenty years. Something had to change! Business cycles were accelerating and IT had to respond! Above all, the sort of technology had emerged to make more rapid and iterative development possible. And one could quickly get to 80% of requirements met in 20% of the time and cost, through a set of iterative prototypes.
Agile vs Waterfall - Which is better?
As they say, ask a stupid question, get a stupid answer. So the TLDR (if you don't want to read the rest of this article) is both and neither. More seriously, one has to approach this question as if you were an engineer. And as if IT were actually a profession. Difficult, I know.
Waterfall is absolutely more appropriate when being right is more important than being quick. For example, if the lives of 7 astronauts are at stake it's kinda important to be correct about those 1,260°C re-entry temperatures! Waterfall is still best when requirements can be clearly articulated and are unlikely to change, but above all for safety-critical solutions required to stand the test of time.
By contrast, Agile works better when being quick is more important than being right. For example, for a new product or new market, or where an entire industry is being disrupted by a new technology or profit model and you are in a race to out-innovate your competitors. Agile is best when requirements cannot be articulated and/or are changing fast.
In short: use Waterfall when "our people will die if you get it wrong" and use Agile when "our business will die if you take too long".
In reality, making the decision on which SDLC to use can be complicated and difficult to get right. It can also be changed during the project lifecycle (and actually most often is)!
> For a tool to help you decide, check out my Agile-o-meter
> If you'd find a little case study helpful, check out "the mayor of Kingston Town"
> For changing SDLC mid-project, try my thought piece on "tri-modal projects"
Agile Project Management
In brief, there is no such thing. Sorry to be a party-pooper, but I will never understand the tendency of the project management community to want to appropriate a Software Development Lifecycle as the basis for a Project Management Method ('PMM').
Firstly, a PMM is completely abstracted from Development Methods. When I worked on the burnishing of PRINCE 2 (at Centrica in the early 00s), everyone was clear that a good PMM could be used on any kind of project. The PMM defined the "management products / deliverables" (e.g. project plan, work breakdown structure, etc.) whilst the "specialist products / deliverables" were determined by the type of project involved. Sure, if it were a systems project, then the specialist deliverables would be defined by the SDLC in use (whether agile or waterfall in nature). But if it were, say, a property / construction project, then the specialist deliverables might be architecture drawings, BIM models, stacking diagrams, and the like. At that time, I would say about 50% of the systems projects I was involved in or managing were Waterfall in nature and 50% were Agile. And no-one saw any great issue with this.
Secondly, the great value of a PMM is in managing dependencies, sequencing, and above all critical path. In other words, what things need to be done in what order and how can we optimise time, cost, and quality across the lot? Laying the walls of a house before one has built solid foundations would be crazy. As would putting in glass before one has built the window frames. That's obvious. But what is less obvious (and you will know if you have built a house before), is that the wise builder orders the window frames (against a plan) well in advance; as the order lead times tend to be long, and she doesn't want the brickies to sit idle. If, like me, you have noticed the gradual disappearance of the Gantt Chart and CPM analysis in the workplace, you will perhaps agree that - by trying to mimic coders - PMs are actually losing the very skills (and management mathematics) that made them so valuable in the first place!
The Impact of Cloud Computing
To my mind, there is a simple explanation for this descent into madness. As always, with technology, it is the technology itself which disrupts things. Agile really took flight with the emergence of Cloud Computing. If the days of Mainframe were like building a house, and Distributed Computing (and Packaged Software) like renting a house, then Cloud Computing is kinda like squatting in a house. Rent or buy, and you still get to redecorate with some permanency, but when squatting the owner can simply turf you out at any point and throw your paint and brushes after you!
What I am trying to say is there is no point trying to gather requirements and develop solutions on top of cloud software. Why reinvent the wheel and pay twice? Once for the thousands of vendor engineer hours (that went into building - say - Workday) and twice for your own dev team? Apart from anything else, the vendor could make a change at any time (to the underlying SaaS) which renders your (usually client layer) development obsolete.
So if you work for that vendor, you will probably be using agile and cutting code. But if you work for one of their corporate customers, your job really becomes configuring (not developing) that solution to 'best fit' your business and then persuading business colleagues to modify their processes to close any residual "fit gap" to "out-of-the-box". In a sense, your job becomes more "solution led" (through 'show-and-tell') and less "requirements led". And the skillset becomes more about adoption and business change management (which is why all the SaaS vendors spend millions on adoption materials and hire thousands of people with PROSCI skills - to focus their customers on that, instead of dev!).
The Existential Identity Crisis of the IT Professional
What am I blathering on about? Well, what I am saying is that an entire generation of IT professionals (and I use that term loosely, obviously) were raised (without really knowing it) on a philosophy founded in the space shuttle programme and grounded in civil & electrical engineering: Bring me your problem and I will design a perfect solution to correctly solve it for you, then build it to optimum time, cost & quality. Whilst technology has advanced multiple generations since the 70s, our thinking remains largely trapped in the habits and methods of the past. So at this point in history, we are largely confused (as a profession) about our entire purpose and therefore our methods. What I call the Existential Identity Crisis of the modern IT Professional.
More specifically, what I am saying is that - outside of (Client Tech) product development (which is increasingly the preserve of SaaS Software Vendors), the majority of (Enterprise Tech) IT people are (a) not really running software development projects anymore, and (b) not really developing software solutions anymore. So it's not just Project Managers who are confused, it's Business Analysts and Software Developers too!
> I explore this topic in greater detail in "the bifurcation of the modern IT profession" [3]
So call it Agile if you want. Or Transformation. Or Apotheosis (my personal favourite). But be clear on what it actually is: Persuading business people to adapt to (and adopt) some very expensive cloud software that was developed by someone else. And you'd be better off taking that PROSCI course than an Agile one!
© David Viney 2023. Licensed for re-use with attribution under CC BY 4.0
Preferred Attribution Code Snippet
The <a href="https://www.david-viney.me/post/agile-vs-waterfall-which-is-best">Agile vs Waterfall (which is best?)</a>, by <a href="https://www.david-viney.me/">David Viney</a>, licensed under <a href="http://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a>Footnotes
Like the Post?
Comments