32 ms·

The Waterfall methodology was a historic accident and they knew it

196 points by janvdberg 8y ago

seanc 8y ago
I don't know about an accident, necessarily. It's got two important things going for it. 1. It lets everyone involved cover their ass. "All I'm responsible for is my function. There was a problem with my inputs, so no wonder my outputs suck. Don't blame me, my inputs are all right there in black and white, and you signed off on them." 2. It gives people the comforting feeling that they can predict the future, and then do things like plan budgets or negotiate fixed price sales contracts. Predicting the future is irresistible for people who handle money. As fictional as it might be. So those two things, I think they have a momentum of their own, and without constant vigilance and effort, for most domains, waterfall development is a sad, inevitable default.
- dreamcompiler 8y ago
  Completely agree. The recent discussion here about Jira confirms this: Any new software development methodology -- however well-intentioned -- eventually will be distorted into the waterfall model with a sexier name. The incentives for doing so are much more powerful than the incentives for sticking with a truly better development process. If we want to change this, we have to change those incentives.
  - mlthoughts2018 8y ago
    I agree with you, except I’d say that the ideal of Agile, the manifesto itself, responding to change, etc., is a significantly worse workflow process than Waterfall (which is also bad). Agile prescribes so many things that are absolutely pushover, toothless concepts in the face of basic corporate bureaucracy that it really is a case where it’s the tool’s own fault that everyone uses it wrong. I’ve seen Waterfall used successfully. Not often, but on a few defense research projects it was great. I have never once seen or heard of any Agile process anywhere actually working with any degree of success. I’ve heard plenty of people indulging some No True Scotsman nonsense about how a “real” Agile process can succeed, and then proceeding to write a diatribe about their “real Agile” workplace that is clearly so fucked with dysfunction that the person writing can’t even realize they are only shipping software in spite of Agile and not the least bit because of it. But I’ve never ever seen or heard of it actually causing success, not across any of the countless blog posts or anecdata on HN, and not across 6 different jobs at companies of various ages, sizes and capitalization levels all using Agile withvarying degrees of formality, pragmatism, Agile coaching and consulting, etc. None of it made any difference. Agile just hinders people. It is not capable of doing otherwise. Now, having said all that, I do agree that for processes that can work, processes that are not overly prescriptive, let different teams do things different ways at different times based on different goals or different personnel, let teams ignore the customer when needed, enforce long-term planning when needed, do pure research with no time boxing when needed, etc., then for those kinds of common sense, non-Agile processes, we have to change the incentives so that people are aligned to care about the objective truth and objective outcomes required to get the job done.
    - com2kid 8y ago
      > I have never once seen or heard of any Agile process anywhere actually working with any degree of success. I ran a team were we shipped (firmware!) every 3 weeks. After the first year of tracking all our numbers we were able to hit our estimates within a few hours +/- for the entire team. 3 week sprints, 2 weeks writing code, 1 week testing and bug fixing. It wasn't perfect, and learning how to do it was painful, but once we got going it worked great. We didn't adhere strongly to any one guiding principle, but being able to use sprints to align work across multiple teams, and as a planning mechanism for management was incredibly useful. "Our breakdown of that feature means it'll take 3 sprints in total, the last of those sprints the work is split across 2 devs, if one of those devs gets reassigned it'll take 4 sprints." Being forced to actually plan work out, and then being held accountable for the hours, meant that over time we got good at estimating, and knowing how far to break work down. There is of course a trade off between gains from breaking work down, and time spent doing the breakdown! Several months were rather painful when we went too far one way, and then inevitably the next few months swung too far in the other direction. But in the end? Yeah it all worked out. We figured out what length of sprint worked for us, what type of estimation system to use, and how much work to put into our planning sessions. FWIW one of the huge benefit of doing the work breakdowns is being able to see opportunities to parallelize work, and also to see natural stopping points. My team was lucky to have someone almost full-time dedicated to managing our workflow, without that the entire process wouldn't have been tenable. The tools just suck up too much time.
      - mlthoughts2018 8y ago
        > “Being forced to actually plan work out, and then being held accountable for the hours, meant that over time we got good at estimating” I’m skeptical of this, because even working in a team that had many person-decades of experience of estimating software delivery, the consensus opinion was that nobody’s estimates are worth anything and that the business would actually be worse off using estimates even as mere guidelines than just working on things, hitting blockers, and adjust expectations as you go. But regardless it sounds like your process was fairly different from Agile. The only superficial similarity was having some notion of a regular sprint cadence, but cadence based scheduling is common to many project management strategies. It sounds like you had something very pragmatic for your exact situation, which is great — that’s the whole goal, the very thing Agile paradoxically does not conceptually support.
- agumonkey 8y ago
  I experienced 1) in another context. We went from traditional coarse task per employee to atomic operation dedicated employees, to increase throughput. The system became blind. Everybody was applying it's tiny task on the stream, except there were always a tiny % of ambiguity that nobody cared enough (we all cared, but the stream pushed and no one upstream / manager wanted to stop and solve issues). We generated a never ending list of crap that got rejected by the final commit step.
  - jrochkind1 8y ago
    but you totally increased throughput, right?
    - agumonkey 8y ago
      for a while, then higher level consultants fixed numbers while we were reorganizing ..
- gthtjtkt 8y ago
  Did I read the wrong article? I don't see how your comment is related to the content of the article at all aside from the word "waterfall".
  - seanc 8y ago
    To my reading, the article suggests that the reason why we have waterfall style development is because some guy at the DoD misread Winston Royce's paper. "In 1985 the US Departement of Defense wanted a documented process for software development: DOD-STD-2167. Since governments and companies reused these requirements, it radiated across the world from there" I contend that there are more fundamental reasons for this, so even if this quirk of history had never occurred, we'd still see a lot of waterfall development.
berkmayer 8y ago
Not a watetfall expert, but don't they teach waterfall in many universities? How could they not see it was based on a bad example?
mannykannot 8y ago
It seems that hardly a week goes by without me seeing another 'agile is better than waterfall' post. We are so busy flogging this dead (and possibly mythical) straw man that we are not examining the real problems of current practices.
- gthtjtkt 8y ago
  That's not what this article is about, not even a little bit.
- adrianN 8y ago
  Waterfall is alive and well in many industries.
paulryanrogers 8y ago
Interesting, I always thought that spec-first and rigorous testing methodologies came from limitations in technology. For example without the capacity for constant updates software had to be as close to perfect and bug free as practical before distribution. I cannot imagine doing agile on a widely distributed software project as it is done today with games and their day-one patches. EDIT: rephrasing for clarity
- whatshisface 8y ago
  >without the capacity for constant updates software had to be as close to perfect and bug free as practical before distribution In some cases, there is a downside to incorrect behavior that is worse than "oops we'll have to patch it." This category includes anything that handles money, industrial equipment, scientific data, or anything that isn't a computer game or adtech.
  - prevedmedved 8y ago
    Adtech handles money. It's financial software that also needs to be (soft) real-time. It's harder than the usual money software.
    - AstralStorm 8y ago
      Adtech loses income, not money. There is a vast difference. Additionally you can have rollbacks - strategy applied by many banks when things break. Funny enough, safe reverts and rollbacks require specialized up front design, it does not happen on its own. Safe automatic updates with no state loss? Same.
      - lmm 8y ago
        > Adtech loses income, not money. There is a vast difference. Placing bids is (potentially) spending money. A bug in your adtech platform can absolutely bankrupt you in the very direct way of spending more money than you have.
deleted 8y ago
[deleted]
deleted 8y ago
[deleted]
timoth3y 8y ago
I think waterfall was adopted for software development, not by accident, but because it was a well-defined methodology with proven effectiveness in areas like construction and manufacturing. In areas where changes to the spec are expensive, like building a bridge, waterfall methodology makes a lot of sense. Now that dev tools are so good and computers so powerful, spec changes are cheap, so waterfall is inappropriate. However, there may be times where it is appropriate; maybe a controller for a pacemaker or an oil refinery. I see it as using the right process for the right job.
- guntars 8y ago
  That’s an interesting viewpoint, in my experience almost 100% of budget overruns and ensuing lack of quality comes from changing requirements or rather truly thinking through the requirements for the first time after the project is about halfway done. It would be my dream to work on a project that has everything figured out upfront, alas unlike bridges, every software project is pretty unique, otherwise why are you rebuilding something that’s already done?
  - jrochkind1 8y ago
    Yeah, I don't think software changes are actually cheap at all. The difference is, I think, more that we are for whatever reasons better at predicting our true bridge requirements than we are at predicting our true software requirements. Which the OP discusses some.
  - taeric 8y ago
    Oddly, most truly disastrous projects I have seen, have had no small amount of over engineering. There has been a bit of not understanding the requirements. But more is in not understanding current personal capabilities or needs. Think of it as training for a marathon. At the end of the day, it is knowable how many calories you will need to burn and how far you are going to run. You are still far more served by running several five and then ten kilometer races, than you are studying the marathon track you want to run. Similar goes for software. Make a few customers happy. Then more. Then more. Don't stall out thinking you need to be at Twitter scale for day one. Not even Twitter did that.
bbunix 8y ago
The original paper is good. http://www-scf.usc.edu/~csci201/lectures/Lecture11/royce1970.pdf http://www-scf.usc.edu/~csci201/lectures/Lecture11/royce1970...
Spooky23 8y ago
It’s no accident. Waterfall is driven by government procurement regulations. If the components of the waterfall look uncannily like a high level work breakdown structure of a project... that’s because it is. The historical philosophy of government procurement is not just an outcome, but a competitive process. In many circumstances each part of delivery let to bid, and bidders performing design work cannot implement. Even today, design-build (aka agile) is fairly limited in most procurement scenarios. The thing that’s hard for people to grasp is that equity in procurement often yields increased cost and timelines. If vendors don’t price risk into bids, they lose money and fail to deliver. The government compensates with more stringent requirements that make it harder to bid, which shrinks the bidding pool, etc.
- cheschire 8y ago
  Indeed. The documents referenced in the OP seem like the eventual attempt at squeezing software development into the confines of Public Law 93-400[0], which predated the DOD Standard doc from OP by a good decade, and the Federal Acquisition Regulation which was an evolution of the Armed Services Procurement Regulation originally published in 1947! In my opinion, these concepts are fundamental to the successful functioning of a monopsony like the government, who is also subject to it's own equitable competition laws. 0: https://www.gpo.gov/fdsys/pkg/STATUTE-88/pdf/STATUTE-88-Pg796.pdf https://www.gpo.gov/fdsys/pkg/STATUTE-88/pdf/STATUTE-88-Pg79...
- igouy 8y ago
  Also, when vendors don't have their bid accepted, they have been known to take the government to court. So the government compensates with a document heavy process -- not to prepare for implementation, but to prepare for a court case.
maxxxxx 8y ago
I really wish we went away from saying things like "OOP is bad, Agile is bad, Waterfall is bad" to realizing that pretty much any methodology can work well or not so well depending on the type of work, the people involved and lot of other factors. The same people who mess up waterfall will also mess up Agile and the people can't do OOP in a reasonable way will not do any better with FP. In the end it comes down to constantly adapting to problems after openly admitting these problems.
- jrochkind1 8y ago
  In my experience even if you're not "the same people who mess up" everything, doing it right is actually kinda _hard_, and advice on how to figure out how to do it right is useful. Which is what these methodologies are, different peoples/communities advice.
gumby 8y ago
Apart from the name, are we really sure Isambard Kingdom Brunel didn't build his bridges the same way? Perhaps the roman aqueduct builders did too?
nopeacehere 8y ago
Good grief.. this comments section is a disaster. Agile is far from perfect but for software development it is FAR superior to waterfall. If you think differently then you either have zero experience delivering software over the last 10 years or you work for an organization that it is clearly in denial.
- himynameisdom 8y ago
  I have to agree with this. What people need to understand is Scrum clearly states the framework itself is not intended to solve your problems. Rather, it's set up to unveil your problems so you and your team can solve them yourselves. So, if you have a ton of issues while using Scrum, that's actually kind of the point. The real issue is not the framework that unveils your issues, but rather the level at which a team is empowered to fix their own issues. Scrum is both powerful and useless depending on how empowered teams are to manage their own work. It's really that simple.
- Murk 8y ago
  I haven't really ever worked in a company where they used straight waterfall in practice; it has usually been design and documentation up front with the main aspects discovered, followed by coding and updating some or all of the documents as you go in an itertive sort of manner re-designing as you go. Typically, the documents updated were screen shots and data formats. You can't call that agile perhpas, but I don't think you can call it waterfall, either.
time-domain0 8y ago
If you're designing something that needs to be high-quality, safe, well-documented and is difficult to change later on, say an elevator, waterfall is a good model to consider. For a webapp startup, hell no.
hondish 8y ago
From the script of The Mother of all Demos (its 50th anniversary noted elsewhere in HN)[1]: "...the general approach for us, empirical but pursuing this monstrous goal, monstrously difficult to like building and trying empirically, and we’re approaching it evolutionary-wise because we feel that it’s a whole system problem, you need to get a person in that environment, working and looking at the many aspects of his working system that are involved in his effectiveness. That’s many more things than just these computerated tools. And in a large system like that, we need to do it evolutionary-wise because we can’t be analytic enough about it at any one point, to decide what best our next thing should be. We can only decide from here as well as we can analyze it, where we can invest our next resources to get the most return that will increase of the effectiveness of the system we have and this item down here is the term boot-strapping applied in a slightly loose sense. We’re applying that to our approach". [1]: http://dougengelbart.org/pubs/video/fjcc68/Englebart's-1968-Transcription.html http://dougengelbart.org/pubs/video/fjcc68/Englebart's-1968-...
gaius 8y ago
There is no such thing as “the waterfall methodology”. It was a strawman set up by the Agile camp to help peddle their wares. Everyone who was around back in the day knows this. If you want to know what methodology was actually being used pre-Agile check out https://en.wikipedia.org/wiki/Dynamic_systems_development_method https://en.wikipedia.org/wiki/Dynamic_systems_development_me... , you might be surprised at how familiar it looks...
thatoneuser 8y ago
Wow speak about a masterpiece to nothing. This was a wonderfully pontificated article about nothing.
gitgud 8y ago
Waterfall is far from perfect, but it has its place in many industries, where well customers have precisely-defined specifications. In software, people usually don't know exactly what they want, let alone be able to precisely define it. This situation lends itself perfectly for rapid-prototyping to discover what works fast... which is basically an agile approach.
jl6 8y ago
I work for a corporation that has worshipped Waterfall for many decades. We are now in the middle of a years-long large scale “switch” to Agile, which is a vast cultural change. My observation is that Agile is all about software development, tuned for software development, and for the benefit of software developers and the customers of software. Obvious, right? What’s not so obvious is that a surprisingly small portion of what people think of as a software project is actually about software development. Interacting with 3rd parties, writing contracts, hiring developers, planning how your project interacts with other projects and the wider organization, responding to regulations and external enquiries, managing the demand for certainty from stakeholders, obtaining funding, maintaining financial control... These are the issues that real software projects have to deal with, and Agile is silent (if not actively hostile) on how you should deal with them. If your project is truly just about developing software, sure, be Agile about it. But it’s not through ignorance that people “fall” back to Waterfall in the face of real world constraints.
- pasta 8y ago
  Specifications are different than constraints. Agile and constraints can go hand in hand imho.
- groestl 8y ago
  The thing is, for software projects, nothing else really works. So if your core business is software, and your organization doesn't embrace this inherent fact and instead does waterfall, there is a risk that all your examples are fiction, including maintaining financial control.
  - gronne 8y ago
    Or you get away with selling bad software at exorbitant prices. Alot of big companies do fine (financially) with the waterfall methodology.
    - close04 8y ago
      > Alot of big companies do fine (financially) with the waterfall methodology. Because it works for them. Right tool for the right job maybe. I see companies trying to switch to Agile without even considering that they're applying it to the wrong projects, like big infrastructure ones. It's a buzz word and they have to go for it to look good. That costs.
AstralStorm 8y ago
Sufficiently accurate program and requirement specification is directly the program itself and requires major programming skills. Which is not possible when the person giving them is not a programmer... The fallacy of all CASE tools stems from this.
fergie 8y ago
Its problematic that Waterfall/Agile stuff regularly gets upvoted to the front page, and then the majority of commenters don't seem to read the article before posting some tangentially related anecdotes on the vague topic of "Agile". This article is making the point that a "Waterfall Method" has never actually been proposed by academia, so it is incorrect to speak about an Agile/Waterfall dichotomy. In other words: The Waterfall Methodology is not a thing and only seems to be used as a "straw man" foil. Personally I think we should also be a little more critical about who defines "Agile", how it differs from common sense, and what research has actually been done on it, but thats another matter.
- jillesvangurp 8y ago
  I would at this point argue agile is the new waterfall. It's so common that the phrase "we're doing agile" vaguely hints at a few practices that may or may not be relevant to the discussion. Mostly it means "we have standups" and "we promoted some juniors to scrum master and pm roles". It's mostly devolved to ritualistic bullshit performed dogmatically by corporate idiots. Every shitty third rate software sweat shop does agile/scrum these days. You need to look pretty hard to find companies that are not very eagerly proclaiming they do agile.It means absolutely nothing until you qualify what that actually means for you. Those claiming they are doing it "right" or by the book are the worst because that means they read some stuff without grasping the meaning of what they read. Basically, the agile manifesto signers have come out arguing against that. That was the whole point of that manifesto. They basically provided us with tools, practices, and ways of working that we're supposed to adapt to our context as needed. They don't say: do this dogmatically and follow my instructions by the letter. They actually say don't do that but if you are getting started you might as well do this and then adapt as needed. That's sort of the same thing that went wrong with waterfall. The article points out a few historical twists to historical context of that. I read the paper by Royce long time ago and found it to be quite modern and insightful. It doesn't mention waterfalls at all (as the article points out). It also points out that, waterfall as we know and love it, basically was institutionalized bureaucratic government organizations using it to sugar coat heavy processes that they came up with that got subsequently adopted as "the way to do things" by equally bureaucratic large software houses they worked with. It got a bad reputation for said corporations making obscene amounts of money with obscenely bad software delivered late and never on budget. This pattern is something that is a constant and was referred to as the software crisis in the 1960s already and lead to Margaret Hamilton along with other famous pioneers in computing coining the notion of software engineering in the 1970s. She was of course working on the lunar program at Nasa (speaking of bureaucratic organizations). The software crisis is alive and kicking. Many scrum/agile organizations continue to peddle atrociously bad software developed. That doesn't mean that nothing has changed but you should beware of anyone claiming to know the one and only way to do stuff. There's a lot of common sense that inspired agile and waterfall. I'd argue the modern take on this is to replace manual processes with automation to facilitate rapid flow of changes from inception to production. Another trend is to decouple decision points to facilitate asynchronous processes and communication. For example, you don't bundle features into releases. Instead you ship pull requests straight to CI and CD pipelines. That means that there are no central planning bottlenecks for delivering new features and they can be developed on their own schedules. This happens continually. Most big brand websites you use, update many times per day. Iterations and sprints are kind of obsolete as a way of orchestrating release schedules. That practice is now creeping into the enterprise. If you are doing scrum and being told not to do this, that's something that you need to challenge.
sgt101 8y ago
A good thing about waterfall is the documentation it generates. We switched to Agile 10 years ago, didn't manage the outsourcing all that well and now we are in the same position as then but without the documentation. More and more I wonder how we can get teams to write about what they are doing and teach the next team how everything works... sadly people suspect that such efforts will be used to outsource and offshore, but without them it's impossible to pick apart what's happened and rationally work out where to go next.
- ___alt 8y ago
  Thinking that "Working software over comprehensive documentation" means "don't write documentation" is one of the biggest misunderstandings about agile and I feel your pain.
- Murk 8y ago
  I've seen this a number of times. People (management and team alike) have said to me things along the lines of: "We don't have documentation, we are agile." Which, of course is not what the agile manifesto says, but is how some people have taken it. Other sayings are "The spec is in the tickets, there is no user manual." "We have unit tests." and so on. All this makes trouble - Especially when you consider in order to change a thing, you should aim to understand it's salient parts in the first place - but perhaps that is not agile or the scrum way according to some.
ternaryoperator 8y ago
A factor overlooked in the comments below (and in the article) is that prior to roughly 2000, projects tended to be slow-moving and could be defined by specifications, requirements, and design docs before any serious coding was undertaken. So the stair-step approach referred to as waterfall was not unreasonable. In my experience, though, what was unreasonable was the sequence "build, then debug." This meant spending huge amounts of time in debugging after the code was written and made it impossible to keep projects to a schedule because the debug/fix/rerun cycle was open-ended. This, in my view, is where Agile has rightfully distinguished itself--by having tests developed in parallel with code, thereby making debugging an "every once in a while" activity rather than a whole endless-seeming cycle.
GrumpyNl 8y ago
As soon as a work method becomes a religion, its doomed to fail. Treat Waterfall, Agile as a tool and modify it to your needs.
deleted 8y ago
[deleted]
bitL 8y ago
So we ended up with "Waterfall" being a historic accident, "Agile" turning into a cult, and with everybody knowing that great teams self-organize to top it all.
bpyne 8y ago
It's amazing how many times over my career I've watched and participated in debates on software development methodologies. HN seems to have one at least annually. The author of the post does a good job summing up the issues. I only skimmed it but have to agree with everything I read. I can add that there's a time and place for Waterfall: when requirements are well-known and static and when lots of oversight is needed. I can also add that no methodology has successfully dealt with the mega-projects we read about in software engineering case studies. Building stuff, whether it's software or anything other product, is hard. It's messy. Often the requirements are not well thought out. In the situations I normally deal with, a requester can state a high-level strategic goal for the organization, but not how the software can get us to it. That's where my team gets involved. One engineer I respect highly once said that we need to get something in front of the requester quickly so that we can find out what (s)he doesn't want. By process of elimination, the idea foments in the requester's mind and requirements become more concrete. Again, though, some organizations are very static or they deal with software whose failure can be catastrophic. In those situations, the highly bureaucratic Waterfall method can be helpful at least for slowing down the process.
- pmart123 8y ago
  I think the "Mythical Man-Month" does a great job accounting for the process and implementation of a mega-project. But as Brooks even stated, there's "No Silver Bullet."
moron4hire 8y ago
I've done a number of projects in both "waterfall"[0] and Agile. On the "waterfall" projects, we wrote lots of documentation. LOTS. Business cases for every major feature. Requirements documents with serial numbered acceptance criteria. Design documents explaining exactly how we were going to implement the requirements, with every REQ bullet needing to be accounted for explicitly. Test plans showing how each DES bullet was implemented correctly and how it proved the REQ was satisfied. And it all had to be correct and fresh. We weren't allowed to have anything disagree or slip in or drop out. If it wasn't in the REQs, it didn't go into the software. That meant if we discovered constraints that necessitated a change in design, we had to change the requirements. If you had to change requirements, it meant you had to reevaluate the business case. And that meant we were prototyping while we were writing the documentation. The documentation wasn't done until the software was about 80% complete. By that time, we were mostly just working the test plans and resolving defect tickets to get to 100%. Some of the best, longest lasting software I've ever written. All of the "Agile" projects I've been on are what I call BTSOOP development: "By The Seat Of Our Pants". There was no attention paid to documentation at all. It was hell getting PMs to write any tickets, let alone ones that were in scope. And of the tickets that were there, most of the devs ignored them, even if the PM didn't. Some people would mark tickets done that they just didn't want to do, hoping the PM would just blanket-accept tickets at the end of the Sprint. Which he most frequently did. The difference wasn't actually process. It was accountability for failing to adhere to the designated process. The Agile shops all had corporate policy on how projects would be ran, and nobody was ever taken to task for it. The "waterfall" shops didn't lack continuous integration and improvement or responsiveness to change, we just wouldn't sign off on a particular document until we knew it represented the final product, with linear dependencies on when different levels of assets could be approved (not when they could be worked on). [0] our product manager called it "the NASA way", and we fairly closely followed their documentation standards, if not their engineering ones, but we were all pretty junior at the time. IDK why I was made senior other than being the best programmer there, which at the time wasn't saying much.

196 points by janvdberg 8y ago

I don't know about an accident, necessarily. It's got two important things going for it. 1. It lets everyone involved cover their ass. "All I'm responsible for is my function. There was a problem with my inputs, so no wonder my outputs suck. Don't blame me, my inputs are all right there in black and white, and you signed off on them." 2. It gives people the comforting feeling that they can predict the future, and then do things like plan budgets or negotiate fixed price sales contracts. Predicting the future is irresistible for people who handle money. As fictional as it might be. So those two things, I think they have a momentum of their own, and without constant vigilance and effort, for most domains, waterfall development is a sad, inevitable default.

dreamcompiler 8y ago
Completely agree. The recent discussion here about Jira confirms this: Any new software development methodology -- however well-intentioned -- eventually will be distorted into the waterfall model with a sexier name. The incentives for doing so are much more powerful than the incentives for sticking with a truly better development process. If we want to change this, we have to change those incentives.
- mlthoughts2018 8y ago
  I agree with you, except I’d say that the ideal of Agile, the manifesto itself, responding to change, etc., is a significantly worse workflow process than Waterfall (which is also bad). Agile prescribes so many things that are absolutely pushover, toothless concepts in the face of basic corporate bureaucracy that it really is a case where it’s the tool’s own fault that everyone uses it wrong. I’ve seen Waterfall used successfully. Not often, but on a few defense research projects it was great. I have never once seen or heard of any Agile process anywhere actually working with any degree of success. I’ve heard plenty of people indulging some No True Scotsman nonsense about how a “real” Agile process can succeed, and then proceeding to write a diatribe about their “real Agile” workplace that is clearly so fucked with dysfunction that the person writing can’t even realize they are only shipping software in spite of Agile and not the least bit because of it. But I’ve never ever seen or heard of it actually causing success, not across any of the countless blog posts or anecdata on HN, and not across 6 different jobs at companies of various ages, sizes and capitalization levels all using Agile withvarying degrees of formality, pragmatism, Agile coaching and consulting, etc. None of it made any difference. Agile just hinders people. It is not capable of doing otherwise. Now, having said all that, I do agree that for processes that can work, processes that are not overly prescriptive, let different teams do things different ways at different times based on different goals or different personnel, let teams ignore the customer when needed, enforce long-term planning when needed, do pure research with no time boxing when needed, etc., then for those kinds of common sense, non-Agile processes, we have to change the incentives so that people are aligned to care about the objective truth and objective outcomes required to get the job done.
  - com2kid 8y ago
    > I have never once seen or heard of any Agile process anywhere actually working with any degree of success. I ran a team were we shipped (firmware!) every 3 weeks. After the first year of tracking all our numbers we were able to hit our estimates within a few hours +/- for the entire team. 3 week sprints, 2 weeks writing code, 1 week testing and bug fixing. It wasn't perfect, and learning how to do it was painful, but once we got going it worked great. We didn't adhere strongly to any one guiding principle, but being able to use sprints to align work across multiple teams, and as a planning mechanism for management was incredibly useful. "Our breakdown of that feature means it'll take 3 sprints in total, the last of those sprints the work is split across 2 devs, if one of those devs gets reassigned it'll take 4 sprints." Being forced to actually plan work out, and then being held accountable for the hours, meant that over time we got good at estimating, and knowing how far to break work down. There is of course a trade off between gains from breaking work down, and time spent doing the breakdown! Several months were rather painful when we went too far one way, and then inevitably the next few months swung too far in the other direction. But in the end? Yeah it all worked out. We figured out what length of sprint worked for us, what type of estimation system to use, and how much work to put into our planning sessions. FWIW one of the huge benefit of doing the work breakdowns is being able to see opportunities to parallelize work, and also to see natural stopping points. My team was lucky to have someone almost full-time dedicated to managing our workflow, without that the entire process wouldn't have been tenable. The tools just suck up too much time.
    - mlthoughts2018 8y ago
      > “Being forced to actually plan work out, and then being held accountable for the hours, meant that over time we got good at estimating” I’m skeptical of this, because even working in a team that had many person-decades of experience of estimating software delivery, the consensus opinion was that nobody’s estimates are worth anything and that the business would actually be worse off using estimates even as mere guidelines than just working on things, hitting blockers, and adjust expectations as you go. But regardless it sounds like your process was fairly different from Agile. The only superficial similarity was having some notion of a regular sprint cadence, but cadence based scheduling is common to many project management strategies. It sounds like you had something very pragmatic for your exact situation, which is great — that’s the whole goal, the very thing Agile paradoxically does not conceptually support.
agumonkey 8y ago
I experienced 1) in another context. We went from traditional coarse task per employee to atomic operation dedicated employees, to increase throughput. The system became blind. Everybody was applying it's tiny task on the stream, except there were always a tiny % of ambiguity that nobody cared enough (we all cared, but the stream pushed and no one upstream / manager wanted to stop and solve issues). We generated a never ending list of crap that got rejected by the final commit step.
- jrochkind1 8y ago
  but you totally increased throughput, right?
  - agumonkey 8y ago
    for a while, then higher level consultants fixed numbers while we were reorganizing ..
gthtjtkt 8y ago
Did I read the wrong article? I don't see how your comment is related to the content of the article at all aside from the word "waterfall".
- seanc 8y ago
  To my reading, the article suggests that the reason why we have waterfall style development is because some guy at the DoD misread Winston Royce's paper. "In 1985 the US Departement of Defense wanted a documented process for software development: DOD-STD-2167. Since governments and companies reused these requirements, it radiated across the world from there" I contend that there are more fundamental reasons for this, so even if this quirk of history had never occurred, we'd still see a lot of waterfall development.

berkmayer 8y ago

Not a watetfall expert, but don't they teach waterfall in many universities? How could they not see it was based on a bad example?

mannykannot 8y ago

It seems that hardly a week goes by without me seeing another 'agile is better than waterfall' post. We are so busy flogging this dead (and possibly mythical) straw man that we are not examining the real problems of current practices.

gthtjtkt 8y ago
That's not what this article is about, not even a little bit.
adrianN 8y ago
Waterfall is alive and well in many industries.

paulryanrogers 8y ago

Interesting, I always thought that spec-first and rigorous testing methodologies came from limitations in technology. For example without the capacity for constant updates software had to be as close to perfect and bug free as practical before distribution. I cannot imagine doing agile on a widely distributed software project as it is done today with games and their day-one patches. EDIT: rephrasing for clarity

whatshisface 8y ago
>without the capacity for constant updates software had to be as close to perfect and bug free as practical before distribution In some cases, there is a downside to incorrect behavior that is worse than "oops we'll have to patch it." This category includes anything that handles money, industrial equipment, scientific data, or anything that isn't a computer game or adtech.
- prevedmedved 8y ago
  Adtech handles money. It's financial software that also needs to be (soft) real-time. It's harder than the usual money software.
  - AstralStorm 8y ago
    Adtech loses income, not money. There is a vast difference. Additionally you can have rollbacks - strategy applied by many banks when things break. Funny enough, safe reverts and rollbacks require specialized up front design, it does not happen on its own. Safe automatic updates with no state loss? Same.
    - lmm 8y ago
      > Adtech loses income, not money. There is a vast difference. Placing bids is (potentially) spending money. A bug in your adtech platform can absolutely bankrupt you in the very direct way of spending more money than you have.

deleted 8y ago

[deleted]

deleted 8y ago

[deleted]

timoth3y 8y ago

I think waterfall was adopted for software development, not by accident, but because it was a well-defined methodology with proven effectiveness in areas like construction and manufacturing. In areas where changes to the spec are expensive, like building a bridge, waterfall methodology makes a lot of sense. Now that dev tools are so good and computers so powerful, spec changes are cheap, so waterfall is inappropriate. However, there may be times where it is appropriate; maybe a controller for a pacemaker or an oil refinery. I see it as using the right process for the right job.

guntars 8y ago
That’s an interesting viewpoint, in my experience almost 100% of budget overruns and ensuing lack of quality comes from changing requirements or rather truly thinking through the requirements for the first time after the project is about halfway done. It would be my dream to work on a project that has everything figured out upfront, alas unlike bridges, every software project is pretty unique, otherwise why are you rebuilding something that’s already done?
- jrochkind1 8y ago
  Yeah, I don't think software changes are actually cheap at all. The difference is, I think, more that we are for whatever reasons better at predicting our true bridge requirements than we are at predicting our true software requirements. Which the OP discusses some.
- taeric 8y ago
  Oddly, most truly disastrous projects I have seen, have had no small amount of over engineering. There has been a bit of not understanding the requirements. But more is in not understanding current personal capabilities or needs. Think of it as training for a marathon. At the end of the day, it is knowable how many calories you will need to burn and how far you are going to run. You are still far more served by running several five and then ten kilometer races, than you are studying the marathon track you want to run. Similar goes for software. Make a few customers happy. Then more. Then more. Don't stall out thinking you need to be at Twitter scale for day one. Not even Twitter did that.

bbunix 8y ago

The original paper is good. http://www-scf.usc.edu/~csci201/lectures/Lecture11/royce1970.pdf http://www-scf.usc.edu/~csci201/lectures/Lecture11/royce1970...

Spooky23 8y ago

It’s no accident. Waterfall is driven by government procurement regulations. If the components of the waterfall look uncannily like a high level work breakdown structure of a project... that’s because it is. The historical philosophy of government procurement is not just an outcome, but a competitive process. In many circumstances each part of delivery let to bid, and bidders performing design work cannot implement. Even today, design-build (aka agile) is fairly limited in most procurement scenarios. The thing that’s hard for people to grasp is that equity in procurement often yields increased cost and timelines. If vendors don’t price risk into bids, they lose money and fail to deliver. The government compensates with more stringent requirements that make it harder to bid, which shrinks the bidding pool, etc.

cheschire 8y ago
Indeed. The documents referenced in the OP seem like the eventual attempt at squeezing software development into the confines of Public Law 93-400[0], which predated the DOD Standard doc from OP by a good decade, and the Federal Acquisition Regulation which was an evolution of the Armed Services Procurement Regulation originally published in 1947! In my opinion, these concepts are fundamental to the successful functioning of a monopsony like the government, who is also subject to it's own equitable competition laws. 0: https://www.gpo.gov/fdsys/pkg/STATUTE-88/pdf/STATUTE-88-Pg796.pdf https://www.gpo.gov/fdsys/pkg/STATUTE-88/pdf/STATUTE-88-Pg79...
igouy 8y ago
Also, when vendors don't have their bid accepted, they have been known to take the government to court. So the government compensates with a document heavy process -- not to prepare for implementation, but to prepare for a court case.

maxxxxx 8y ago

I really wish we went away from saying things like "OOP is bad, Agile is bad, Waterfall is bad" to realizing that pretty much any methodology can work well or not so well depending on the type of work, the people involved and lot of other factors. The same people who mess up waterfall will also mess up Agile and the people can't do OOP in a reasonable way will not do any better with FP. In the end it comes down to constantly adapting to problems after openly admitting these problems.

jrochkind1 8y ago
In my experience even if you're not "the same people who mess up" everything, doing it right is actually kinda _hard_, and advice on how to figure out how to do it right is useful. Which is what these methodologies are, different peoples/communities advice.

gumby 8y ago

Apart from the name, are we really sure Isambard Kingdom Brunel didn't build his bridges the same way? Perhaps the roman aqueduct builders did too?

nopeacehere 8y ago

Good grief.. this comments section is a disaster. Agile is far from perfect but for software development it is FAR superior to waterfall. If you think differently then you either have zero experience delivering software over the last 10 years or you work for an organization that it is clearly in denial.

himynameisdom 8y ago
I have to agree with this. What people need to understand is Scrum clearly states the framework itself is not intended to solve your problems. Rather, it's set up to unveil your problems so you and your team can solve them yourselves. So, if you have a ton of issues while using Scrum, that's actually kind of the point. The real issue is not the framework that unveils your issues, but rather the level at which a team is empowered to fix their own issues. Scrum is both powerful and useless depending on how empowered teams are to manage their own work. It's really that simple.
Murk 8y ago
I haven't really ever worked in a company where they used straight waterfall in practice; it has usually been design and documentation up front with the main aspects discovered, followed by coding and updating some or all of the documents as you go in an itertive sort of manner re-designing as you go. Typically, the documents updated were screen shots and data formats. You can't call that agile perhpas, but I don't think you can call it waterfall, either.

time-domain0 8y ago

If you're designing something that needs to be high-quality, safe, well-documented and is difficult to change later on, say an elevator, waterfall is a good model to consider. For a webapp startup, hell no.

hondish 8y ago

From the script of The Mother of all Demos (its 50th anniversary noted elsewhere in HN)[1]: "...the general approach for us, empirical but pursuing this monstrous goal, monstrously difficult to like building and trying empirically, and we’re approaching it evolutionary-wise because we feel that it’s a whole system problem, you need to get a person in that environment, working and looking at the many aspects of his working system that are involved in his effectiveness. That’s many more things than just these computerated tools. And in a large system like that, we need to do it evolutionary-wise because we can’t be analytic enough about it at any one point, to decide what best our next thing should be. We can only decide from here as well as we can analyze it, where we can invest our next resources to get the most return that will increase of the effectiveness of the system we have and this item down here is the term boot-strapping applied in a slightly loose sense. We’re applying that to our approach". [1]: http://dougengelbart.org/pubs/video/fjcc68/Englebart's-1968-Transcription.html http://dougengelbart.org/pubs/video/fjcc68/Englebart's-1968-...

gaius 8y ago

There is no such thing as “the waterfall methodology”. It was a strawman set up by the Agile camp to help peddle their wares. Everyone who was around back in the day knows this. If you want to know what methodology was actually being used pre-Agile check out https://en.wikipedia.org/wiki/Dynamic_systems_development_method https://en.wikipedia.org/wiki/Dynamic_systems_development_me... , you might be surprised at how familiar it looks...

thatoneuser 8y ago

Wow speak about a masterpiece to nothing. This was a wonderfully pontificated article about nothing.

gitgud 8y ago

Waterfall is far from perfect, but it has its place in many industries, where well customers have precisely-defined specifications. In software, people usually don't know exactly what they want, let alone be able to precisely define it. This situation lends itself perfectly for rapid-prototyping to discover what works fast... which is basically an agile approach.

jl6 8y ago

I work for a corporation that has worshipped Waterfall for many decades. We are now in the middle of a years-long large scale “switch” to Agile, which is a vast cultural change. My observation is that Agile is all about software development, tuned for software development, and for the benefit of software developers and the customers of software. Obvious, right? What’s not so obvious is that a surprisingly small portion of what people think of as a software project is actually about software development. Interacting with 3rd parties, writing contracts, hiring developers, planning how your project interacts with other projects and the wider organization, responding to regulations and external enquiries, managing the demand for certainty from stakeholders, obtaining funding, maintaining financial control... These are the issues that real software projects have to deal with, and Agile is silent (if not actively hostile) on how you should deal with them. If your project is truly just about developing software, sure, be Agile about it. But it’s not through ignorance that people “fall” back to Waterfall in the face of real world constraints.

pasta 8y ago
Specifications are different than constraints. Agile and constraints can go hand in hand imho.
groestl 8y ago
The thing is, for software projects, nothing else really works. So if your core business is software, and your organization doesn't embrace this inherent fact and instead does waterfall, there is a risk that all your examples are fiction, including maintaining financial control.
- gronne 8y ago
  Or you get away with selling bad software at exorbitant prices. Alot of big companies do fine (financially) with the waterfall methodology.
  - close04 8y ago
    > Alot of big companies do fine (financially) with the waterfall methodology. Because it works for them. Right tool for the right job maybe. I see companies trying to switch to Agile without even considering that they're applying it to the wrong projects, like big infrastructure ones. It's a buzz word and they have to go for it to look good. That costs.

AstralStorm 8y ago

Sufficiently accurate program and requirement specification is directly the program itself and requires major programming skills. Which is not possible when the person giving them is not a programmer... The fallacy of all CASE tools stems from this.

fergie 8y ago

Its problematic that Waterfall/Agile stuff regularly gets upvoted to the front page, and then the majority of commenters don't seem to read the article before posting some tangentially related anecdotes on the vague topic of "Agile". This article is making the point that a "Waterfall Method" has never actually been proposed by academia, so it is incorrect to speak about an Agile/Waterfall dichotomy. In other words: The Waterfall Methodology is not a thing and only seems to be used as a "straw man" foil. Personally I think we should also be a little more critical about who defines "Agile", how it differs from common sense, and what research has actually been done on it, but thats another matter.

jillesvangurp 8y ago
I would at this point argue agile is the new waterfall. It's so common that the phrase "we're doing agile" vaguely hints at a few practices that may or may not be relevant to the discussion. Mostly it means "we have standups" and "we promoted some juniors to scrum master and pm roles". It's mostly devolved to ritualistic bullshit performed dogmatically by corporate idiots. Every shitty third rate software sweat shop does agile/scrum these days. You need to look pretty hard to find companies that are not very eagerly proclaiming they do agile.It means absolutely nothing until you qualify what that actually means for you. Those claiming they are doing it "right" or by the book are the worst because that means they read some stuff without grasping the meaning of what they read. Basically, the agile manifesto signers have come out arguing against that. That was the whole point of that manifesto. They basically provided us with tools, practices, and ways of working that we're supposed to adapt to our context as needed. They don't say: do this dogmatically and follow my instructions by the letter. They actually say don't do that but if you are getting started you might as well do this and then adapt as needed. That's sort of the same thing that went wrong with waterfall. The article points out a few historical twists to historical context of that. I read the paper by Royce long time ago and found it to be quite modern and insightful. It doesn't mention waterfalls at all (as the article points out). It also points out that, waterfall as we know and love it, basically was institutionalized bureaucratic government organizations using it to sugar coat heavy processes that they came up with that got subsequently adopted as "the way to do things" by equally bureaucratic large software houses they worked with. It got a bad reputation for said corporations making obscene amounts of money with obscenely bad software delivered late and never on budget. This pattern is something that is a constant and was referred to as the software crisis in the 1960s already and lead to Margaret Hamilton along with other famous pioneers in computing coining the notion of software engineering in the 1970s. She was of course working on the lunar program at Nasa (speaking of bureaucratic organizations). The software crisis is alive and kicking. Many scrum/agile organizations continue to peddle atrociously bad software developed. That doesn't mean that nothing has changed but you should beware of anyone claiming to know the one and only way to do stuff. There's a lot of common sense that inspired agile and waterfall. I'd argue the modern take on this is to replace manual processes with automation to facilitate rapid flow of changes from inception to production. Another trend is to decouple decision points to facilitate asynchronous processes and communication. For example, you don't bundle features into releases. Instead you ship pull requests straight to CI and CD pipelines. That means that there are no central planning bottlenecks for delivering new features and they can be developed on their own schedules. This happens continually. Most big brand websites you use, update many times per day. Iterations and sprints are kind of obsolete as a way of orchestrating release schedules. That practice is now creeping into the enterprise. If you are doing scrum and being told not to do this, that's something that you need to challenge.

sgt101 8y ago

A good thing about waterfall is the documentation it generates. We switched to Agile 10 years ago, didn't manage the outsourcing all that well and now we are in the same position as then but without the documentation. More and more I wonder how we can get teams to write about what they are doing and teach the next team how everything works... sadly people suspect that such efforts will be used to outsource and offshore, but without them it's impossible to pick apart what's happened and rationally work out where to go next.

___alt 8y ago
Thinking that "Working software over comprehensive documentation" means "don't write documentation" is one of the biggest misunderstandings about agile and I feel your pain.
Murk 8y ago
I've seen this a number of times. People (management and team alike) have said to me things along the lines of: "We don't have documentation, we are agile." Which, of course is not what the agile manifesto says, but is how some people have taken it. Other sayings are "The spec is in the tickets, there is no user manual." "We have unit tests." and so on. All this makes trouble - Especially when you consider in order to change a thing, you should aim to understand it's salient parts in the first place - but perhaps that is not agile or the scrum way according to some.

ternaryoperator 8y ago

A factor overlooked in the comments below (and in the article) is that prior to roughly 2000, projects tended to be slow-moving and could be defined by specifications, requirements, and design docs before any serious coding was undertaken. So the stair-step approach referred to as waterfall was not unreasonable. In my experience, though, what was unreasonable was the sequence "build, then debug." This meant spending huge amounts of time in debugging after the code was written and made it impossible to keep projects to a schedule because the debug/fix/rerun cycle was open-ended. This, in my view, is where Agile has rightfully distinguished itself--by having tests developed in parallel with code, thereby making debugging an "every once in a while" activity rather than a whole endless-seeming cycle.

GrumpyNl 8y ago

As soon as a work method becomes a religion, its doomed to fail. Treat Waterfall, Agile as a tool and modify it to your needs.

deleted 8y ago

[deleted]

bitL 8y ago

So we ended up with "Waterfall" being a historic accident, "Agile" turning into a cult, and with everybody knowing that great teams self-organize to top it all.

bpyne 8y ago

It's amazing how many times over my career I've watched and participated in debates on software development methodologies. HN seems to have one at least annually. The author of the post does a good job summing up the issues. I only skimmed it but have to agree with everything I read. I can add that there's a time and place for Waterfall: when requirements are well-known and static and when lots of oversight is needed. I can also add that no methodology has successfully dealt with the mega-projects we read about in software engineering case studies. Building stuff, whether it's software or anything other product, is hard. It's messy. Often the requirements are not well thought out. In the situations I normally deal with, a requester can state a high-level strategic goal for the organization, but not how the software can get us to it. That's where my team gets involved. One engineer I respect highly once said that we need to get something in front of the requester quickly so that we can find out what (s)he doesn't want. By process of elimination, the idea foments in the requester's mind and requirements become more concrete. Again, though, some organizations are very static or they deal with software whose failure can be catastrophic. In those situations, the highly bureaucratic Waterfall method can be helpful at least for slowing down the process.

pmart123 8y ago
I think the "Mythical Man-Month" does a great job accounting for the process and implementation of a mega-project. But as Brooks even stated, there's "No Silver Bullet."

moron4hire 8y ago

I've done a number of projects in both "waterfall"[0] and Agile. On the "waterfall" projects, we wrote lots of documentation. LOTS. Business cases for every major feature. Requirements documents with serial numbered acceptance criteria. Design documents explaining exactly how we were going to implement the requirements, with every REQ bullet needing to be accounted for explicitly. Test plans showing how each DES bullet was implemented correctly and how it proved the REQ was satisfied. And it all had to be correct and fresh. We weren't allowed to have anything disagree or slip in or drop out. If it wasn't in the REQs, it didn't go into the software. That meant if we discovered constraints that necessitated a change in design, we had to change the requirements. If you had to change requirements, it meant you had to reevaluate the business case. And that meant we were prototyping while we were writing the documentation. The documentation wasn't done until the software was about 80% complete. By that time, we were mostly just working the test plans and resolving defect tickets to get to 100%. Some of the best, longest lasting software I've ever written. All of the "Agile" projects I've been on are what I call BTSOOP development: "By The Seat Of Our Pants". There was no attention paid to documentation at all. It was hell getting PMs to write any tickets, let alone ones that were in scope. And of the tickets that were there, most of the devs ignored them, even if the PM didn't. Some people would mark tickets done that they just didn't want to do, hoping the PM would just blanket-accept tickets at the end of the Sprint. Which he most frequently did. The difference wasn't actually process. It was accountability for failing to adhere to the designated process. The Agile shops all had corporate policy on how projects would be ran, and nobody was ever taken to task for it. The "waterfall" shops didn't lack continuous integration and improvement or responsiveness to change, we just wouldn't sign off on a particular document until we knew it represented the final product, with linear dependencies on when different levels of assets could be approved (not when they could be worked on). [0] our product manager called it "the NASA way", and we fairly closely followed their documentation standards, if not their engineering ones, but we were all pretty junior at the time. IDK why I was made senior other than being the best programmer there, which at the time wasn't saying much.