7 ms·

I think waterfall was adopted for software development, not by accident, but because it was a well-defined methodology with proven effectiveness in areas like c

I think waterfall was adopted for software development, not by accident, but because it was a well-defined methodology with proven effectiveness in areas like construction and manufacturing. In areas where changes to the spec are expensive, like building a bridge, waterfall methodology makes a lot of sense. Now that dev tools are so good and computers so powerful, spec changes are cheap, so waterfall is inappropriate. However, there may be times where it is appropriate; maybe a controller for a pacemaker or an oil refinery. I see it as using the right process for the right job.

guntars 8y ago
That’s an interesting viewpoint, in my experience almost 100% of budget overruns and ensuing lack of quality comes from changing requirements or rather truly thinking through the requirements for the first time after the project is about halfway done. It would be my dream to work on a project that has everything figured out upfront, alas unlike bridges, every software project is pretty unique, otherwise why are you rebuilding something that’s already done?
- jrochkind1 8y ago
  Yeah, I don't think software changes are actually cheap at all. The difference is, I think, more that we are for whatever reasons better at predicting our true bridge requirements than we are at predicting our true software requirements. Which the OP discusses some.
- taeric 8y ago
  Oddly, most truly disastrous projects I have seen, have had no small amount of over engineering. There has been a bit of not understanding the requirements. But more is in not understanding current personal capabilities or needs. Think of it as training for a marathon. At the end of the day, it is knowable how many calories you will need to burn and how far you are going to run. You are still far more served by running several five and then ten kilometer races, than you are studying the marathon track you want to run. Similar goes for software. Make a few customers happy. Then more. Then more. Don't stall out thinking you need to be at Twitter scale for day one. Not even Twitter did that.
  - sgt101 8y ago
    I've seen this too. My conclusion is that this happens when people who are strong communicators and networkers - influential even if they don't think of themselves that way - either misunderstand the need or value of some component, or they wilfully use the project to do a thing that they personally want to do. This "project as a vehicle for something else" is really catastrophic and is often a way for someone to exit the organisation with a gilded CV. What's important is that it screws everyone else. On the other hand, the obsessed super engineer is not being malicious, they believe wholeheartedly that they are right. The thing is to recognise which is which... one is a cancer that must be cut out fast, the other is a hugely valuable contributor who is misaligned (and in this case much listening is in order as it might be the project is misaligned).
  - Retric 8y ago
    Twitter also failed regularly. The cost of failure varies, but in government work their are non zero risks of your failures making national news. For example the healthcare.gov’s failure the scale. Twitter only scaled well when they started over rather than trying to use Agle to bolt on add hock solutions.
- lmm 8y ago
  In my experience disastrous projects come of spending too much time thinking through requirements and not enough time actually validating them - what's perceived as "changing requirements" is more often than not actually a case of misunderstanding the requirement in the first place. Dozens of people in months of meetings can very easily specify something that is obviously impossible as soon as someone spends 5 minutes with a working prototype. That's why I've found Agile so much more effective - if you get a system that works end-to-end at the earliest possible stage it becomes so much clearer what's possible and what's not, and your design discussions are much more productive. You still have to make design decisions and tradeoffs, but you get to make them much later on when you have much more information to work with.
agumonkey 8y ago
Quite plausible. Software is ontologically different from the real world where everything takes times and lots of energy. And also you don't have recursive towers of processing stages. Software attracts the idea of agile/adaptive/recomputed almost analog (continuous stream) that is clashing with clear and solid planned steps.
- speedplane 8y ago
  I'm don't think software is necessarily "different" than many other types of projects. To implement agile project management you need high-quality instant communication. Tools like e-mail, video-conferencing, and Github made agile possible. In a pre-internet era, where every change could take weeks to permeate an organization, change became extraordinarily expensive. With faster communication, the cost of change decreased, and made agile development more attractive.
  - agumonkey 8y ago
    It's not the same. A computing system is theoretically capable of recompiling itself on the fly; traditional projects don't even think of that kind of frequency.
wpietri 8y ago
I don't think this bit is correct: > it was a well-defined methodology with proven effectiveness in areas like construction and manufacturing. If you look at major construction projects, waterfall approaches don't have a great track record. E.g., the San Francisco Bay Bridge had enormous cost overruns: https://www.citylab.com/equity/2015/10/from-250-million-to-65-billion-the-bay-bridge-cost-overrun/410254/ https://www.citylab.com/equity/2015/10/from-250-million-to-6... In contrast, the Empire State Building was not built with a waterfall approach. It was delivered on time and 18% under budget. I saw Mary Poppendieck, who has written some excellent books on software development, give a fantastic talk on it. There's a write-up of a related talk of hers here: https://chrisgagne.com/1255/mary-poppendiecks-the-tyranny-of-the-plan/ https://chrisgagne.com/1255/mary-poppendiecks-the-tyranny-of... My belief is that the waterfall approach is common not because it's effective, but because it conforms to certain naive fantasies people -- especially powerful people -- have about projects. I think it's basically King Canute writ large. (Large, and into a zillion GANTT charts.) I do agree that it's inappropriate for modern software projects. But I think this is also suspect: > maybe a controller for a pacemaker I've talked with a number of medical-device people, including Nancy van Schooenderwoert, who has been involved in the Agile world for at least a decade: https://sdmdglobal.iqpc.co.uk/speakers/nancy-van-schooenderwoert https://sdmdglobal.iqpc.co.uk/speakers/nancy-van-schooenderw... They believe that Agile approaches are way better even for life-critical systems. You do need much more rigorous testing, but given the way Agile approaches deliver core functionality early, you end up having way more time for that testing, and many more opportunities to discover ways to improve the testing approach.
- speedplane 8y ago
  > My belief is that the waterfall approach is common not because it's effective, but because it conforms to certain naive fantasies people -- especially powerful people -- have about projects. That is itself pretty naive. There is a reason why waterfall used to be popular, but it no longer is. Is it because people's ego's have ebbed? Doubtful. It's more likely that something else happened in the interim. The obvious change is the proliferation of instant communications. If communicating is hard and expensive, it often makes more sense to change things as little as possible. If it's easy, then change becomes less expensive. The internet gave us instant communication (e-mail, video conferencing, github, etc). Tools like these made agile more attractive. Humanity didn't just become enlightened.
ATsch 8y ago
I've heard before that Making software is like making a bridge... Except the design part takes almost all of the time, and building takes minutes at most.
manmal 8y ago
> Now that dev tools are so good and computers so powerful, spec changes are cheap I‘d like to differ. Changing the spec in every iteration is going to end you up with a suboptimal architecture, because at some point devs will stop tearing it all apart and just refactor „a bit“. Of course, that’s different for various project sizes - a mega project like FB obviously has to allow changed requirements, all the time. But I‘m not sure that should actually be called a project. An atomic project should be of finite duration, and it should be finishable. Ideally not longer than 6 months. For those, requirements should not change much after a short cool-down phase. Just like tasks/stories/features/epics, projects are made up of different levels with very different aspects. I find the ideas presented in the article appealing, because they are refreshingly realistic even for one-off freelance projects to pull off.
vanviegen 8y ago
I'm quite sure the work preparation stage of construction (design, technical drawings, calculations, planning, etc) is usually rather iterative. It's just that when creating software, the project is done when we complete that stage.
eksemplar 8y ago
It also works better than agile when you’re buying software and you can’t just sign on on something without knowing what it’ll cost. Which is 99% of the time in the real world. I think the golden middleway is somewhere in between, where you do the upfront analysis and figure out what to do, why you’re doing it and with whom, but don’t lock everything down tight in an unchangable requirement specification. I think methologies like service design can be really great, if you don’t know what you’re trying to do. I think agile methologies are decent for the actual development of software, but a software project is so much more than that, including benefit realisation. I think it’s embarrassing that we can plan and build a organic house for thousands of employees, and deliver on time and within budget, but we can’t build a piece of software to handle the digital workflow of labelling and storing documents. Hell, we plan city infrastructure and succeed, every year without going over budget. But building a piece of software to digitise vacation registration? Fuck no, that’ll go one year over time and cost four times what we expected, even though you could rightly build the same thing in a few months as a single focused developer. I’ve seen all sorts of project management models in action, none of them work for software. But agile is especially stupid from the buyers perspective. When I buy a new house I wouldn’t sign a contract for an unknown amount of finished rooms within the agreed upon timeframe and budget either. Mostly though, I think software developers just need to do better work, and stop blaming the project management for their own inability to deliver good work in a timely manner.
- walrus1066 8y ago
  'Mostly though, I think software developers just need to do better work, and stop blaming the project management for their own inability to deliver good work in a timely manner.' Even if this is the case, PM still responsible for blown deadline, because they did not estimate their teams delivery velocity correctly. They should either: a) give longer estimate, if their team quality is low. b) pay more to hire better, more productive developers. A good PM will also account for uncertainty in devs estimation, by padding etc Predictability of delivery is what ultimately counts, not giving optimistic deadlines.
- Agathos 8y ago
  > When I buy a new house I wouldn’t sign a contract for an unknown amount of finished rooms within the agreed upon timeframe and budget either. You hope you don't. But halfway into construction, everyone might realize the blueprints attached to that contract don't cover some detail you really need. Then if you want that done, you'll have to pay extra. That's why change orders are so common in construction.
_shadi 8y ago
Waterfall seems more like a factory assembly line to me, on each step someone will add a component and at the end you will have the full product, its just in software usually you don't know what the end product should be when you begin.
C1sc0cat 8y ago
I would agree "Agile" optimises for speed over anything else, Quality and cost are secondary concerns. If your developing software that manages your countrys core ip network getting the design right is key. In the above case we did waterfall design (nine months) and then used RAD/Agile to develop the software in 12 weeks

guntars 8y ago

That’s an interesting viewpoint, in my experience almost 100% of budget overruns and ensuing lack of quality comes from changing requirements or rather truly thinking through the requirements for the first time after the project is about halfway done. It would be my dream to work on a project that has everything figured out upfront, alas unlike bridges, every software project is pretty unique, otherwise why are you rebuilding something that’s already done?

jrochkind1 8y ago
Yeah, I don't think software changes are actually cheap at all. The difference is, I think, more that we are for whatever reasons better at predicting our true bridge requirements than we are at predicting our true software requirements. Which the OP discusses some.
taeric 8y ago
Oddly, most truly disastrous projects I have seen, have had no small amount of over engineering. There has been a bit of not understanding the requirements. But more is in not understanding current personal capabilities or needs. Think of it as training for a marathon. At the end of the day, it is knowable how many calories you will need to burn and how far you are going to run. You are still far more served by running several five and then ten kilometer races, than you are studying the marathon track you want to run. Similar goes for software. Make a few customers happy. Then more. Then more. Don't stall out thinking you need to be at Twitter scale for day one. Not even Twitter did that.
- sgt101 8y ago
  I've seen this too. My conclusion is that this happens when people who are strong communicators and networkers - influential even if they don't think of themselves that way - either misunderstand the need or value of some component, or they wilfully use the project to do a thing that they personally want to do. This "project as a vehicle for something else" is really catastrophic and is often a way for someone to exit the organisation with a gilded CV. What's important is that it screws everyone else. On the other hand, the obsessed super engineer is not being malicious, they believe wholeheartedly that they are right. The thing is to recognise which is which... one is a cancer that must be cut out fast, the other is a hugely valuable contributor who is misaligned (and in this case much listening is in order as it might be the project is misaligned).
- Retric 8y ago
  Twitter also failed regularly. The cost of failure varies, but in government work their are non zero risks of your failures making national news. For example the healthcare.gov’s failure the scale. Twitter only scaled well when they started over rather than trying to use Agle to bolt on add hock solutions.
lmm 8y ago
In my experience disastrous projects come of spending too much time thinking through requirements and not enough time actually validating them - what's perceived as "changing requirements" is more often than not actually a case of misunderstanding the requirement in the first place. Dozens of people in months of meetings can very easily specify something that is obviously impossible as soon as someone spends 5 minutes with a working prototype. That's why I've found Agile so much more effective - if you get a system that works end-to-end at the earliest possible stage it becomes so much clearer what's possible and what's not, and your design discussions are much more productive. You still have to make design decisions and tradeoffs, but you get to make them much later on when you have much more information to work with.

agumonkey 8y ago

Quite plausible. Software is ontologically different from the real world where everything takes times and lots of energy. And also you don't have recursive towers of processing stages. Software attracts the idea of agile/adaptive/recomputed almost analog (continuous stream) that is clashing with clear and solid planned steps.

speedplane 8y ago
I'm don't think software is necessarily "different" than many other types of projects. To implement agile project management you need high-quality instant communication. Tools like e-mail, video-conferencing, and Github made agile possible. In a pre-internet era, where every change could take weeks to permeate an organization, change became extraordinarily expensive. With faster communication, the cost of change decreased, and made agile development more attractive.
- agumonkey 8y ago
  It's not the same. A computing system is theoretically capable of recompiling itself on the fly; traditional projects don't even think of that kind of frequency.

wpietri 8y ago

I don't think this bit is correct: > it was a well-defined methodology with proven effectiveness in areas like construction and manufacturing. If you look at major construction projects, waterfall approaches don't have a great track record. E.g., the San Francisco Bay Bridge had enormous cost overruns: https://www.citylab.com/equity/2015/10/from-250-million-to-65-billion-the-bay-bridge-cost-overrun/410254/ https://www.citylab.com/equity/2015/10/from-250-million-to-6... In contrast, the Empire State Building was not built with a waterfall approach. It was delivered on time and 18% under budget. I saw Mary Poppendieck, who has written some excellent books on software development, give a fantastic talk on it. There's a write-up of a related talk of hers here: https://chrisgagne.com/1255/mary-poppendiecks-the-tyranny-of-the-plan/ https://chrisgagne.com/1255/mary-poppendiecks-the-tyranny-of... My belief is that the waterfall approach is common not because it's effective, but because it conforms to certain naive fantasies people -- especially powerful people -- have about projects. I think it's basically King Canute writ large. (Large, and into a zillion GANTT charts.) I do agree that it's inappropriate for modern software projects. But I think this is also suspect: > maybe a controller for a pacemaker I've talked with a number of medical-device people, including Nancy van Schooenderwoert, who has been involved in the Agile world for at least a decade: https://sdmdglobal.iqpc.co.uk/speakers/nancy-van-schooenderwoert https://sdmdglobal.iqpc.co.uk/speakers/nancy-van-schooenderw... They believe that Agile approaches are way better even for life-critical systems. You do need much more rigorous testing, but given the way Agile approaches deliver core functionality early, you end up having way more time for that testing, and many more opportunities to discover ways to improve the testing approach.

speedplane 8y ago
> My belief is that the waterfall approach is common not because it's effective, but because it conforms to certain naive fantasies people -- especially powerful people -- have about projects. That is itself pretty naive. There is a reason why waterfall used to be popular, but it no longer is. Is it because people's ego's have ebbed? Doubtful. It's more likely that something else happened in the interim. The obvious change is the proliferation of instant communications. If communicating is hard and expensive, it often makes more sense to change things as little as possible. If it's easy, then change becomes less expensive. The internet gave us instant communication (e-mail, video conferencing, github, etc). Tools like these made agile more attractive. Humanity didn't just become enlightened.

ATsch 8y ago

I've heard before that Making software is like making a bridge... Except the design part takes almost all of the time, and building takes minutes at most.

manmal 8y ago

> Now that dev tools are so good and computers so powerful, spec changes are cheap I‘d like to differ. Changing the spec in every iteration is going to end you up with a suboptimal architecture, because at some point devs will stop tearing it all apart and just refactor „a bit“. Of course, that’s different for various project sizes - a mega project like FB obviously has to allow changed requirements, all the time. But I‘m not sure that should actually be called a project. An atomic project should be of finite duration, and it should be finishable. Ideally not longer than 6 months. For those, requirements should not change much after a short cool-down phase. Just like tasks/stories/features/epics, projects are made up of different levels with very different aspects. I find the ideas presented in the article appealing, because they are refreshingly realistic even for one-off freelance projects to pull off.

vanviegen 8y ago

I'm quite sure the work preparation stage of construction (design, technical drawings, calculations, planning, etc) is usually rather iterative. It's just that when creating software, the project is done when we complete that stage.

eksemplar 8y ago

It also works better than agile when you’re buying software and you can’t just sign on on something without knowing what it’ll cost. Which is 99% of the time in the real world. I think the golden middleway is somewhere in between, where you do the upfront analysis and figure out what to do, why you’re doing it and with whom, but don’t lock everything down tight in an unchangable requirement specification. I think methologies like service design can be really great, if you don’t know what you’re trying to do. I think agile methologies are decent for the actual development of software, but a software project is so much more than that, including benefit realisation. I think it’s embarrassing that we can plan and build a organic house for thousands of employees, and deliver on time and within budget, but we can’t build a piece of software to handle the digital workflow of labelling and storing documents. Hell, we plan city infrastructure and succeed, every year without going over budget. But building a piece of software to digitise vacation registration? Fuck no, that’ll go one year over time and cost four times what we expected, even though you could rightly build the same thing in a few months as a single focused developer. I’ve seen all sorts of project management models in action, none of them work for software. But agile is especially stupid from the buyers perspective. When I buy a new house I wouldn’t sign a contract for an unknown amount of finished rooms within the agreed upon timeframe and budget either. Mostly though, I think software developers just need to do better work, and stop blaming the project management for their own inability to deliver good work in a timely manner.

walrus1066 8y ago
'Mostly though, I think software developers just need to do better work, and stop blaming the project management for their own inability to deliver good work in a timely manner.' Even if this is the case, PM still responsible for blown deadline, because they did not estimate their teams delivery velocity correctly. They should either: a) give longer estimate, if their team quality is low. b) pay more to hire better, more productive developers. A good PM will also account for uncertainty in devs estimation, by padding etc Predictability of delivery is what ultimately counts, not giving optimistic deadlines.
Agathos 8y ago
> When I buy a new house I wouldn’t sign a contract for an unknown amount of finished rooms within the agreed upon timeframe and budget either. You hope you don't. But halfway into construction, everyone might realize the blueprints attached to that contract don't cover some detail you really need. Then if you want that done, you'll have to pay extra. That's why change orders are so common in construction.

_shadi 8y ago

Waterfall seems more like a factory assembly line to me, on each step someone will add a component and at the end you will have the full product, its just in software usually you don't know what the end product should be when you begin.

C1sc0cat 8y ago

I would agree "Agile" optimises for speed over anything else, Quality and cost are secondary concerns. If your developing software that manages your countrys core ip network getting the design right is key. In the above case we did waterfall design (nine months) and then used RAD/Agile to develop the software in 12 weeks