6 ms·
“Waterfall” doesn't mean what you think it means
- CharlieDigital 3y agoThe most balanced and sensible paradigm I've come across is Basecamp's Shape Up[0]. I led an engineering team building validated systems that supported GxP processes in life sciences and found that a Shape Up-like process produced the best balance between delivering software that could be validated[1] and speed. I only came across it after several years of trial and error, but I was amazed at how it almost exactly described what we ended up doing. A side effect is that work-life balance is amazing once you get this process working. If you know exactly how much work you need to do and when it needs to get done by for the next 4-6 weeks, then you can pace yourself much better. When you plan 6 weeks at a time, someone going out sick or taking some time off has less of an impact than when you plan 1 or 2 weeks at a time. The customers and end-users know what to expect and when they'll get it; everyone gets more predictability in the process. Yes, it takes 1-2 weeks of planning at the outset, but we used that downtime for the engineering team to catch up on tech, build some experiments, work on side projects, or do research. (No refactoring since the validated process requires us to plan and document all code changes that make it into production) I recommend it for any team that wants some sanity in the processes. [0] https://basecamp.com/shapeup https://basecamp.com/shapeup [1] Validated in this context means that every feature, function, and infrastructure component can be traced back to a specification through the full lifecycle from design to install to post-install verification and operations.
- Zigurd 3y agoThere's a very practical reason why waterfall fails in most software development projects: in the waterfall model you do project planning and budgeting based on the specifications you write. I'm not arguing against the specifications other than to say that in a lot of cases in modern software development you haven't got time to write all of those specs to the proper level of detail. But then you go on to do a critical path analysis. That requires assigning people to tasks and having detailed task descriptions at a very early stage of the project. The odds of that being right are very slim. It can tell you at best whether your project is plausible. Again I have no problem with that. But once you start managing the project, the agile approach of assigning people to tasks in iteration planning meetings works a lot better than trying to stick to your painstakingly created critical path analysis. You also don't need highly detailed task descriptions and implementation plans until you get to your iteration planning meetings. Then they are crucial. But putting off that work until you need it saves a lot of time at project start.
- CharlieDigital 3y ago> The odds of that being right are very slim This really depends on the industry and the domain experience of the team. The deeper the domain experience, the more likely you are to get it right. It also helps to limit the scope of what you do (big part of the Shape Up process). Works well for life sciences, for example, where we're replacing a paper process with a digital process. Not so well for SV startups where first time founders are trying to figure out product-market-fit. > ...saves a lot of time at project start Often just shifting the work to a later cycle to get it right. In life sciences when supporting a GxP process, validation (which is rooted in documentation and specifications) is unavoidable. In that environment, figuring out how to move fast despite the constraints yielded a Basecamp Shape Up like process. I think every team that believes big-A Agile is The One True Way take a look at Shape Up.
- Zigurd 3y ago,> This really depends on the industry... That's correct. One of the commonest ways for agile to fail is to apply it to the wrong kind of project. If you have strong dependencies between tasks and things like lead times for hardware you will want to do project planning that can take those things into account early in the project.
- jacobr1 3y agoOne of the bigger issues with Scrum style agile, is that very rarely does it reach the maturity where "everything is delivered within a sprint." Even if you complete all the cards without carryover or other issues, the product isn't necessarily shippable. So in practice you some form of release planning, often quarterly to align with the business, that gets overlaid. The big insight with ShapeUp for me, was that is really is more a change in how to approach Release planning, than sprint/iteration process. You really can ship an end-to-end feature for a wide variety of software products in a 6 week timeframe, and it is short enough to give a lean focus, outcome oriented on delivery rather than just shipping a code increment. Two weeks is too tight a timeframe to meaningfully pivot larger projects, or make decision about overall feasibility. There always is another 2-weeks to expand upon your sunk costs. But again 6 weeks hits more of a sweet spot in being able to step back and make roadmap tradeoffs.
- MetaWhirledPeas 3y ago> A side effect is that work-life balance is amazing once you get this process working. If you know exactly how much work you need to do and when it needs to get done by for the next 4-6 weeks, then you can pace yourself much better. For you this may sound like heaven, but for me (possibly ADHD?) it sounds a little closer to hell. It reminds me of my school days; short assignments were fine. Large multi-week projects were a bit terrifying. So in short, you need to account for a multitude of personality types. People with shorter attention spans need shorter deadlines. People with longer attention spans need less pampering. Pick something that works for both.
- CharlieDigital 3y agoThey're not multi-week projects per se; more like: 1. "This is a big feature we are shipping in 6 weeks" 2. "Here are the 20-40 tasks that need to be completed to implement this feature" The only difference is that we've fully planned out 6 weeks of work rather than planning 1 or 2 weeks at a time. I would summarize and say that the difference isn't in the execution; the difference is in the planning. My experience is that it's the opposite for those with short attention spans; because everything is planned out, the engineering team can then kind of walk up to the "buffet table" and pick what they want to work on since you have the full spread. Within that 6 week cycle, we would still build, iterate, and test internally. It's not one giant ticket, but still broken down into parts. At the end of the 6 week cycle, there's a very clear picture of what will be released and there's a formal validation and release period that mostly involves testing and ensuring that the product meets the defined specification.
- MetaWhirledPeas 3y agoThis sounds a lot like Scrum with longer-than-average sprints. Or maybe its Kanban? I've lost the patience to pay attention to the intricacies of them all. The length of the cycle is going to vary based on your product and your customer. There's a lot to be gained by delivering rapidly if you can do it, with feature flags to hide what you want to keep hidden. 6 weeks for a release is a pretty long time to wait for most projects. Personally, this is my favorite approach, and maybe you'd agree to a point: 1. Team that is focused on the one "big feature" at a time. Maintainers operate on a different team. 2. Big feature should have a planned duration that spans multiple iterations, but not too long. 6 weeks sounds reasonable. 3. Test and deliver as rapidly as you can based on your capabilities. If all your testing is automated, consider Continuous Integration at the very least. If you rely on a team of testers then come up with an iteration length that gives testers enough time to regression test a snapshot of the code to prepare a release candidate, while still allowing enough time in-between for testing new changes. The reason 3 is important is because, whenever possible, you want to avoid the terrifying mountain of risk that comes with releasing code that has not seen the light of day for months on end. Typically the longer you spend preparing a release, the longer you have everyone panicking when the deadline is approaching and this massive release has a massive backlog of bugs.
- AstralStorm 3y agoSo funny thing about software prototypes. Time and again a prototype is pushed into production by greedy management seeking immediate returns - and then the actual system is never done.
- michaelcampbell 3y agoIt doesn't matter, but does your post here mean: "So, funny thing about software prototypes." or "SO FUNNY thing about software prototypes." Either is valid, but it struck me that I couldn't exactly divine your intent.
- carrja99 3y agoYeah I read it as the latter myself.
- freitzkriesler2 3y agoYup. Time and time again, the MVP ends up becoming the full production version with zero improvements. I'm glad I moved into hardware development.
- ghusto 3y agoIf the MVP stays in production without further investment, then it's the product, there's nothing minimal about it. That's a decision. If a prototype or PoC is in production though, something's gone wrong. I build my prototypes and PoCs in such a way that they literally can't be in production, which is I think the correct way to build a PoC anyway.
- swader999 3y agoWaterfall works fairly well on your third or fourth try at the same project.
- datadrivenangel 3y agoThis is the fundamental insight of the manifesto for agile software development. I like your brevity more though.
- swader999 3y agoI was being cheeky but true waterfall has iterative and self reflection built in yet these were rarely acknowledged and practised. And it honestly takes a couple of redos to get it all figured out and really understand the whole problem enough to make radical leaps. I've seen some projects go right to the ultimate solution only with exceptional people that have a very similar project already under their belts.
- codemac 3y agoWell, it does mean what I think it means because language is descriptive, not prescriptive :) I will never forget sitting in a 4-8 hour meeting for a day reviewing our entire dependency tree, and whether or not SJ-BCOMKA-4.11 had a dependency on SJ-TARGET-4.32. Management moved our "productive percentage time" from 80% to 85% to make the schedule look good to upper management. The scars of "waterfall" run very very deep. Second, it's clear you need to reapply anything written 35+ years ago to modern sw development. For example, "operations" as a stage of sw development, whereas today we have devops vs SRE vs production engineering vs whatever. Things like cleanroom development, zero defect software, etc are completely under utilized by software shops today, because we price the cost of maintenance & bugs far far too low. Which is was also unfortunately missing from the waterfall paper compared to Capers Jones or others analysis that showed it's not just that you should have preceding steps to "coding" - the preceding steps (aka "analysis") remove defects from your final product far far faster.
- xboxnolifes 3y ago> Well, it does mean what I think it means because language is descriptive, not prescriptive :) Individuals can still be wrong in their word usage following descriptive language thought.
- gedy 3y agoThis seems to skip over the main issue I've had with waterfall - management picking dates before detailed analysis. So you get a nasty situation where the detailed analysis eats into your available time for the project. This is mostly fine for people like UX, product or up-front architect types who don't have to deliver the software or deal with crunch time at end, but everything depends on what they spec out. So, sorry, not sorry, I greatly prefer agile/incremental work to that.
- rawgabbit 3y agoArbitrary deadlines also occur in agile scrum in my experience. Management says stop working on project X and start working on project Y. Project X backlog is effectively canceled and the product is in maintenance mode
- gedy 3y agoTrue, but my bias is to mainly avoid crunch time or death marches.
- backendanon 3y ago> This seems to skip over the main issue I've had with waterfall - management picking dates before detailed analysis. Yup. In Agile though, the opposite happens, we the devs are encouraged or coerced to agree to work tickets whose scope is not baked enough to start serious coding which should happen during planning or grooming...except raising a question you often get the "what part is not clear" condescending tone which discourages discourse and fleshing out of the ticket's scope.
- orwin 3y agoNot a fan of agile (I'm mostly working on Sre and security), but this is a failure of management, not agile.
- AnimalMuppet 3y agoNothing wrong with management picking dates, as long as they understand that they're movable. Find out that there's more work during detailed analysis? Move the dates or change the scope. Find something missed in analysis during implementation? Ditto. Find it during debugging? Ditto. The problem isn't picking the date. It's the response to the date being wrong. (The date is usually wrong, unless you're in a domain that your team knows very well.) And why do they respond badly to the date being wrong? Because they think the date can't be wrong. They expect it to be right. Well, if experience has shown us anything, it's that the date is wrong. And this is what's wrong with waterfall. They take the initial planning as cast in concrete, as brought down by Moses from Mount Sinai on tablets of stone. It's not. The initial plan is written on sand. Acting like that is true is the difference between waterfall and agile.
- mrkeen 3y agoThis is a long one. Is there anything beyond: "Don't call waterfall bad because what we call waterfall is actually iterative and iterative is good" ?
- mrweasel 3y agoNot really, but it does bear repeating, as many in sales, purchasing, project management, management in general, politics and possibly more, does not seem to grasp that iterations are an absolute requirement in a software development process. If we where able to design and deliver the exact requirement and perfectly estimate the time required, then that time would be zero, as the software would already have been written.
- rawgabbit 3y agoWow. This is what I learned almost 25 years ago. Documentation is king. It beats pair programming. It beats knowledge transition. It is supposed to be a living document. You are supposed to track-note where you took a different path. I assume the article’s reference to SAGE is referring to the system that routed early warning radar station data to NORAD. https://en.m.wikipedia.org/wiki/SAGE_radar_stations https://en.m.wikipedia.org/wiki/SAGE_radar_stations
- quacked 3y agoAll design methodologies have their drawbacks, but I think the most hellish design method is "Agilefall", where you use the trappings of Agile/Scrum/Kanban development (Jira, standups, tickets, etc.) to implement a series of hardcore requirements the clients thought up before anyone started building anything. I worked at a software company that built extremely sensitive survey applications (iOS, Android, and web-based). A significant part of my job entailed breaking down a literal design specification document into Jira tickets, which were then worked on by a rotating staff of programmers. The rotating programmers were assigned to 10-30 apps at one time and often had to make enormous customizations to the base product. The base product was formed out of 15 separate sub-products whose teams did not correspond with each other. The clients were allowed to make broad changes to the entire system by (a) changing the design document (b) emailing furious demands to the sales team/my manager (c) adding requirements to the master contract and not telling us. There was no traceability established between the contract/design doc/tickets, and I was not allowed to standardize any of those documents. There was no standardized format for writing the Jira tickets, so every person on my team made up their own method. The design document was missing about 40% of the context necessary to deliver a working app, so I added that context into the Jira tickets, but the clients weren't allowed to review the Jira tickets. I spent 20% of my time in "Scrum" meetings including sprint planning, sprint reflection, and hour-long standups. My company was worth hundreds of millions of dollars. I made a fat chunk of change in a year and a half, put a down payment on a house, and quit.
- randomdata 3y ago> and hour-long standups What's the point of standups, anyway? It seems like it always addresses the wrong people. I don't give a rat's ass about what other developers have done. Or, more accurately, I do care, but their work is highly visible. I don't need them to retell the same story. I already know what they are doing. What is not visible is what the leaders at the company do all day. Having a better understanding of their work would be far more impactful in allowing me to make better decisions around my work. Yet, if they show up to standups at all, they don't say much.
- quacked 3y ago
- mannykannot 3y ago"Waterfall" has become a straw man, to the extent that an article merely arguing that X is better than waterfall is wasting time that could be better spent on saying what X gets right. Similarly, arguing over what "waterfall" really means is just so much flogging of a dead horse.
- convolvatron 3y agoanother over-arching observation here is - how is it that we decided that there was going to be one set of process rules that was going to make us an effective organization. if we just ... use a 5 day development cycle, and a team meeting on Thursdays, and estimate tasks using a 5 level system and readjust on Tuesday then all our problems are going to go away. some process is necessary, but its not the heart of the matter
- deleted 3y ago[deleted]
- randomdata 3y ago> "Waterfall" has become a straw man It was always intended to be. Royce told that it was risky and invited failure and that organizations should include things like iterative processes to hedge against those risks. Basically, it was created to use as an example of what not to do.
- efortis 3y agoThe biggest problem is that we don't have adequate tools for writing requirements. Even wireframing tools convert the process into graphic design problems. https://uidrafter.com https://uidrafter.com is a free tool that domain-experts can use for prototyping without graphic design concerns.
- backendanon 3y ago> The biggest problem is that we don't have adequate tools for writing requirements. All that's needed is a text field and someone who can write requirements.
- efortis 3y agoWould you tell an architect all she needs is a pencil?
- kstrauser 3y agoHistorical note: that line of thinking also got us UML, which is now largely used as a guide to which shapes to draw on a whiteboard. Not saying your idea doesn’t have merit, just that I don’t want to end up with a dusty Universal Requirements Language book sitting on my shelf. Anyone who wants such a thing, learn how UML was written and then don’t do it that way.
- efortis 3y agoThey are different. UML is for thinking about the data, while UI Drafter is for focusing on the user needs. I'd say ERP tools are the result of what you are worring about.
- baumschubser 3y agoI do customization projects (the whole waterfall from requirements, specification and dev to docs, packaging, delivery and bugfixing) on a business application (that is developed in a more agile team). The customer wants a specific custom feature and once they have it it should be done and over with the project. They want to know the cost and they want to know the delivery date (the actual person I am talking with is a sales rep who is talking to the real customer, so they usually want to know those things in reverse order, but that’s manageable). The well-known problem: The customer doesn’t know what they really want until they see a first iteration in practice. The solution: train the sales rep (by courses and experience) to join you in becoming an awesome author team for req and spec documents. Keep the scope small enough to be able to oversee all cross-effects of the new features. Keep the discussion alive with the customer to talk about these effects. Will this always work? No, I had absolutely insane projects out of everyone’s control. But it will work most of the time and if it did not the cake was just too big to swallow in one piece. Bottom line: If waterfall fits or fits not depends in my experience on the scale of the project.
- apercu 3y agoThis is the same process I learned after too many years of unrealistic expectations and turning relationships tense by "change requisition clients" to death. Estimate the overall project effort the best that you can, then add 20-30% for risk/scope unknowns. Then pull 30% of the total level of effort in to a small contract that covers planning, specs, UX/prototyping and perhaps a limited scope technical PoC.
- marcosdumay 3y agoNah, "waterfall" means exactly what people think it means. For one reason, because that's how language works. But on a more impactful way, because what people think about it is exactly what was widely practiced at the time the paper was posted. Corporations did hire some requirements gathering consultant. That consultant would make a lot of documents, sign them down and go to another client. Then the corporation would do an auction and get a "supplier" based on the requirements docs, make a contract and sign down that the docs completely described what they expected and was exactly what should be delivered. Then an administrator would test the received software against the contract, and shove it in a shelf where the software wouldn't get in anybody's way, because nobody would want that thing. This is exactly how large corporations developed software. Of course, the researcher that coined the name was so out of touch with the industry that he thought he was creating a strawmen. That makes his description not very useful, even though it doesn't detract anything from the paper's quality.
- datadrivenangel 3y agoDeveloped past-tense? Large corporations and organizations still do this to an unfortunate degree.
- rawgabbit 3y agoI agree large corporations still RFP and contract out this way. The vendors are now “agile” but deadlines and hard requirements are still baked into modern contracts. I can’t tell the difference between now and the bad old days.
- throwbadubadu 3y agoAnd to be realistic, also owed to their size also some things need to go that way? And it is also wrong, because even those corps don't throw the product away, but start iterating then usually. And unfortunately on the other hand "modern", scrum becomes just smaller waterfalls. It all seems to be just a spectrum of sprint length and size of your feedback loops? But I'd encourage everybody to not just bitch about it but try to live that situation, a big tanker is just much different steering than a speedboat. Unlimited scrum trainers and agile mindsets won't help that. In this sense I'm not sure what neither waterfall nor modern agile in its many variation means, just certainly it is always easy to laugh from the outside.
- asow92 3y agoDoes it bother anyone else that we use the word "sprint" in agile? Does it make sense to always be "sprinting"? Personally, the idea of floating down a waterfall sounds way more appealing.
- mindcrime 3y agoDoes it bother anyone else that we use the word "sprint" in agile? FWIW, that's not a universal thing. A lot of people use "iteration" in preference to "sprint" - but sadly, not enough. I think the term "sprint" should be abolished personally. Does it make sense to always be "sprinting"? Absolutely not. And it may be "just a metaphor" but I think the metaphors we use matter. And you absolutely can't be "always sprinting". It's an absurd notion.
- topaz0 3y agoMaybe the next methodology should be "lazy river" or similar.
- broast 3y agoThe part where you are floating is not the waterfall
- asow92 3y agoTouché, but can we call standup the splash zone?
- crazygringo 3y agoIt's meant to be a metaphor only for distance, not for speed. In running/swimming, the short events are called sprints. In contrast with long-distance events. Does anybody have a better metaphor, a noun word that suggests a short distance, in contrast against a long distance? (E.g. "block" or "jump" or "hop" don't really work because none of them inherently communicate the idea of shorter -- a hop is longer than a step, for instance.)
- rawgabbit 3y ago
- OliverJones 3y agoIf you're seriously interested in this topic, it's worth taking a look at Jen Pahlka's book Recoding America. https://www.worldcat.org/title/Recoding-America-:-why-government-is-failing-in-the-digital-age-and-how-we-can-do-better/oclc/1338301096 https://www.worldcat.org/title/Recoding-America-:-why-govern... Ms. Pahlka crawls through some large-system horror stories, and some success stories. The horror stories share something in common: design documentation that outlives its usefulness, takes on a life of its own, and drives implementation decisions in harmful directions. Brandow's article (the subject of this thread) emphasizes the usefulness of documentation. And it's certain that long-lived software systems (like his php example) need excellent documentation. That documentation is targeted at people like many of us. It answers questions like, "how do I use that blankety-blank IMAP module?". But long-lived documentation is often misunderstood and misused. For an overstated example, some "non-technical" administrator might read the php docs and somehow infer that "systems developed in php MUST use the IMAP module, or it won't pass acceptance testing." Then, the developers -- employees of some contract development team -- will comply. They'll have to figure out some way to incorporate email store-and-forward systems into the product, even if it makes no sense to do that. There's a danger inherent in extensive documentation: it WILL be misunderstood in the future, and those misunderstandings may turn out to be costly. More systematic documentation -- for example boilerplate "scope" sections at the beginning of docs -- are not a great way to mitigate that danger. Seriously, go read Pahlka's book.
- KronisLV 3y agoI recall once developing a new module for an existing system in a more waterfall-like approach than one would expect nowadays and it actually worked out nicely. There were about 50 pages of specs for it: descriptions, wireframes for the UI, how everything should fit together and what the typical workflows with it would be. All of this was available ahead of time, before writing code even started, which actually let me get a better idea about the expectations and needs, than focusing more on immediate needs in a more iterative approach. I could also discuss any uncertain items and get some other questions answered, to avoid confusion later down the line. It helped me write the code better and anticipate how to structure things, so less refactoring needed to be done. I could also focus on the "happy path" first and foremost, getting the critical functionality in place and then adding the nice to haves around it. Admittedly, there were problems with the specs, which I brought up and promptly solved when it became relevant, so the final solution differed slightly, but by then the specs were also updated. With the critical stuff done early, it was also possible to release the thing early on and then switch to a more iterative method of development, with improvements and all. Not quite the traditional waterfall approach, not exactly scrum either, but something that worked nicely regardless. Complete specs (even an overview of the big picture) can definitely be a nice to have in some cases, if doing that additional planning at the start isn't too much of a burden. Take whatever works from all of the different methodologies out there.
- marcus_holmes 3y ago> 3. No matter how much you plan, unexpected problems will arise while building. So we're ignoring this, or what? But otherwise, yes, I've long maintained that waterfall is actually iterative done badly. Nice to see it recognised.
- karmakaze 3y agoA waterfall process has its current common perceptions. Knowing that, I would be skeptical of a project that declared that it was following a waterfall development process without clarifications of deviations/adjustments.
- albert_e 3y agoOn a tangent - I find the typography on this site very pleasant to read. Seems to be "Sana Sans" (but probably more accurately "Sana Sans Alt"?) Bookmarking this to study it more.
- dave1999x 3y agoI didn't take that away from that paper, I thought waterfall was a bad one and royce presented an alternative. But the truth is the people who advocate waterfall in this day and age, either never have never actually done it or have some rose-tinted nostalgia. Crunching for weeks on end to hit a deadline sucks. Especially when some manager beats you up for not following a specification that makes no sense.