6 ms·

Rewrites are Waterfall

60 points by jeffrey-sherman 3y ago

chupapimunyenyo 3y ago
[flagged]
- brazzy 3y ago
  Recent? The meme basically started with this article: https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-... - note the date in the URL.
  - paiute 3y ago
    I’m not sure that article really aged well. Code rots pretty quickly nowadays. Convoluted old code no one understands starts to create wicked bugs as future teams wedge in new features. Processors get overloaded. Memory leaks appear. Libraries evolve. Design trends change.
  - chupapimunyenyo 3y ago
    I'm just seeing a lot of articles about bad rewrites recently, it could have started even before this one. It's just it wasn't as prominent as it is now I think
tempodox 3y ago
An astute observation, and presented without any extraneous fluff.
redundantly 3y ago
I appreciate that this isn't an overly long blog post that says too little by repeating itself a lot, but I'd prefer a little more to it. Some personal or second hand experiences of completed or failed rewrites would be nice. Alas, as it stands, this post has very little to offer. tl;dr: show me the money
mitthrowaway2 3y ago
It might be correct, it might not; either way I would find this more convincing if there were even one supporting argument behind these assertions. I certainly don't believe them as presented here.
mlinhares 3y ago
If you're looking for a good read on why large scale rewrites are unlikely to succeed go read Fred Brooks "Second System Effect" essay (and go and read the whole book anyway): https://archive.org/details/mythicalmanmonth00broo/page/52/mode/2up https://archive.org/details/mythicalmanmonth00broo/page/52/m... Large scale rewrites are really good at promoting the people that had the idea and "delivered" the rewrite with one or two bogus metrics improved and leaving a dumpster fire that those that will have to maintain it in the long run will pay the actual price for. Only get involved in large scale rewrites if you're the ideator that will be blessed with the laurels of the first delivery, if you're going to own the shitshow later, run away while you can.
- nathants 3y ago
  this is why it’s important to always rewrite twice in rapid succession. keep 3rd if better than 1st.
- hinkley 3y ago
  The only successful rewrites I’ve been involved with were clandestine ones. A conspiracy of developers and occasionally one first level manager to swap all the parts of a “working” system for ones that didn’t need scare quotes. Usually trying to stay just ahead of requirements that feel like giant left turns for the original code, but are tenable in the new.
gfody 3y ago
that last sentence doesn't make sense to me - as a leader I enjoy succeeding rather than failing so if a rewrite is what's necessary then that's the fun option. rewrites happen when changing requirements bent your code past its limit. they're natural and necessary and resisting one on principle is way less fun than the alternative.
brazzy 3y ago
> First you need to determine the requirements. All of them. In excruciating detail. Funny, I am currently in charge of a rewrite that did exactly the opposite (start with an MVP, do agile increments), and just recently went into production with great success. It was the third attempt to replace a 25 year old web app written in C. The previous attempts both failed exactly because they tried to "determine the requirements in excruciating detail" - they never got past that stage.
wheels 3y ago
This feels too dogmatic to be very useful. I'll ofter a softer version that I think everyone can agree to? - Don't mix rewriting code and adding features in the same commits - If you're squashing commits on a feature branch, and there are rewrites in there, cherry-pick them to master and rebase That at least leaves you with the ability to track bugs down to rewrites vs. feature additions. The advice in the original post feels like it's potentially useful if it's in a large system that sees heavy use with good test coverage. Then it's probably the right impulse. But a lot of code isn't that.
PedroBatista 3y ago
The only fun part about re-writes are the days we spend fantasizing about how much better it will be, that's it. Don't do a re-write kids, but if you do, do it from within the existing software like a parasite or a chest bursting creature from the movie Aliens. Either way it always be messy, terrifying and most of the time ends in tragedy.
- deleted 3y ago
  [deleted]
- zmgsabst 3y ago
  This is where the derivative of the facade pattern comes from: The smother pattern. Where you write a facade, then replace calls iteratively with the rewrite until you’ve removed all the calls to the old API. Then you turn it off. Sometimes rewrites are genuinely a good idea, eg, you’ve lifted-and-shifted a service but would like to better integrate with cloud services.
- hinkley 3y ago
  The other fun thing is that management tends to give you benefit of the doubt for the first six months give or take. They don’t know it’s a boondoggle yet that you will eventually come to regret almost as much as the original (and which they will regret more). This realization dawns slowly enough for most people that they mistake it for a desire to move on to another project. A certain number of the guilty parties will have moved on before the bill comes due.
- j45 3y ago
  Rerouted are definitely more work than usually anticipated, unless most or all of the original team is available. The other option to figure it out again from scratch has its own issues. If existing business logic works and has been hardened, it can be worth looking at wrapping it an API instead of rewriting and reimagining from scratch.
- pinkgolem 3y ago
  >Don't do a re-write kids, but if you do, do it from within the existing software like a parasite or a chest bursting creature from the movie Aliens. Honestly, this approach worked well for me in the past. Did rewrite parts of an vb6 application in .net really was a Frankenstein monster.
crazygringo 3y ago
This post doesn't provide any evidence or even any arguments. There are lots of different types of rewrites, and rarely does anyone find themselves in the position to rewrite an existing system in a way that perfectly reproduces behavior 1:1, because rewrites are often motivated by a ton of ugly features/flows that were forced as kludges, where the whole point is to improve them, along with making it vastly easier/cheaper to write new features. And no, waterfall works just as badly for rewrites as everything else, because when it comes time to program it, you run into bugs and unexpected behavior with libraries and the realization that your new architecture has problems and all of the problems that come with regular non-rewrite programming. That idea that "First you need to determine the requirements. All of them. In excruciating detail." is the problem waterfall always encounters -- to write the perfect list of all requirements, that's ultimately writing all the code. Such supposedly perfect waterfall specifications are an illusion. So no. Approach rewrites with the modern common sense of any software project, which involves defining and building out working components incrementally with testing, according to defined high-level priorities, and flexibly dealing with issues along the way as you run into the same unpredictable problems you always do. And finally: > Leaders who have delivered successful rewrites often describe the process as one of the greatest accomplishments in their career, but they never describe it as fun. This sounds made-up. What leaders? Why would a leader ever describe a rewrite as one of the greatest accomplishments of their career? I've never heard of such a thing. And why shouldn't it be fun? Rewriting something to wipe away all the old horrible tech debt and start over from scratch can be super-fun!
- saghm 3y ago
  > > Leaders who have delivered successful rewrites often describe the process as one of the greatest accomplishments in their career, but they never describe it as fun. > This sounds made-up. What leaders? Why would a leader ever describe a rewrite as one of the greatest accomplishments of their career? I've never heard of such a thing. And why shouldn't it be fun? Rewriting something to wipe away all the old horrible tech debt and start over from scratch can be super-fun! It's also pretty hard to argue against a statement like that; if you point out someone who says otherwise, they can just say "they're not a leader". If anything, it's somewhat circular, since basically they're defining "leaders" as people who agree with them.
teeray 3y ago
There’s a lot of orgs out there that will do all of: 1. Never consider a rewrite / v2. 2. Never schedule refactoring work. 3. Not accept any refactoring work as part of feature work, because the refactoring is always deemed more risky (despite tests). “Let’s split this PR into feature and refactor.” So my question to these orgs is: when are we supposed to clean up the hacks to ship yesterday’s features?
- Scarblac 3y ago
  You know the answer: never. It works, doesn't it? And they're right more often than I like. Edit: I'm definitely not saying they're always right, or even often. I'm usually with the line of thinking of the people commenting to me. It's just, I've also seen systems that had ten years of further life and never needed their hacks fixed. We developers are sometimes too fixed on the quality.
  - rjblackman 3y ago
    they're right over and over, until they're not and then the company tanks because they need a rewrite.
    - phito 3y ago
      Yep exactly my experience working in a startup. When it was time to make a real product our of the shit spaghetti code they wrote to win customers with """working""" demos, it was too late. Bankruptcy followed.
      - majikandy 3y ago
        Isn’t this a slightly different problem? The product built was just vapourware and not really doing the job. So customers went elsewhere when the real version didn’t appear in time? Often companies will win the customers with a genuine working product but complicated code and then hire like crazy and all the new developers struggle and want to rewrite it all. But they mustn’t because it is a working product earning money. So gradual improvement and controlled rewrites of sub areas that need to change the most are focused on to succeed. If a company has gone bankrupt because of spaghetti code of a demo product, it feels to me like the problem was the product being just a demo rather than it being spaghetti code?
taspeotis 3y ago
Company I used to work for did a rewrite for one app with great results. This was waterfall style: shipped it when it was done. Same company did a refactor for one app with great results. Waterfall style. Same company did a rewrite that crashed and burned. Shipped the new app side by side the existing one as they gradually added features. TL;DR Anecdotes
- rjblackman 3y ago
  this is interesting. in my experience it is always the third situation. they try to migrate the data slowly from "old system" to "nexgen". it never works. it sounds like maybe the answer is do a big bang switch over to the new system or don't rewrite at all. (i.e. the system you are rewriting probably needs to be small enough)
  - hinkley 3y ago
    Ship of Theseus is the other way. Fact is a lot of companies have code that has been rewritten and they don’t know it, because the devs that did it know they would have been stopped if people knew. But piece by piece, bit by bit, everything was shifted to a new organizational paradigm from the original. You can’t really do objective case studies on development processes that involve subterfuge. Even if they work.
linster 3y ago
But we could, facetiously, do a rewrite every two weeks in a time box and call it agile.
ahhfgsado6698 3y ago
You guys are asking for permission and scheduling time to do your rewrites? Leaders? Pfft. Bang it out over a weekend and then slide it in on a slow Monday morning.
js8 3y ago
I like Agile, but I think the industry needs lot more Waterfall. Imagine building a house with Agile, you would start from the roof (rain is the most pressing customer's problem), and the foundation would be built last (after, 5 years into the future, a crack appeared in the house wall). I also think what's problematic is this debate goes too much around project management, rather than actual methodology of building software (software engineering). I have an idea how we can reframe the debate to this effect. When software engineers plan and write software, they actually build two things - the product itself and the infrastructure/tooling for it. From the project and product management perspective, the focus is on the first and the latter is often neglected as unimportant (and waste of time). This sort of jives with the human intuition, we want to build a house, while foundation and scaffolding are only incidental to it. However, I can't help but notice that the software products that are actually most "agile", and easy to change, are written in a way where there is a huge amount of generic infrastructure and tooling with a tiny veneer of product ("business logic") on top of them (and in fact this goes all the way down). For a canonical example, Emacs, to the point where people even joke about it. But also, this is completely different to how the commercial industry builds software. The focus is on very specific features, pieces of business logic, instead of generic infrastructure, that makes delivering changing features easy. That is a very short-term view, which actually becomes more expensive as the time goes on. I think the way we should build (commercial) software products is the first one, lots of infrastructure and tooling which only incidentally happens to create a product. See also functional core / imperative shell, or Unix philosophy. That's why I think Agile failed in the industry, because the project management people who pushed for it, didn't understand the above distinction. This distinction is also related to Eric Raymond's the Cathedral and the Bazaar distinction, although his is more about social organization around building software than the actual software architecture. (The relation manifests in Conway's law.) To conclude, I think we need more Waterfall in the sense that we need to think (plan) more deeply about the infrastructure and tooling that we need to build our products. Jumping directly into building a product that solves customer problem will have detrimental effects on long-term productivity. I would also add I see role of open source only as a relatively small part of all the infrastructure and tooling. As you get closer to the actual product, more and more infrastructure is specific to your business domain. The open source will not save you, because it's too generic. Emacs is not just a generic OS or a Lisp interpreter. It has huge amount of infrastructure around text editing specifically.
1letterunixname 3y ago
Rewriting != waterfall. Waterfall is a formal development model (5-10+ phases) that includes strict requirements, specifications, documentation, development, testing, and deployment. If you need to build an elevator control system or a SCADA system for a steelworks plant, then definitely go there. Rewrite is throwing away code (and downstream) and starting over. There comes a point in some projects when support costs exceed greenfield development costs, or prevent forward velocity due to tech debt. This is rare, but it happens. The problem of massive rewrites is they are too often not lean and equivalent to big design upfront (BDU) because they cannot deliver an MVP and they throw away features users depended on (but failed to questionnaire, focus group, or look at telemetry for). The other issue is massive rewrites too often are used an excuse to avoid refactoring and selective rewrites with the magical thinking that "new == better". Too often, this is reinventing the wheel that isn't much better for 2x the effort.
jeffrey-sherman 3y ago
Author of the post here - Lots of comments about the piece being a short opinion piece without evidence. Totally fair! If you’d like to learn more I have a podcast, https://www.neverrewrite.com/ https://www.neverrewrite.com/ Our latest episode is about Khan Academy’s 3.5 year migration from Python to Go. https://youtu.be/LqAkIkNs1Ws?si=INgD-QVDhL8INH8e https://youtu.be/LqAkIkNs1Ws?si=INgD-QVDhL8INH8e
zem 3y ago
rewrites (successful ones) are some of the most fun I've had in my entire career! it's incredibly satisfying to already have all the requirements and end user features crystallized and battle tested, and have your sole job be to figure out a better way to express them in designs, algorithms and code.
retrocryptid 3y ago
Sherman does not appear to understand what waterfall is or why people rewrite code. Waterfall specifies that phases of a single implementation are performed one after the other, completing one before the next phase begins. The classic example lists this as "design->coding->testing". What Sherman is describing is "iterative design," where you build something, put it in front of customers, then make changes suggested by customers (or maybe there was a list of features you wanted to add yourself.) Throwing away code is okay if it leads to "better" code where better means faster, easier to fix, smaller code size, etc. Often you do not know what the issues with your code are until you write it. Writing code for "experimental" purposes is decidedly iterative, not waterfall.
lasereyes136 3y ago
I have never seen a rewrite where there were no new features added to the software. Maybe the is part of the message, don't add features until you hhave replaced the old system. That sounds like a good idea but almost no one approaches it like that, which is one of the reasons rewrites often fail.

chupapimunyenyo 3y ago

[flagged]

brazzy 3y ago
Recent? The meme basically started with this article: https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-... - note the date in the URL.
- paiute 3y ago
  I’m not sure that article really aged well. Code rots pretty quickly nowadays. Convoluted old code no one understands starts to create wicked bugs as future teams wedge in new features. Processors get overloaded. Memory leaks appear. Libraries evolve. Design trends change.
- chupapimunyenyo 3y ago
  I'm just seeing a lot of articles about bad rewrites recently, it could have started even before this one. It's just it wasn't as prominent as it is now I think

tempodox 3y ago

An astute observation, and presented without any extraneous fluff.

redundantly 3y ago

I appreciate that this isn't an overly long blog post that says too little by repeating itself a lot, but I'd prefer a little more to it. Some personal or second hand experiences of completed or failed rewrites would be nice. Alas, as it stands, this post has very little to offer. tl;dr: show me the money

mitthrowaway2 3y ago

It might be correct, it might not; either way I would find this more convincing if there were even one supporting argument behind these assertions. I certainly don't believe them as presented here.

mlinhares 3y ago

If you're looking for a good read on why large scale rewrites are unlikely to succeed go read Fred Brooks "Second System Effect" essay (and go and read the whole book anyway): https://archive.org/details/mythicalmanmonth00broo/page/52/mode/2up https://archive.org/details/mythicalmanmonth00broo/page/52/m... Large scale rewrites are really good at promoting the people that had the idea and "delivered" the rewrite with one or two bogus metrics improved and leaving a dumpster fire that those that will have to maintain it in the long run will pay the actual price for. Only get involved in large scale rewrites if you're the ideator that will be blessed with the laurels of the first delivery, if you're going to own the shitshow later, run away while you can.

nathants 3y ago
this is why it’s important to always rewrite twice in rapid succession. keep 3rd if better than 1st.
hinkley 3y ago
The only successful rewrites I’ve been involved with were clandestine ones. A conspiracy of developers and occasionally one first level manager to swap all the parts of a “working” system for ones that didn’t need scare quotes. Usually trying to stay just ahead of requirements that feel like giant left turns for the original code, but are tenable in the new.

gfody 3y ago

that last sentence doesn't make sense to me - as a leader I enjoy succeeding rather than failing so if a rewrite is what's necessary then that's the fun option. rewrites happen when changing requirements bent your code past its limit. they're natural and necessary and resisting one on principle is way less fun than the alternative.

brazzy 3y ago

> First you need to determine the requirements. All of them. In excruciating detail. Funny, I am currently in charge of a rewrite that did exactly the opposite (start with an MVP, do agile increments), and just recently went into production with great success. It was the third attempt to replace a 25 year old web app written in C. The previous attempts both failed exactly because they tried to "determine the requirements in excruciating detail" - they never got past that stage.

wheels 3y ago

This feels too dogmatic to be very useful. I'll ofter a softer version that I think everyone can agree to? - Don't mix rewriting code and adding features in the same commits - If you're squashing commits on a feature branch, and there are rewrites in there, cherry-pick them to master and rebase That at least leaves you with the ability to track bugs down to rewrites vs. feature additions. The advice in the original post feels like it's potentially useful if it's in a large system that sees heavy use with good test coverage. Then it's probably the right impulse. But a lot of code isn't that.

PedroBatista 3y ago

The only fun part about re-writes are the days we spend fantasizing about how much better it will be, that's it. Don't do a re-write kids, but if you do, do it from within the existing software like a parasite or a chest bursting creature from the movie Aliens. Either way it always be messy, terrifying and most of the time ends in tragedy.

deleted 3y ago
[deleted]
zmgsabst 3y ago
This is where the derivative of the facade pattern comes from: The smother pattern. Where you write a facade, then replace calls iteratively with the rewrite until you’ve removed all the calls to the old API. Then you turn it off. Sometimes rewrites are genuinely a good idea, eg, you’ve lifted-and-shifted a service but would like to better integrate with cloud services.
hinkley 3y ago
The other fun thing is that management tends to give you benefit of the doubt for the first six months give or take. They don’t know it’s a boondoggle yet that you will eventually come to regret almost as much as the original (and which they will regret more). This realization dawns slowly enough for most people that they mistake it for a desire to move on to another project. A certain number of the guilty parties will have moved on before the bill comes due.
j45 3y ago
Rerouted are definitely more work than usually anticipated, unless most or all of the original team is available. The other option to figure it out again from scratch has its own issues. If existing business logic works and has been hardened, it can be worth looking at wrapping it an API instead of rewriting and reimagining from scratch.
pinkgolem 3y ago
>Don't do a re-write kids, but if you do, do it from within the existing software like a parasite or a chest bursting creature from the movie Aliens. Honestly, this approach worked well for me in the past. Did rewrite parts of an vb6 application in .net really was a Frankenstein monster.

crazygringo 3y ago

This post doesn't provide any evidence or even any arguments. There are lots of different types of rewrites, and rarely does anyone find themselves in the position to rewrite an existing system in a way that perfectly reproduces behavior 1:1, because rewrites are often motivated by a ton of ugly features/flows that were forced as kludges, where the whole point is to improve them, along with making it vastly easier/cheaper to write new features. And no, waterfall works just as badly for rewrites as everything else, because when it comes time to program it, you run into bugs and unexpected behavior with libraries and the realization that your new architecture has problems and all of the problems that come with regular non-rewrite programming. That idea that "First you need to determine the requirements. All of them. In excruciating detail." is the problem waterfall always encounters -- to write the perfect list of all requirements, that's ultimately writing all the code. Such supposedly perfect waterfall specifications are an illusion. So no. Approach rewrites with the modern common sense of any software project, which involves defining and building out working components incrementally with testing, according to defined high-level priorities, and flexibly dealing with issues along the way as you run into the same unpredictable problems you always do. And finally: > Leaders who have delivered successful rewrites often describe the process as one of the greatest accomplishments in their career, but they never describe it as fun. This sounds made-up. What leaders? Why would a leader ever describe a rewrite as one of the greatest accomplishments of their career? I've never heard of such a thing. And why shouldn't it be fun? Rewriting something to wipe away all the old horrible tech debt and start over from scratch can be super-fun!

saghm 3y ago
> > Leaders who have delivered successful rewrites often describe the process as one of the greatest accomplishments in their career, but they never describe it as fun. > This sounds made-up. What leaders? Why would a leader ever describe a rewrite as one of the greatest accomplishments of their career? I've never heard of such a thing. And why shouldn't it be fun? Rewriting something to wipe away all the old horrible tech debt and start over from scratch can be super-fun! It's also pretty hard to argue against a statement like that; if you point out someone who says otherwise, they can just say "they're not a leader". If anything, it's somewhat circular, since basically they're defining "leaders" as people who agree with them.

teeray 3y ago

There’s a lot of orgs out there that will do all of: 1. Never consider a rewrite / v2. 2. Never schedule refactoring work. 3. Not accept any refactoring work as part of feature work, because the refactoring is always deemed more risky (despite tests). “Let’s split this PR into feature and refactor.” So my question to these orgs is: when are we supposed to clean up the hacks to ship yesterday’s features?

Scarblac 3y ago
You know the answer: never. It works, doesn't it? And they're right more often than I like. Edit: I'm definitely not saying they're always right, or even often. I'm usually with the line of thinking of the people commenting to me. It's just, I've also seen systems that had ten years of further life and never needed their hacks fixed. We developers are sometimes too fixed on the quality.
- rjblackman 3y ago
  they're right over and over, until they're not and then the company tanks because they need a rewrite.
  - phito 3y ago
    Yep exactly my experience working in a startup. When it was time to make a real product our of the shit spaghetti code they wrote to win customers with """working""" demos, it was too late. Bankruptcy followed.
    - majikandy 3y ago
      Isn’t this a slightly different problem? The product built was just vapourware and not really doing the job. So customers went elsewhere when the real version didn’t appear in time? Often companies will win the customers with a genuine working product but complicated code and then hire like crazy and all the new developers struggle and want to rewrite it all. But they mustn’t because it is a working product earning money. So gradual improvement and controlled rewrites of sub areas that need to change the most are focused on to succeed. If a company has gone bankrupt because of spaghetti code of a demo product, it feels to me like the problem was the product being just a demo rather than it being spaghetti code?

taspeotis 3y ago

Company I used to work for did a rewrite for one app with great results. This was waterfall style: shipped it when it was done. Same company did a refactor for one app with great results. Waterfall style. Same company did a rewrite that crashed and burned. Shipped the new app side by side the existing one as they gradually added features. TL;DR Anecdotes

rjblackman 3y ago
this is interesting. in my experience it is always the third situation. they try to migrate the data slowly from "old system" to "nexgen". it never works. it sounds like maybe the answer is do a big bang switch over to the new system or don't rewrite at all. (i.e. the system you are rewriting probably needs to be small enough)
- hinkley 3y ago
  Ship of Theseus is the other way. Fact is a lot of companies have code that has been rewritten and they don’t know it, because the devs that did it know they would have been stopped if people knew. But piece by piece, bit by bit, everything was shifted to a new organizational paradigm from the original. You can’t really do objective case studies on development processes that involve subterfuge. Even if they work.

linster 3y ago

But we could, facetiously, do a rewrite every two weeks in a time box and call it agile.

ahhfgsado6698 3y ago

You guys are asking for permission and scheduling time to do your rewrites? Leaders? Pfft. Bang it out over a weekend and then slide it in on a slow Monday morning.

js8 3y ago

I like Agile, but I think the industry needs lot more Waterfall. Imagine building a house with Agile, you would start from the roof (rain is the most pressing customer's problem), and the foundation would be built last (after, 5 years into the future, a crack appeared in the house wall). I also think what's problematic is this debate goes too much around project management, rather than actual methodology of building software (software engineering). I have an idea how we can reframe the debate to this effect. When software engineers plan and write software, they actually build two things - the product itself and the infrastructure/tooling for it. From the project and product management perspective, the focus is on the first and the latter is often neglected as unimportant (and waste of time). This sort of jives with the human intuition, we want to build a house, while foundation and scaffolding are only incidental to it. However, I can't help but notice that the software products that are actually most "agile", and easy to change, are written in a way where there is a huge amount of generic infrastructure and tooling with a tiny veneer of product ("business logic") on top of them (and in fact this goes all the way down). For a canonical example, Emacs, to the point where people even joke about it. But also, this is completely different to how the commercial industry builds software. The focus is on very specific features, pieces of business logic, instead of generic infrastructure, that makes delivering changing features easy. That is a very short-term view, which actually becomes more expensive as the time goes on. I think the way we should build (commercial) software products is the first one, lots of infrastructure and tooling which only incidentally happens to create a product. See also functional core / imperative shell, or Unix philosophy. That's why I think Agile failed in the industry, because the project management people who pushed for it, didn't understand the above distinction. This distinction is also related to Eric Raymond's the Cathedral and the Bazaar distinction, although his is more about social organization around building software than the actual software architecture. (The relation manifests in Conway's law.) To conclude, I think we need more Waterfall in the sense that we need to think (plan) more deeply about the infrastructure and tooling that we need to build our products. Jumping directly into building a product that solves customer problem will have detrimental effects on long-term productivity. I would also add I see role of open source only as a relatively small part of all the infrastructure and tooling. As you get closer to the actual product, more and more infrastructure is specific to your business domain. The open source will not save you, because it's too generic. Emacs is not just a generic OS or a Lisp interpreter. It has huge amount of infrastructure around text editing specifically.

1letterunixname 3y ago

Rewriting != waterfall. Waterfall is a formal development model (5-10+ phases) that includes strict requirements, specifications, documentation, development, testing, and deployment. If you need to build an elevator control system or a SCADA system for a steelworks plant, then definitely go there. Rewrite is throwing away code (and downstream) and starting over. There comes a point in some projects when support costs exceed greenfield development costs, or prevent forward velocity due to tech debt. This is rare, but it happens. The problem of massive rewrites is they are too often not lean and equivalent to big design upfront (BDU) because they cannot deliver an MVP and they throw away features users depended on (but failed to questionnaire, focus group, or look at telemetry for). The other issue is massive rewrites too often are used an excuse to avoid refactoring and selective rewrites with the magical thinking that "new == better". Too often, this is reinventing the wheel that isn't much better for 2x the effort.

jeffrey-sherman 3y ago

Author of the post here - Lots of comments about the piece being a short opinion piece without evidence. Totally fair! If you’d like to learn more I have a podcast, https://www.neverrewrite.com/ https://www.neverrewrite.com/ Our latest episode is about Khan Academy’s 3.5 year migration from Python to Go. https://youtu.be/LqAkIkNs1Ws?si=INgD-QVDhL8INH8e https://youtu.be/LqAkIkNs1Ws?si=INgD-QVDhL8INH8e

zem 3y ago

rewrites (successful ones) are some of the most fun I've had in my entire career! it's incredibly satisfying to already have all the requirements and end user features crystallized and battle tested, and have your sole job be to figure out a better way to express them in designs, algorithms and code.

retrocryptid 3y ago

Sherman does not appear to understand what waterfall is or why people rewrite code. Waterfall specifies that phases of a single implementation are performed one after the other, completing one before the next phase begins. The classic example lists this as "design->coding->testing". What Sherman is describing is "iterative design," where you build something, put it in front of customers, then make changes suggested by customers (or maybe there was a list of features you wanted to add yourself.) Throwing away code is okay if it leads to "better" code where better means faster, easier to fix, smaller code size, etc. Often you do not know what the issues with your code are until you write it. Writing code for "experimental" purposes is decidedly iterative, not waterfall.

lasereyes136 3y ago

I have never seen a rewrite where there were no new features added to the software. Maybe the is part of the message, don't add features until you hhave replaced the old system. That sounds like a good idea but almost no one approaches it like that, which is one of the reasons rewrites often fail.