7 ms·

Rewriting from scratch

59 points by callum85 13y ago

mbesto 13y ago
So I know re-writing a technical system from scratch is always consider a big no-no. A couple of observations to counter that argument: 1. Re-writing code is the Innovator's Solution to the Innovator's Dilemma.[0] 2. Mozilla wouldn't exist today if the Netscape team didn't decide to rewrite their browser engine.[1] 3. Isn't Basecamp New a rewrite?[2] [0]http://www.amazon.com/The-Innovators-Solution-Sustaining-Successful/dp/1578518520 http://www.amazon.com/The-Innovators-Solution-Sustaining-Suc... [1]http://en.wikipedia.org/wiki/Mozilla#History http://en.wikipedia.org/wiki/Mozilla#History [2]http://basecamp.com/new http://basecamp.com/new Thoughts?
- casual_slacker 13y ago
  I've spent two jobs rewriting software. The two main reasons for it: 1) the original was a prototype; 2) the requirements have changed so much that the original architecture doesn't make sense. If it's either of these situations, then a rewrite can make sense. Contrast that with developers wanting to rewrite because of vague frustrations or poorly-defined technical debt. Without a strict definition of the problem, a rewrite is doomed to repeat the same shortcoming as the original. I've seen the most success happen on teams which can figure out how to rewrite the app one module at a time, while keeping the existing parts functional. This reduces the risk, and allows you to compare the new effort against old code more directly.
- lmm 13y ago
  1. Without buying the book that sounds like a meaningless buzzword. 2. Mozilla wouldn't exist today if the Netscape team didn't destroy their company by deciding to rewrite their browser engine, no. Whether we'd be worse or better off is an open question - a living commercial netscape could have kept the browser wars and associated innovation alive longer. On the opensource side maybe the better-architectured KHTML would have gathered more mindshare and funding and evolved into something like today's WebKit faster without the distraction of gecko.
  - mbesto 13y ago
    > that sounds like a meaningless buzzword. I can assure you they are not meaningless buzzwords. I'd recommend reading the book before commenting since your reply adds nothing to the discussion. > 2. Mozilla wouldn't exist today if the Netscape team didn't destroy their company Any source to back up the claim that the Netscape team maliciously destroyed the company? What would be their motive? From everything I've read the codebase was an absolute mess, hence the motivation for a re-write.
    - lmm 13y ago
      >I can assure you they are not meaningless buzzwords. I'd recommend reading the book before commenting since your reply adds nothing to the discussion. Perhaps naively, I assume you hadn't realised you were talking in what sounded like meaningless buzzwords. > Any source to back up the claim that the Netscape team maliciously destroyed the company? I never claimed malice. I'm sure the decision to rewrite the code was made with the best of intentions. It still destroyed the company.
    - rwallace 13y ago
      The code being a mess usually calls for refactoring not rewriting. As I understand it, the showstopper that made a rewrite necessary was that they had made the mistake (understandable since they were in a desperate hurry but mistake nonetheless) of using proprietary libraries.
      - 13b9f227ecf0 13y ago
        Technical debt can accrue to the point where a piece of software is "bankrupt." Any successful refactoring amounts to extracting a subset and redoing everything else, which can practically amount to a rewrite.
  - ceejayoz 13y ago
    Would Netscape have survived if they hadn't rewritten the browser engine? I'd've thought they were fucked either way.
- jerf 13y ago
  It isn't always a big no-no, it's just that it's a known rationality failure on software engineers part, as they consider the benefits of a rewrite but brutally underestimate the costs. Also, the standard and IMHO valid riposte to the Mozilla rewrite is to observe that while it may all have turned out alright for the public in the end, Netscape didn't survive it. If you are in a business context, Netscape is still a warning about failure, not an example of success!
- everyone 13y ago
  "re-writing a technical system from scratch is always consider a big no-no" Why is this? (Apart from the fact that it takes time) Also what do you mean by "technical system" ?
  - stonemetal 13y ago
    There should be a big for a business stuck on there. The primary reasons are loss of quality, practically by definition new code is less well tested than old code; loss of time, in the rewrite you spend time re-implementing features your system already had which doesn't provide new value for end users(it is hard to sell an upgrade of no new features.)
  - mbesto 13y ago
    I guess I considered it common knowledge. I would say these two points illustrate my basis: 1. http://www.joelonsoftware.com/articles/fog0000000069.html http://www.joelonsoftware.com/articles/fog0000000069.html 2. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working simple system. http://en.wikipedia.org/wiki/Systemantics#Elementary_Systems_functions http://en.wikipedia.org/wiki/Systemantics#Elementary_Systems... > "technical system" I could have said software, but software can't be very complex or very simple. I probably should have said "complex software" instead.
    - everyone 13y ago
      "A complex system designed from scratch never works" This is based on the work of John Gall who I've never read and cannot comment on. However he was a pediatrician who has never gone near software, I wonder how relevant the personal anecdotes and experience his books draw their conclusions from are. Also what is his definition of complex relating to the above quote? I'm assuming he could plot a graph with complexity along one axis and utility of starting from scratch along the other. At some point it might cross a certain threshold. Also is there much in software development that could really be considered 'complex'? or, how does complexity scale in software? Software could be described as "1000 layers of abstraction" if you are, say, re-writing a program in a high level language you are not starting your system from scratch. All the lower level elements of how it is assembled, run on the computer + all the protocols/ libraries drivers etc. used in the system are not changing.
- InclinedPlane 13y ago
  There are right ways and wrong ways to do a rewrite. Joel Spolsky says you should never do it but he's flat wrong, mostly, he's right in that it's almost never done correctly. Most rewrites are undertaken too lightly and with too much naivete. They are approached as projects with a lot of major unspoken assumptions. Often assumptions that are counter-factual. Assumptions such as: at code complete we will have a system that is as refined and defect free as a well-worn system that has been in real-world use for years. Or, there will be no problem with forcing every user to use the new system because it will be in every way objectively superior to the old system starting on day 1. Or, taking on the work of a rewrite won't cause a serious shortage of manpower needed to maintain the old system (or vice versa, the rewrite won't face a serious shortage due to the needs of maintaining the old system). And so on. Yet such obvious fallacies remain as latent assumptions behind the majority of rewrites. There are different ways to do rewrites effectively and none of them are easy. One way is to build a parallel product using an entirely separate team, and having that product compete against your older product directly in the marketplace. This is obviously an expensive route, which is why a lot of companies shy away from it, but sometimes it's the best way to go about things. It gives a chance for a product to mature before it replaces the original. Microsoft did this with Windows, having two parallel teams work on different kernels (NT and 9x) and it took over half a decade for the "rewrite" to win out. This is even more astounding when you consider that originally Microsoft had intended for Windows 95 to be simply Windows 4.0 based on the NT kernel. A goal they didn't really achieve until Windows XP was released 6 years later. An alternate way to do a rewrite is through refactoring style techniques. Start building the new system inside the old and slowly deprecate and migrate components piece by piece until the old system is entirely gone. This is a very effective way of doing things and has the advantage of being able to scale relative to the amount of development effort available but certainly has its fair share of limitations and downsides (which I won't enumerate). As far as the Netscape example, they ended up coding themselves into annihilation, as their constant rewrites and architectural buffoonery led them to create a very unstable and clunky browser that was markedly inferior to IE 4 when it hit the market, and they just were not able to respond with any degree of innovation software wise. Also, my understanding is that the seed code for the mozilla project was a horrendous mess that didn't even represent a functional browser at the time and required extensive effort to get to that point.
_pmf_ 13y ago
> Our old dashboards were at a local maximum, making it hard to iterate. Introducing elements of a variable Direct Debit interface would have added complexity, with no benefit until the whole new interface was ready. > For a fundamental change in UI that lack of iteration wasn't acceptable. Starting from scratch with an early beta let us to collect feedback on the new interface immediately. Our speed of iteration was also increased as we didn't have to worry about breaking an existing interface. I would have said "I've created an unmaintainable mess.", but from now on, I'll be using your nice euphemisms. "Local maximum" ... yeah, sounds proactive and empowering. I'll have to work on delivering it with a straight face, though.
ninjakeyboard 13y ago
There is a huge body of experience out there that does state this is insane and generally crashes and burns and fails horribly. Good on ya to go against the odds and win.
beat 13y ago
Consider the Big Ball of Mud: http://www.laputan.org/mud/ http://www.laputan.org/mud/ This can be either a pattern or an antipattern, depending on your perspective. I tend to think of it as an antipattern. Rewrites are often driven by difficulty in refactoring more than fundamental flaws, but the refactoring problems are caused by coupling between layers/systems. The problem with rewriting to avoid a refactoring is that you're most likely going to wind up with a shiny new Ball of Mud.
Siecje 13y ago
In the article you didn't need to do a re-write of the entire site, just a re-design of the UI. You needed to add functionality but you could have left the old working functionality. Another case for a re-right is when the project uses old technology that is hard to maintain with the people you have. Re-writing the system in a framework that your team understands will take less time than trying to understand another framework.
txutxu 13y ago
In my opinion, It's ok to rewrite, if you finish, on time, and it works as expected. You can see all kind of "rewrites" some with more luck, and others with worse. This one seems successful (I talk without numbers). I've a personal pet project I'm always rewriting (from scratch). But, it's the single project in my life I can't validate the result, I've try many approaches. There is a few corner cases, were the only exit is to use unclean workarounds by language limitation... and even after a few renames and even language evolution, I think I'll retire "still rewriting from scratch" if I feel it can be better, and I've the freedom/budget/time to do it.
Chanel_Bunnell 13y ago
We're actually starting a rewrite of an old Windows product next month. It was first released in 2005, and it simply doesn't stand up anymore -- looks like a Windows 3.1 product trying to run on Windows 8 -- but it still sells. It was our first product, which meant a terrible code base, shocking feature creep, and a hundred and one mistakes made at every level. We considered using some of the old code, but the logic is so entwined with the GUI that it's simply not possible. First product sins! We're using the rewrite as an opportunity to re-brand it as a new product, removing 40% of the old features, and adding just a tiny number of new ones. Our hope is that the rewrite will keep sales coming in for a few more years. The rewrite should be made easier by the fact that the 40% of features being removed are the most complex (and worthless) features -- the price of listening to the small number of techie early adopters, who we've now learned to ignore. Really looking forward to rolling out the new version.

59 points by callum85 13y ago

So I know re-writing a technical system from scratch is always consider a big no-no. A couple of observations to counter that argument: 1. Re-writing code is the Innovator's Solution to the Innovator's Dilemma.[0] 2. Mozilla wouldn't exist today if the Netscape team didn't decide to rewrite their browser engine.[1] 3. Isn't Basecamp New a rewrite?[2] [0]http://www.amazon.com/The-Innovators-Solution-Sustaining-Successful/dp/1578518520 http://www.amazon.com/The-Innovators-Solution-Sustaining-Suc... [1]http://en.wikipedia.org/wiki/Mozilla#History http://en.wikipedia.org/wiki/Mozilla#History [2]http://basecamp.com/new http://basecamp.com/new Thoughts?

casual_slacker 13y ago
I've spent two jobs rewriting software. The two main reasons for it: 1) the original was a prototype; 2) the requirements have changed so much that the original architecture doesn't make sense. If it's either of these situations, then a rewrite can make sense. Contrast that with developers wanting to rewrite because of vague frustrations or poorly-defined technical debt. Without a strict definition of the problem, a rewrite is doomed to repeat the same shortcoming as the original. I've seen the most success happen on teams which can figure out how to rewrite the app one module at a time, while keeping the existing parts functional. This reduces the risk, and allows you to compare the new effort against old code more directly.
lmm 13y ago
1. Without buying the book that sounds like a meaningless buzzword. 2. Mozilla wouldn't exist today if the Netscape team didn't destroy their company by deciding to rewrite their browser engine, no. Whether we'd be worse or better off is an open question - a living commercial netscape could have kept the browser wars and associated innovation alive longer. On the opensource side maybe the better-architectured KHTML would have gathered more mindshare and funding and evolved into something like today's WebKit faster without the distraction of gecko.
- mbesto 13y ago
  > that sounds like a meaningless buzzword. I can assure you they are not meaningless buzzwords. I'd recommend reading the book before commenting since your reply adds nothing to the discussion. > 2. Mozilla wouldn't exist today if the Netscape team didn't destroy their company Any source to back up the claim that the Netscape team maliciously destroyed the company? What would be their motive? From everything I've read the codebase was an absolute mess, hence the motivation for a re-write.
  - lmm 13y ago
    >I can assure you they are not meaningless buzzwords. I'd recommend reading the book before commenting since your reply adds nothing to the discussion. Perhaps naively, I assume you hadn't realised you were talking in what sounded like meaningless buzzwords. > Any source to back up the claim that the Netscape team maliciously destroyed the company? I never claimed malice. I'm sure the decision to rewrite the code was made with the best of intentions. It still destroyed the company.
  - rwallace 13y ago
    The code being a mess usually calls for refactoring not rewriting. As I understand it, the showstopper that made a rewrite necessary was that they had made the mistake (understandable since they were in a desperate hurry but mistake nonetheless) of using proprietary libraries.
    - 13b9f227ecf0 13y ago
      Technical debt can accrue to the point where a piece of software is "bankrupt." Any successful refactoring amounts to extracting a subset and redoing everything else, which can practically amount to a rewrite.
- ceejayoz 13y ago
  Would Netscape have survived if they hadn't rewritten the browser engine? I'd've thought they were fucked either way.
jerf 13y ago
It isn't always a big no-no, it's just that it's a known rationality failure on software engineers part, as they consider the benefits of a rewrite but brutally underestimate the costs. Also, the standard and IMHO valid riposte to the Mozilla rewrite is to observe that while it may all have turned out alright for the public in the end, Netscape didn't survive it. If you are in a business context, Netscape is still a warning about failure, not an example of success!
everyone 13y ago
"re-writing a technical system from scratch is always consider a big no-no" Why is this? (Apart from the fact that it takes time) Also what do you mean by "technical system" ?
- stonemetal 13y ago
  There should be a big for a business stuck on there. The primary reasons are loss of quality, practically by definition new code is less well tested than old code; loss of time, in the rewrite you spend time re-implementing features your system already had which doesn't provide new value for end users(it is hard to sell an upgrade of no new features.)
- mbesto 13y ago
  I guess I considered it common knowledge. I would say these two points illustrate my basis: 1. http://www.joelonsoftware.com/articles/fog0000000069.html http://www.joelonsoftware.com/articles/fog0000000069.html 2. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working simple system. http://en.wikipedia.org/wiki/Systemantics#Elementary_Systems_functions http://en.wikipedia.org/wiki/Systemantics#Elementary_Systems... > "technical system" I could have said software, but software can't be very complex or very simple. I probably should have said "complex software" instead.
  - everyone 13y ago
    "A complex system designed from scratch never works" This is based on the work of John Gall who I've never read and cannot comment on. However he was a pediatrician who has never gone near software, I wonder how relevant the personal anecdotes and experience his books draw their conclusions from are. Also what is his definition of complex relating to the above quote? I'm assuming he could plot a graph with complexity along one axis and utility of starting from scratch along the other. At some point it might cross a certain threshold. Also is there much in software development that could really be considered 'complex'? or, how does complexity scale in software? Software could be described as "1000 layers of abstraction" if you are, say, re-writing a program in a high level language you are not starting your system from scratch. All the lower level elements of how it is assembled, run on the computer + all the protocols/ libraries drivers etc. used in the system are not changing.
InclinedPlane 13y ago
There are right ways and wrong ways to do a rewrite. Joel Spolsky says you should never do it but he's flat wrong, mostly, he's right in that it's almost never done correctly. Most rewrites are undertaken too lightly and with too much naivete. They are approached as projects with a lot of major unspoken assumptions. Often assumptions that are counter-factual. Assumptions such as: at code complete we will have a system that is as refined and defect free as a well-worn system that has been in real-world use for years. Or, there will be no problem with forcing every user to use the new system because it will be in every way objectively superior to the old system starting on day 1. Or, taking on the work of a rewrite won't cause a serious shortage of manpower needed to maintain the old system (or vice versa, the rewrite won't face a serious shortage due to the needs of maintaining the old system). And so on. Yet such obvious fallacies remain as latent assumptions behind the majority of rewrites. There are different ways to do rewrites effectively and none of them are easy. One way is to build a parallel product using an entirely separate team, and having that product compete against your older product directly in the marketplace. This is obviously an expensive route, which is why a lot of companies shy away from it, but sometimes it's the best way to go about things. It gives a chance for a product to mature before it replaces the original. Microsoft did this with Windows, having two parallel teams work on different kernels (NT and 9x) and it took over half a decade for the "rewrite" to win out. This is even more astounding when you consider that originally Microsoft had intended for Windows 95 to be simply Windows 4.0 based on the NT kernel. A goal they didn't really achieve until Windows XP was released 6 years later. An alternate way to do a rewrite is through refactoring style techniques. Start building the new system inside the old and slowly deprecate and migrate components piece by piece until the old system is entirely gone. This is a very effective way of doing things and has the advantage of being able to scale relative to the amount of development effort available but certainly has its fair share of limitations and downsides (which I won't enumerate). As far as the Netscape example, they ended up coding themselves into annihilation, as their constant rewrites and architectural buffoonery led them to create a very unstable and clunky browser that was markedly inferior to IE 4 when it hit the market, and they just were not able to respond with any degree of innovation software wise. Also, my understanding is that the seed code for the mozilla project was a horrendous mess that didn't even represent a functional browser at the time and required extensive effort to get to that point.

_pmf_ 13y ago

> Our old dashboards were at a local maximum, making it hard to iterate. Introducing elements of a variable Direct Debit interface would have added complexity, with no benefit until the whole new interface was ready. > For a fundamental change in UI that lack of iteration wasn't acceptable. Starting from scratch with an early beta let us to collect feedback on the new interface immediately. Our speed of iteration was also increased as we didn't have to worry about breaking an existing interface. I would have said "I've created an unmaintainable mess.", but from now on, I'll be using your nice euphemisms. "Local maximum" ... yeah, sounds proactive and empowering. I'll have to work on delivering it with a straight face, though.

ninjakeyboard 13y ago

There is a huge body of experience out there that does state this is insane and generally crashes and burns and fails horribly. Good on ya to go against the odds and win.

beat 13y ago

Consider the Big Ball of Mud: http://www.laputan.org/mud/ http://www.laputan.org/mud/ This can be either a pattern or an antipattern, depending on your perspective. I tend to think of it as an antipattern. Rewrites are often driven by difficulty in refactoring more than fundamental flaws, but the refactoring problems are caused by coupling between layers/systems. The problem with rewriting to avoid a refactoring is that you're most likely going to wind up with a shiny new Ball of Mud.

Siecje 13y ago

In the article you didn't need to do a re-write of the entire site, just a re-design of the UI. You needed to add functionality but you could have left the old working functionality. Another case for a re-right is when the project uses old technology that is hard to maintain with the people you have. Re-writing the system in a framework that your team understands will take less time than trying to understand another framework.

txutxu 13y ago

In my opinion, It's ok to rewrite, if you finish, on time, and it works as expected. You can see all kind of "rewrites" some with more luck, and others with worse. This one seems successful (I talk without numbers). I've a personal pet project I'm always rewriting (from scratch). But, it's the single project in my life I can't validate the result, I've try many approaches. There is a few corner cases, were the only exit is to use unclean workarounds by language limitation... and even after a few renames and even language evolution, I think I'll retire "still rewriting from scratch" if I feel it can be better, and I've the freedom/budget/time to do it.

Chanel_Bunnell 13y ago

We're actually starting a rewrite of an old Windows product next month. It was first released in 2005, and it simply doesn't stand up anymore -- looks like a Windows 3.1 product trying to run on Windows 8 -- but it still sells. It was our first product, which meant a terrible code base, shocking feature creep, and a hundred and one mistakes made at every level. We considered using some of the old code, but the logic is so entwined with the GUI that it's simply not possible. First product sins! We're using the rewrite as an opportunity to re-brand it as a new product, removing 40% of the old features, and adding just a tiny number of new ones. Our hope is that the rewrite will keep sales coming in for a few more years. The rewrite should be made easier by the fact that the 40% of features being removed are the most complex (and worthless) features -- the price of listening to the small number of techie early adopters, who we've now learned to ignore. Really looking forward to rolling out the new version.