10 ms·

Lessons Learned from Software Rewrites

196 points by amartinsn 10y ago

gbacon 10y ago
But the siren song of the from-scratch rewrite is so difficult to resist.
- karmajunkie 10y ago
  I think of this as a saner version of the from-scratch rewrite, personally.
cottonseed 10y ago
When I hear about rewrites, I always think about this comment: https://news.ycombinator.com/item?id=11554288 https://news.ycombinator.com/item?id=11554288
- iammyIP 10y ago
  I expected some wisdom, but that comment basically says that a rewrite either isn't needed because programmers are already excellent in the first run, or the rewrite will end with a similar shit code in paraphrase, because programmers have learnt nothing. Also it takes time, which costs money...
  - hinkley 10y ago
    This is probably going to sound weird, possibly cheesy, but cottonseed got something out of that post, possibly more than I intended to convey, so I hesitate to follow up for fear of taking that away. What I think I was getting at then, and certainly believe now, is that some people grow with the code, learn from many of their mistakes, and do the long quiet job of expunging their worst errors as they go. Their code is the best proof that they could accomplish a rewrite, because it already is a rewrite. They just didn't need to stop the world to do it, or at least not all at once. Perhaps a month here and a couple weeks there. For the rest of the time it has been like the Ship of Theseus. All of the parts are new, and yet it is the same ship. Meanwhile the rewrite guys are continuing with their mess and hoping they get a do-over, putting off things hoping for Some Day. And on that magical day, all of the bad habits of the entire team (management included) will correct themselves overnight. Bad habits are too hard to break. The best of us know that you don't abstain from bad habits, you crowd them out with new, better ones. But that takes time, practice and determination. If you want to condense it down to nothing new under the sun, that's fair, and I would offer that I wanted to believe the rewrite folks were onto something, but at the end of the day, breaking down the problems and using relentless refactoring seems to be the only thing that works, without changing the definition of success to achieve victory.
    - Terr_ 10y ago
      And like the Ship of Theseus, it help the shipwright a lot when the guy who owns it is rich enough to check everything thoroughly before going out to sea. :p
    - animal531 10y ago
      Additionally, as time passes you get new people working on the system that does not understand it as well as the original designers/developers, simply because they already have the system and their knowledge becomes pigeon holed inside of it. They don't have the original scope. And then to top it off; if you did rewrite the system you not only need to reverse engineer N years of code, but also N years of bug fixes and random miscellaneous snippets that made it into the system and that no one who still works at the company understands the purpose of any longer.
    - iammyIP 10y ago
      edit: Yes, no problem, i understood now what you meant, crdoconnor put me to it.
  - crdoconnor 10y ago
    No it isn't. It says that good programmers are doing mini rewrites all the time negating the need for a big bang rewrite whereas bad programmers are probably just piling the debt up while asking for permission to rewrite and will do once again when they rewrite. There's some truth to this I think. Big bang rewrites are best avoided where possible and there's plenty of ways you can incrementalize refactoring that don't occur to most people.
    - iammyIP 10y ago
      I think you're right and i misread it. So if the mini-rewrites in silence equal relentless refactoring, that might equal a rewrite nobody ever observed, excellent.
- deleted 10y ago
  [deleted]
hcarvalhoalves 10y ago
Sometimes a better approach is to wrap the system you want to deprecate rather than have a parallel implementation. First, because you might not fully comprehend the current implementation to replicate it. Second, because in real life often there's not enough time, budget and coordination between teams to migrate old clients. Since their main requirement was having a REST API for new clients, and apparently the old implementation worked fine, they could have implemented a EJB -> HTTP layer on top, and have new features talk straight to the DB. Eventually, they replicate all the features talking straight to the DB (w/ the added benefit they can test against the codepath going thru the old system). Finally, death by phagocytosis.
- allendoerfer 10y ago
  I think the problem was, that they had to implement new features for the clients only supporting the old version, too. At this point you are left with a HTTP wrapper around your old software, which does not qualify as a rewrite.
  - zigzigzag 10y ago
    Well, yeah. It sounded like an HTTP wrapper is all they actually needed. Or in fact any well specified wire protocol with lots of implementations. It could have been protocol buffers over raw sockets also.
- gpderetta 10y ago
  Unfortunately that doesn't work either. I have seen libraries with 5 (five!!) levels of wrapping because each developer that wrote a wrapping level didn't understand the layer below/thought it could do a better job with the interface/was prematurely generalizing.
lgleason 10y ago
Strangler pattern FTW, well that and never have two systems share the same database. Always have one source of truth.
- vinchuco 10y ago
  TIL: >The strangler pattern is a software architecture pattern where a legacy system or feature is strangled with new code. >Make something new that obsoletes a small percentage of something old. Put them live together. Rinse and repeat. Strangler Applications http://paulhammant.com/2013/07/14/legacy-application-strangulation-case-studies/ http://paulhammant.com/2013/07/14/legacy-application-strangu...
- Zardoz84 10y ago
  Could be worse. Imagine a situation where you have to work with an old Java code base, where there is a lot work arounds that had sense ten years ago but not any more, and you are using obsolete library versions like Hibernate 2. Add that you can find similar functionality made on totally different ways. Add that you can find abuse of some programming patterns. Add that you have your own XML based framework where you end doing XML based programming, and not sane way of debugging it except doing the equivalent of debug with "print("CACA"); return -1". Add that for some operations, the java code must do the same stuff that does a heavy client wrote on Visual Basic 6 that works directly against the same database. Finally add that the database design isn't the best (BNF form ? What are you talking? )and that you are stuck on using a internal tool that does the same thing that Liquid base, but worse, and it's based on some old abandoned Apache libs that no body knows well how are working or how we modify. At least, we know that we have a problem and where are trying to fix it. And this is my diary work. PD: I have a friend that lives on a worse situation, where they have an application based on Java 1.5 applets, with bad or not existent documentation, and nobody knows how works internally.
  - dasil003 10y ago
    Are you also Brazilian like the OA? I ask because I know diário means daily in portuguese, but in English "diary" means a personal journal and doesn't make sense in this context.
Terr_ 10y ago
I've been dealing with a corporate ERP system for some time now, and -- whether rewrite or refactor -- the biggest challenge has been that nobody is willing to assert how it ought to work. At best, it's "reverse engineer it, tell me what it does, and then I'll tell you what feature I'd like changed."
- rtpg 10y ago
  This is a pretty eternal problem in software development. I now think that it is our jobs as developers to properly identify how things should work. We're often not experts in the domain we're working on, but by asking questions about a user's workflow on a more holistic level, we probably have more insight into what workflow would be the most productive from a systems standpoint. Of course, this is a back-and-forth. Sometimes users aren't sure of what they want. And sometimes we're not sure of what is the "best" system for what the users want. Instead of asking the user where the button should be, ask them when they press the button.
  - blowski 10y ago
    Behavioural driven development (BDD) encourages this process. The developers flesh out specific examples with the business people using readable language, and that becomes user acceptance criteria. It's no panacea, but it helps because it tries to prevent the business people dictating tech specifics, which the developers take too literally. For example, a business person says "when I press the submit button the results must be stored in MySQL". What they really mean is that whatever they entered into a form must be accessible later, but the developers now believe they must use MySQL and have submit buttons. By focusing on the "When I complete the form with data x, when I look at the form again later, then I see data x" you've moved the conversation to the business requirements and away from the perceived technical requirements.
    - ben_jones 10y ago
      "And I need you to write it in PHP because it's cheaper" (not hating on modern PHP, just its reputation and 'outsourceability')
Ace17 10y ago
Joel Spolsky has a very nice writeup about the subject, "Things You Should Never Do, part 1": http://www.joelonsoftware.com/articles/fog0000000069.html http://www.joelonsoftware.com/articles/fog0000000069.html
- 35bge57dtjku 10y ago
  From the guy who had a new language invented instead of re-writing something...
  - hinkley 10y ago
    I think Joel's articles were probably best when they convinced you of something you knew but hadn't articulated yet. If you aren't there yet I don't know that he always delivers. And if he's selling an idea you've already abandoned, he can leave you flat for sure. His strength is in introductions, not reconciliations,
- tonyedgecombe 10y ago
  Binary advice is rarely useful, you should never rewrite, you should have 100% test coverage, you should only hire graduates and on and on.
amelius 10y ago
> So once a functionality was implemented on the new stack, with all characterization tests passing, all we needed to do was to tell the router to use this new implementation, instead of the old one, and everyone consuming this endpoint would be automatically using the new implementation! How would that work without also migrating the database and while keeping the databases of the old and new implementations in perfect sync?
- biot 10y ago
  Migrate the DB in a backwards compatible way. So both version n and n-1 can use it. The next update (n+1) can then remove support for n-1.
gcb0 10y ago
to grasp how bad that system agree, globo is Brazilian fox news. and only got on the web 10 yrs ago... and they're already on version 4
emdeha 10y ago
Great article! It's interesting though, how did you convince management to remove features no one was using?
- tonyedgecombe 10y ago
  If no one was using them why do you need to convince management?
  - akerro 10y ago
    Because when trying to sell the application list of features is longer, which means we know what we're doing and are able to maintain it.
  - emdeha 10y ago
    Because management decisions ain't always driven by logic. The simplest reason may be attachment to the feature--the feature is of no use but the person who's invented it is scared by the possibility of removing it.
- di4na 10y ago
  From my PoV : Pull the plug and wait to see if someone yell. If noone yell... then noone need to know. If someone yell, be sure to put your own contact proeminently everywhere. So that your management do not know. And just roll it back
  - OpenDrapery 10y ago
    Problem with this is the feature that is used by some external actor once every 18 months. You'll pull the plug, and when 6 months go by, you'll think you're safe. Then when shit blows up, you'll have no context and spend 3 days tracking it back to the "pulled plug"
auvrw 10y ago
it's difficult to get a picture of all the nuances of this project, of course, from any quick read.. one thing that the article left me wondering a/b is the outline of "the new stack" which is "agnostic to any programming language". i mean, i'm not the biggest Java fan, but it's alright, and there are some pretty good RESTful frameworks https://www.playframework.com/ https://www.playframework.com/ while the strangler application (always a fowler link, always, in these types of articles) makes a ton of sense for making sanity out of incoming requests, i wonder if some effort could have been &/or was saved behind the transport layer by reusing (possibly with refactor) the existing Java implementation? whether doing so would make sense would be really situation dependent, of course, and an "outside" perspective from a quick article isn't going to have enough information to say anything conclusive. the (lazy) feel is that reuse is better than duplication of effort. ------------ aaaaanyway, super-good article. this is of of the kinds of things that's important to think about as a workaday web dev.
- amartinsn 10y ago
  Hey, thanks for your reply. Your line of thought is very similar to what we had. We considered refactoring the existing Java implementation. It used a really old Jersey version, with too many overengineered customizations tightened to that version. We tried to upgrade it and get rid of some of those customizations, but the system was so fragile that other parts of the system stopped working, which made that a really difficult road to drive. We wanted to simplify the design. We chose Scala and Finagle for the new stack.
faragon 10y ago
In my opinion, asking for "removing unused features" is one of the consequences of full rewrites. Also, a reason for having to maintain "dual systems" for years. It is not infrequent having "dual systems" and having to deprecate the "new", just because after e.g. 10 years you can not get rid of the old, or even worse: the "new" performing worse than the old (e.g. old simple centralized system working much faster on new hardware vs "new" pseudo decentralized mess that can not scale as good because inter-node communication overhead). BTW, every time I read Martin Fowler quoted in a post as full rewrite justification for the "faith jump", I get scared. I don't know why the poster didn't implemented the new API, using it for the new features, and eventually, rewrite old API commands (one by one, not all at once) calling the new API, so in the end, new API would remain, and old API would be just some wrapped calls to the new API.
- amartinsn 10y ago
  That was exactly our approach, for the old HTTP API, we kept the same contracts (URIs, response body, etc.) For the EJB one, as only one remaining client was using it, we deprioritized that, but we were going to pick the features they were using, and make the Java client point to the new API implementation, also keeping the same contract.
ensiferum 10y ago
This isn't a rewrite in the traditional "big bang" sense of a rewrite but normal software evolution and maintenance. Anyway sounds like their approach was quite reasonable, just a bad title for the blog entry.
- amartinsn 10y ago
  It's an incremental rewrite. Rewrite one feature, switch everyone to use it, then destroy the old implementation. Do the same for all other features.
- amartinsn 10y ago
  It's an incremental rewrite. Rewrite one feature, switch everyone to use it, then destroy the old implementation. Do the same for all other features.

196 points by amartinsn 10y ago

gbacon 10y ago

But the siren song of the from-scratch rewrite is so difficult to resist.

karmajunkie 10y ago
I think of this as a saner version of the from-scratch rewrite, personally.

cottonseed 10y ago

When I hear about rewrites, I always think about this comment: https://news.ycombinator.com/item?id=11554288 https://news.ycombinator.com/item?id=11554288

iammyIP 10y ago
I expected some wisdom, but that comment basically says that a rewrite either isn't needed because programmers are already excellent in the first run, or the rewrite will end with a similar shit code in paraphrase, because programmers have learnt nothing. Also it takes time, which costs money...
- hinkley 10y ago
  This is probably going to sound weird, possibly cheesy, but cottonseed got something out of that post, possibly more than I intended to convey, so I hesitate to follow up for fear of taking that away. What I think I was getting at then, and certainly believe now, is that some people grow with the code, learn from many of their mistakes, and do the long quiet job of expunging their worst errors as they go. Their code is the best proof that they could accomplish a rewrite, because it already is a rewrite. They just didn't need to stop the world to do it, or at least not all at once. Perhaps a month here and a couple weeks there. For the rest of the time it has been like the Ship of Theseus. All of the parts are new, and yet it is the same ship. Meanwhile the rewrite guys are continuing with their mess and hoping they get a do-over, putting off things hoping for Some Day. And on that magical day, all of the bad habits of the entire team (management included) will correct themselves overnight. Bad habits are too hard to break. The best of us know that you don't abstain from bad habits, you crowd them out with new, better ones. But that takes time, practice and determination. If you want to condense it down to nothing new under the sun, that's fair, and I would offer that I wanted to believe the rewrite folks were onto something, but at the end of the day, breaking down the problems and using relentless refactoring seems to be the only thing that works, without changing the definition of success to achieve victory.
  - Terr_ 10y ago
    And like the Ship of Theseus, it help the shipwright a lot when the guy who owns it is rich enough to check everything thoroughly before going out to sea. :p
  - animal531 10y ago
    Additionally, as time passes you get new people working on the system that does not understand it as well as the original designers/developers, simply because they already have the system and their knowledge becomes pigeon holed inside of it. They don't have the original scope. And then to top it off; if you did rewrite the system you not only need to reverse engineer N years of code, but also N years of bug fixes and random miscellaneous snippets that made it into the system and that no one who still works at the company understands the purpose of any longer.
  - iammyIP 10y ago
    edit: Yes, no problem, i understood now what you meant, crdoconnor put me to it.
- crdoconnor 10y ago
  No it isn't. It says that good programmers are doing mini rewrites all the time negating the need for a big bang rewrite whereas bad programmers are probably just piling the debt up while asking for permission to rewrite and will do once again when they rewrite. There's some truth to this I think. Big bang rewrites are best avoided where possible and there's plenty of ways you can incrementalize refactoring that don't occur to most people.
  - iammyIP 10y ago
    I think you're right and i misread it. So if the mini-rewrites in silence equal relentless refactoring, that might equal a rewrite nobody ever observed, excellent.
deleted 10y ago
[deleted]

hcarvalhoalves 10y ago

Sometimes a better approach is to wrap the system you want to deprecate rather than have a parallel implementation. First, because you might not fully comprehend the current implementation to replicate it. Second, because in real life often there's not enough time, budget and coordination between teams to migrate old clients. Since their main requirement was having a REST API for new clients, and apparently the old implementation worked fine, they could have implemented a EJB -> HTTP layer on top, and have new features talk straight to the DB. Eventually, they replicate all the features talking straight to the DB (w/ the added benefit they can test against the codepath going thru the old system). Finally, death by phagocytosis.

allendoerfer 10y ago
I think the problem was, that they had to implement new features for the clients only supporting the old version, too. At this point you are left with a HTTP wrapper around your old software, which does not qualify as a rewrite.
- zigzigzag 10y ago
  Well, yeah. It sounded like an HTTP wrapper is all they actually needed. Or in fact any well specified wire protocol with lots of implementations. It could have been protocol buffers over raw sockets also.
gpderetta 10y ago
Unfortunately that doesn't work either. I have seen libraries with 5 (five!!) levels of wrapping because each developer that wrote a wrapping level didn't understand the layer below/thought it could do a better job with the interface/was prematurely generalizing.

lgleason 10y ago

Strangler pattern FTW, well that and never have two systems share the same database. Always have one source of truth.

vinchuco 10y ago
TIL: >The strangler pattern is a software architecture pattern where a legacy system or feature is strangled with new code. >Make something new that obsoletes a small percentage of something old. Put them live together. Rinse and repeat. Strangler Applications http://paulhammant.com/2013/07/14/legacy-application-strangulation-case-studies/ http://paulhammant.com/2013/07/14/legacy-application-strangu...
Zardoz84 10y ago
Could be worse. Imagine a situation where you have to work with an old Java code base, where there is a lot work arounds that had sense ten years ago but not any more, and you are using obsolete library versions like Hibernate 2. Add that you can find similar functionality made on totally different ways. Add that you can find abuse of some programming patterns. Add that you have your own XML based framework where you end doing XML based programming, and not sane way of debugging it except doing the equivalent of debug with "print("CACA"); return -1". Add that for some operations, the java code must do the same stuff that does a heavy client wrote on Visual Basic 6 that works directly against the same database. Finally add that the database design isn't the best (BNF form ? What are you talking? )and that you are stuck on using a internal tool that does the same thing that Liquid base, but worse, and it's based on some old abandoned Apache libs that no body knows well how are working or how we modify. At least, we know that we have a problem and where are trying to fix it. And this is my diary work. PD: I have a friend that lives on a worse situation, where they have an application based on Java 1.5 applets, with bad or not existent documentation, and nobody knows how works internally.
- dasil003 10y ago
  Are you also Brazilian like the OA? I ask because I know diário means daily in portuguese, but in English "diary" means a personal journal and doesn't make sense in this context.

Terr_ 10y ago

I've been dealing with a corporate ERP system for some time now, and -- whether rewrite or refactor -- the biggest challenge has been that nobody is willing to assert how it ought to work. At best, it's "reverse engineer it, tell me what it does, and then I'll tell you what feature I'd like changed."

rtpg 10y ago
This is a pretty eternal problem in software development. I now think that it is our jobs as developers to properly identify how things should work. We're often not experts in the domain we're working on, but by asking questions about a user's workflow on a more holistic level, we probably have more insight into what workflow would be the most productive from a systems standpoint. Of course, this is a back-and-forth. Sometimes users aren't sure of what they want. And sometimes we're not sure of what is the "best" system for what the users want. Instead of asking the user where the button should be, ask them when they press the button.
- blowski 10y ago
  Behavioural driven development (BDD) encourages this process. The developers flesh out specific examples with the business people using readable language, and that becomes user acceptance criteria. It's no panacea, but it helps because it tries to prevent the business people dictating tech specifics, which the developers take too literally. For example, a business person says "when I press the submit button the results must be stored in MySQL". What they really mean is that whatever they entered into a form must be accessible later, but the developers now believe they must use MySQL and have submit buttons. By focusing on the "When I complete the form with data x, when I look at the form again later, then I see data x" you've moved the conversation to the business requirements and away from the perceived technical requirements.
  - ben_jones 10y ago
    "And I need you to write it in PHP because it's cheaper" (not hating on modern PHP, just its reputation and 'outsourceability')

Ace17 10y ago

Joel Spolsky has a very nice writeup about the subject, "Things You Should Never Do, part 1": http://www.joelonsoftware.com/articles/fog0000000069.html http://www.joelonsoftware.com/articles/fog0000000069.html

35bge57dtjku 10y ago
From the guy who had a new language invented instead of re-writing something...
- hinkley 10y ago
  I think Joel's articles were probably best when they convinced you of something you knew but hadn't articulated yet. If you aren't there yet I don't know that he always delivers. And if he's selling an idea you've already abandoned, he can leave you flat for sure. His strength is in introductions, not reconciliations,
tonyedgecombe 10y ago
Binary advice is rarely useful, you should never rewrite, you should have 100% test coverage, you should only hire graduates and on and on.

amelius 10y ago

> So once a functionality was implemented on the new stack, with all characterization tests passing, all we needed to do was to tell the router to use this new implementation, instead of the old one, and everyone consuming this endpoint would be automatically using the new implementation! How would that work without also migrating the database and while keeping the databases of the old and new implementations in perfect sync?

biot 10y ago
Migrate the DB in a backwards compatible way. So both version n and n-1 can use it. The next update (n+1) can then remove support for n-1.

gcb0 10y ago

to grasp how bad that system agree, globo is Brazilian fox news. and only got on the web 10 yrs ago... and they're already on version 4

emdeha 10y ago

Great article! It's interesting though, how did you convince management to remove features no one was using?

tonyedgecombe 10y ago
If no one was using them why do you need to convince management?
- akerro 10y ago
  Because when trying to sell the application list of features is longer, which means we know what we're doing and are able to maintain it.
- emdeha 10y ago
  Because management decisions ain't always driven by logic. The simplest reason may be attachment to the feature--the feature is of no use but the person who's invented it is scared by the possibility of removing it.
di4na 10y ago
From my PoV : Pull the plug and wait to see if someone yell. If noone yell... then noone need to know. If someone yell, be sure to put your own contact proeminently everywhere. So that your management do not know. And just roll it back
- OpenDrapery 10y ago
  Problem with this is the feature that is used by some external actor once every 18 months. You'll pull the plug, and when 6 months go by, you'll think you're safe. Then when shit blows up, you'll have no context and spend 3 days tracking it back to the "pulled plug"

auvrw 10y ago

it's difficult to get a picture of all the nuances of this project, of course, from any quick read.. one thing that the article left me wondering a/b is the outline of "the new stack" which is "agnostic to any programming language". i mean, i'm not the biggest Java fan, but it's alright, and there are some pretty good RESTful frameworks https://www.playframework.com/ https://www.playframework.com/ while the strangler application (always a fowler link, always, in these types of articles) makes a ton of sense for making sanity out of incoming requests, i wonder if some effort could have been &/or was saved behind the transport layer by reusing (possibly with refactor) the existing Java implementation? whether doing so would make sense would be really situation dependent, of course, and an "outside" perspective from a quick article isn't going to have enough information to say anything conclusive. the (lazy) feel is that reuse is better than duplication of effort. ------------ aaaaanyway, super-good article. this is of of the kinds of things that's important to think about as a workaday web dev.

amartinsn 10y ago
Hey, thanks for your reply. Your line of thought is very similar to what we had. We considered refactoring the existing Java implementation. It used a really old Jersey version, with too many overengineered customizations tightened to that version. We tried to upgrade it and get rid of some of those customizations, but the system was so fragile that other parts of the system stopped working, which made that a really difficult road to drive. We wanted to simplify the design. We chose Scala and Finagle for the new stack.

faragon 10y ago

In my opinion, asking for "removing unused features" is one of the consequences of full rewrites. Also, a reason for having to maintain "dual systems" for years. It is not infrequent having "dual systems" and having to deprecate the "new", just because after e.g. 10 years you can not get rid of the old, or even worse: the "new" performing worse than the old (e.g. old simple centralized system working much faster on new hardware vs "new" pseudo decentralized mess that can not scale as good because inter-node communication overhead). BTW, every time I read Martin Fowler quoted in a post as full rewrite justification for the "faith jump", I get scared. I don't know why the poster didn't implemented the new API, using it for the new features, and eventually, rewrite old API commands (one by one, not all at once) calling the new API, so in the end, new API would remain, and old API would be just some wrapped calls to the new API.

amartinsn 10y ago
That was exactly our approach, for the old HTTP API, we kept the same contracts (URIs, response body, etc.) For the EJB one, as only one remaining client was using it, we deprioritized that, but we were going to pick the features they were using, and make the Java client point to the new API implementation, also keeping the same contract.

ensiferum 10y ago

This isn't a rewrite in the traditional "big bang" sense of a rewrite but normal software evolution and maintenance. Anyway sounds like their approach was quite reasonable, just a bad title for the blog entry.

amartinsn 10y ago
It's an incremental rewrite. Rewrite one feature, switch everyone to use it, then destroy the old implementation. Do the same for all other features.
amartinsn 10y ago
It's an incremental rewrite. Rewrite one feature, switch everyone to use it, then destroy the old implementation. Do the same for all other features.