5 ms·

Unfortunately this is a lesson that often has to be learned the hard way when you have a lead or team that lacks experience/wisdom. If you can survive that edu

by fallous 10y ago

Unfortunately this is a lesson that often has to be learned the hard way when you have a lead or team that lacks experience/wisdom. If you can survive that educational expense, you end up much better for it but it's painful nonetheless. Aesthetics are never a legitimate reason to replace a functioning system with something else. I do not care how it offends the eye, the mind, the ear, or any other sensation you have... if it does not have a fundamental functional problem your best option is to maintain or plan an incremental rolling replacement that is tightly scoped at each step. For large systems that do present functional problems, identify the egregious offenders in that system, abstract the entities/behaviors most responsible, and replace-in-place in a Ship of Theseus fashion. Despite how familiar you believe you are with a monolithic or complex inter-dependent system, you will discover how much you don't actually understand about how it works as a whole if you try and re-implement in its entirety.

tonyarkles 10y ago
I perpetually love the references to Ship of Theseus, and find it particularly applicable to this problem. You mention fundamental functional problems, and I'd like to add something: sometimes it's not a functional problem, but a changeability problem. The code functions fine, but the process of adding a new feature is incredibly painful. I've done big-bang partial rewrites of systems before, quite successfully, but I've got a Ship of Theseus rule of my own that I follow: no new features during the rewrite, and no missing features. The first example that comes to mind was a rather complicated front-end application that had become a total spaghetti disaster. It had been written using Backbone, and from my research it fit the Angular model of things quite well. I took a pound of coffee with me out to the cabin for a weekend, and rewrote the whole frontend. Side-by-side. I started with the first screen, and walked through every code path to populate a queue of screens that were reachable from there. Implement a screen, capture its outbound links, repeat. Nothing changed, but everything changed. The stylesheet was gnarly too, but I left it 100% untouched. That comes later. By keeping the stylesheet intact, I (somewhat) ensured that the new implementation used identical markup to the old system. The markup was gnarly too, but keeping it identical helped to ensure that nothing got missed. 48-72ish hours later, I emerged a bit scraggly and the rest of the team started going over it. They found 1 or 2 minor things, but within a week or so we did a full cut-over. The best part? Unlike the article, clients had no outward indication that anything had changed. There was no outcry, even though about a quarter of the code in the system had been thrown out.
- radicalbyte 10y ago
  I have similar experience: the "incremental rewrite" is usually the most effective tactic to apply. It reduces risk, cuts "time to market" and lets you apply Pareto's rule - making the process efficient. Very rarely a rewrite is the answer: for example if the product is functioning poorly, where even low risk changes cause random regressions. Where large system-level changes are needed to make the system work - say because the original developers didn't implement authorization checks after the login.html page. Where none of the original developers are available and no-one knows the requirements. Where the system is a hodgepodge of 4 different frameworks including one custom one (whose developer is long gone, leaving 0 documentation or tests behind). In cases like that the software artifacts are a liability; it's better to use the assets you've built up (domain knowledge) and develop a new product in parallel. Put the old one in zombie mode and spend two years building the new product with feature parity. There is one other situation where a full rewrite is good: if v1 is a throwaway prototype. However that would never have been put into production for any significant number of users.
  - dpark 10y ago
    >In cases like that the software artifacts are a liability; it's better to use the assets you've built up (domain knowledge) and develop a new product in parallel. Put the old one in zombie mode and spend two years building the new product with feature parity. In two years your devs should be able to understand the existing code, fix the auth problems, and excise the worst of the code and hodgepodge framework mess. If they can't, then they certainly can't maintain the old system while building a replacement. If you can't understand the old code, you can't replace it. If you don't understand the requirements, I'm not sure you can even maintain the existing system. It shouldn't take two years to add auth, or remove an unsupported framework. This is the textbook case of when a rewrite will fail, because the scope is not just too big, but unknown (because you don't even understand the requirements). The choice to rewrite in this situation is not logical. It's emotional. The existing system is a mess, and the fix seems so difficult. But the rewrite estimate is likely poor because you don't actually understand the system you're rebuilding, and even if it's an accurate estimate you lose two years guaranteed just to ship feature parity. You can make a whole lot of improvements to a codebase in two years while also shipping features.
p4wnc6 10y ago
While I generally agree with this, there is one situation when scorched earth re-write should at least be kicked around as an option. If you are dealing with a very bad system that has not been maintained and has tons of candidates for those tightly-scoped incremental fixes, but at the same time, you are embedded in a giant, faceless conglomerate sort of company where there is effectively 0% chance that any of those tightly-scoped incremental fixes will ever be greenlighted and everyone in the team knows it. Then you should consider the scorched earth approach, because it is the only way that the incremental fixes will ever happen. Bureaucrats will always find an excuse why this particular short term time frame is not the right one for the incremental fix that slightly slows productivity and gets them dinged on their bonus. And they will always find a way to pin the blame regarding underinvestment in critical incremental fixes on the development staff. So sometimes all you can do is deny them that option, even if you know how painful it will be. The aggregated long-run pain will be less, though it won't feel that way for a long time. I just want to reiterate that I mostly agree with you. It's especially bad to turn your nose up at legacy code that actually has valuable tests, because the tests make the maintenance and incremental fixes so much better. When there are solid tests, you should almost never throw it away. Nonetheless, sometimes you have to torch it all to deny the bureaucrats the chance to slowly suck the life out of it (and you).
- dack 10y ago
  I dunno, after you do the rewrite, then what? If the bureaucrats won't ever let you make incremental fixes/cleanup, then even your rewritten version will eventually rot. I think the idea of building software "the right way, once and for all" is mostly just wishful thinking on our part. Even the ideal rewrite will have ongoing maintenance that needs to be done. Therefore, my only two options in that situation would be to a) make those incremental changes as part of features without telling them (estimated as one chunk), or b) look for a new job.
  - DougWebb 10y ago
    At my old job, we built, maintained, and operated a huge customer-facing system. We had 2-3 big releases per year, and in every release cycle we devoted around 20% of our time to maintenance, whatever time was necessary for critical bugs (generally from the new code released in the last cycle) and the rest to new development. We (the dev team) put this split into our estimates and development plans, and we had official approval from the project management team and c-level execs. That 20% time was ours to use as we saw fit, and we generally used it for refactoring code we felt was hampering productivity on new development. But we also used it for side projects; I built a performance/metrics gathering and reporting system that was incredibly useful to me, and ultimately became incorporated in marketing materials and the ceo's presentations. We were lucky to be able to use this 20% time openly, but if we weren't we would have increased our new dev estimates to get the time.
aytekin 10y ago
One of the cool things about doing replace-in-place is that you can A/B test the old and new versions. We are doing that on our current Form Builder rewrite at JotForm. We rewrite one piece and make it live to 50% of users. Then we receive daily morning emails about each test. If some metric is in red, we discuss (or watch fullstory, or talk to users) what might be causing it and improve the new version. Here is a fresh example, we released the new version of PayPal payments pro/express integration, and the success rates stayed in red. The old version was beating the new version. It was 3x better even though almost everything was same. After some head scratching, we found that the old version had a link to a direct PayPal page where the can get their api credentials, and the new version was missing it. From there, the fix was easy and things turned green. This is a story that has happened over and over again. When you rewrite software, you lose all those hundreds of tiny things which were added for really good reasons. Don't do it blindly.
- hinkley 10y ago
  Usually these problems come down to the information architecture, and I wonder sometimes if the skills necessary to do a replace-in-place to fix those sorts of problem might be better off, at least in general, being applied to some other problem. If people want broken stuff, or things that violate the laws of physics, information theory, and common decency, no amount of heroics is gonna keep the wheels on forever. Maybe writing software to cure cancer or improve food distribution (ie, cure hunger) would be a better use of your energy.
  - aytekin 10y ago
    Our product is used by 2 million people and some of them are literally curing cancer and hunger. That's actually a great motivational reason for us to continuously improve it.
    - DonHopkins 10y ago
      Yes, it's a great feeling and strong motivation to work on something like that, and it gives you the will to stick to it for the long run. I've been working on a system for improving health and preventing diabetes since 2005 [1], which is complicated by the fact that it's designed from the start to support controlled clinical trials, followup questionnaires, and analytics to measure how well it works. That's enabled us measure how well it works (which literally involved drawing blood), publish papers proving its effectiveness, and feed that learning back into many changes to improve the system over the years. Parts of the system really needed to be rewritten, but were originally intertwined all throughout, and there was no time for a rewrite. So over the years I've been incrementally refactoring and isolating those parts to make it easier to remove them in the future, even when we were under crunch and there was no time to rewrite. I finally found the time to completely replace the bad parts, and it turned out to be much easier (and more satisfying) that I expected, significantly reduced the complexity and lines of code, and made it possible for us to hire contractors to work on the code without their heads exploding. But I think it went so well because I'd been thinking about it and chipping away at it for years. [1] http://turnaroundhealth.com http://turnaroundhealth.com
reitanqild 10y ago
For anyone who has to look up the "Ship of Thesus", here it is: https://en.m.wikipedia.org/wiki/Ship_of_Theseus https://en.m.wikipedia.org/wiki/Ship_of_Theseus
- HillRat 10y ago
  Or Fowler's "Strangler Pattern," a similar approach.
ralphael 10y ago
...this is a good reminder that sometimes its a good idea to hire for certain positions, based on prior experience.
- fallous 10y ago
  Indeed, but then the organization has to know what it does not know... which presumes a certain wisdom in the first place. ;)

by fallous 10y ago

tonyarkles 10y ago

I perpetually love the references to Ship of Theseus, and find it particularly applicable to this problem. You mention fundamental functional problems, and I'd like to add something: sometimes it's not a functional problem, but a changeability problem. The code functions fine, but the process of adding a new feature is incredibly painful. I've done big-bang partial rewrites of systems before, quite successfully, but I've got a Ship of Theseus rule of my own that I follow: no new features during the rewrite, and no missing features. The first example that comes to mind was a rather complicated front-end application that had become a total spaghetti disaster. It had been written using Backbone, and from my research it fit the Angular model of things quite well. I took a pound of coffee with me out to the cabin for a weekend, and rewrote the whole frontend. Side-by-side. I started with the first screen, and walked through every code path to populate a queue of screens that were reachable from there. Implement a screen, capture its outbound links, repeat. Nothing changed, but everything changed. The stylesheet was gnarly too, but I left it 100% untouched. That comes later. By keeping the stylesheet intact, I (somewhat) ensured that the new implementation used identical markup to the old system. The markup was gnarly too, but keeping it identical helped to ensure that nothing got missed. 48-72ish hours later, I emerged a bit scraggly and the rest of the team started going over it. They found 1 or 2 minor things, but within a week or so we did a full cut-over. The best part? Unlike the article, clients had no outward indication that anything had changed. There was no outcry, even though about a quarter of the code in the system had been thrown out.

radicalbyte 10y ago
I have similar experience: the "incremental rewrite" is usually the most effective tactic to apply. It reduces risk, cuts "time to market" and lets you apply Pareto's rule - making the process efficient. Very rarely a rewrite is the answer: for example if the product is functioning poorly, where even low risk changes cause random regressions. Where large system-level changes are needed to make the system work - say because the original developers didn't implement authorization checks after the login.html page. Where none of the original developers are available and no-one knows the requirements. Where the system is a hodgepodge of 4 different frameworks including one custom one (whose developer is long gone, leaving 0 documentation or tests behind). In cases like that the software artifacts are a liability; it's better to use the assets you've built up (domain knowledge) and develop a new product in parallel. Put the old one in zombie mode and spend two years building the new product with feature parity. There is one other situation where a full rewrite is good: if v1 is a throwaway prototype. However that would never have been put into production for any significant number of users.
- dpark 10y ago
  >In cases like that the software artifacts are a liability; it's better to use the assets you've built up (domain knowledge) and develop a new product in parallel. Put the old one in zombie mode and spend two years building the new product with feature parity. In two years your devs should be able to understand the existing code, fix the auth problems, and excise the worst of the code and hodgepodge framework mess. If they can't, then they certainly can't maintain the old system while building a replacement. If you can't understand the old code, you can't replace it. If you don't understand the requirements, I'm not sure you can even maintain the existing system. It shouldn't take two years to add auth, or remove an unsupported framework. This is the textbook case of when a rewrite will fail, because the scope is not just too big, but unknown (because you don't even understand the requirements). The choice to rewrite in this situation is not logical. It's emotional. The existing system is a mess, and the fix seems so difficult. But the rewrite estimate is likely poor because you don't actually understand the system you're rebuilding, and even if it's an accurate estimate you lose two years guaranteed just to ship feature parity. You can make a whole lot of improvements to a codebase in two years while also shipping features.

p4wnc6 10y ago

While I generally agree with this, there is one situation when scorched earth re-write should at least be kicked around as an option. If you are dealing with a very bad system that has not been maintained and has tons of candidates for those tightly-scoped incremental fixes, but at the same time, you are embedded in a giant, faceless conglomerate sort of company where there is effectively 0% chance that any of those tightly-scoped incremental fixes will ever be greenlighted and everyone in the team knows it. Then you should consider the scorched earth approach, because it is the only way that the incremental fixes will ever happen. Bureaucrats will always find an excuse why this particular short term time frame is not the right one for the incremental fix that slightly slows productivity and gets them dinged on their bonus. And they will always find a way to pin the blame regarding underinvestment in critical incremental fixes on the development staff. So sometimes all you can do is deny them that option, even if you know how painful it will be. The aggregated long-run pain will be less, though it won't feel that way for a long time. I just want to reiterate that I mostly agree with you. It's especially bad to turn your nose up at legacy code that actually has valuable tests, because the tests make the maintenance and incremental fixes so much better. When there are solid tests, you should almost never throw it away. Nonetheless, sometimes you have to torch it all to deny the bureaucrats the chance to slowly suck the life out of it (and you).

dack 10y ago
I dunno, after you do the rewrite, then what? If the bureaucrats won't ever let you make incremental fixes/cleanup, then even your rewritten version will eventually rot. I think the idea of building software "the right way, once and for all" is mostly just wishful thinking on our part. Even the ideal rewrite will have ongoing maintenance that needs to be done. Therefore, my only two options in that situation would be to a) make those incremental changes as part of features without telling them (estimated as one chunk), or b) look for a new job.
- DougWebb 10y ago
  At my old job, we built, maintained, and operated a huge customer-facing system. We had 2-3 big releases per year, and in every release cycle we devoted around 20% of our time to maintenance, whatever time was necessary for critical bugs (generally from the new code released in the last cycle) and the rest to new development. We (the dev team) put this split into our estimates and development plans, and we had official approval from the project management team and c-level execs. That 20% time was ours to use as we saw fit, and we generally used it for refactoring code we felt was hampering productivity on new development. But we also used it for side projects; I built a performance/metrics gathering and reporting system that was incredibly useful to me, and ultimately became incorporated in marketing materials and the ceo's presentations. We were lucky to be able to use this 20% time openly, but if we weren't we would have increased our new dev estimates to get the time.

aytekin 10y ago

One of the cool things about doing replace-in-place is that you can A/B test the old and new versions. We are doing that on our current Form Builder rewrite at JotForm. We rewrite one piece and make it live to 50% of users. Then we receive daily morning emails about each test. If some metric is in red, we discuss (or watch fullstory, or talk to users) what might be causing it and improve the new version. Here is a fresh example, we released the new version of PayPal payments pro/express integration, and the success rates stayed in red. The old version was beating the new version. It was 3x better even though almost everything was same. After some head scratching, we found that the old version had a link to a direct PayPal page where the can get their api credentials, and the new version was missing it. From there, the fix was easy and things turned green. This is a story that has happened over and over again. When you rewrite software, you lose all those hundreds of tiny things which were added for really good reasons. Don't do it blindly.

hinkley 10y ago
Usually these problems come down to the information architecture, and I wonder sometimes if the skills necessary to do a replace-in-place to fix those sorts of problem might be better off, at least in general, being applied to some other problem. If people want broken stuff, or things that violate the laws of physics, information theory, and common decency, no amount of heroics is gonna keep the wheels on forever. Maybe writing software to cure cancer or improve food distribution (ie, cure hunger) would be a better use of your energy.
- aytekin 10y ago
  Our product is used by 2 million people and some of them are literally curing cancer and hunger. That's actually a great motivational reason for us to continuously improve it.
  - DonHopkins 10y ago
    Yes, it's a great feeling and strong motivation to work on something like that, and it gives you the will to stick to it for the long run. I've been working on a system for improving health and preventing diabetes since 2005 [1], which is complicated by the fact that it's designed from the start to support controlled clinical trials, followup questionnaires, and analytics to measure how well it works. That's enabled us measure how well it works (which literally involved drawing blood), publish papers proving its effectiveness, and feed that learning back into many changes to improve the system over the years. Parts of the system really needed to be rewritten, but were originally intertwined all throughout, and there was no time for a rewrite. So over the years I've been incrementally refactoring and isolating those parts to make it easier to remove them in the future, even when we were under crunch and there was no time to rewrite. I finally found the time to completely replace the bad parts, and it turned out to be much easier (and more satisfying) that I expected, significantly reduced the complexity and lines of code, and made it possible for us to hire contractors to work on the code without their heads exploding. But I think it went so well because I'd been thinking about it and chipping away at it for years. [1] http://turnaroundhealth.com http://turnaroundhealth.com

reitanqild 10y ago

For anyone who has to look up the "Ship of Thesus", here it is: https://en.m.wikipedia.org/wiki/Ship_of_Theseus https://en.m.wikipedia.org/wiki/Ship_of_Theseus

HillRat 10y ago
Or Fowler's "Strangler Pattern," a similar approach.

ralphael 10y ago

...this is a good reminder that sometimes its a good idea to hire for certain positions, based on prior experience.

fallous 10y ago
Indeed, but then the organization has to know what it does not know... which presumes a certain wisdom in the first place. ;)