5 ms·

Lessons learned from the big rewrite

75 points by antonpirker 12y ago

joosters 12y ago
Lesson 1: Don't ever attempt a big rewrite?
- antonpirker 12y ago
  Sure you can attempt a rewrite! But you must make sure that you know what are getting into. It is not an easy thing to do, but once the rewrite is done your live gets way better :)
  - arethuza 12y ago
    Having had rewrites of large projects go spectacularly well and also spectacularly badly, I suspect one factor to be mindful about is if the rewrite facilitates the "second-system effect" - I've had that happen where the rewrite of a messy but successful suddenly allows the introduction of lots of cool features you've wanted to add for ages and you actually end up with a "simplified" system that is actually a lot more complex than the original. NB I've only had one project go badly wrong that way and I guess it was a useful learning experience :-)
- chriswarbo 12y ago
  I agree; instead of doing big rewrites, do a bunch of small rewrites. Especially try to avoid "big bang" switchovers.
  - twic 12y ago
    The OP's big rewrite was a lot of work. But I suspect that incrementally improving the legacy system to the point where it was as good as the new one would have been even more work. Incremental improvement is the right way to make small improvements. But a rewrite can be the right way to achieve root-and-branch reform.
    - chriswarbo 12y ago
      I wasn't advocating a gradual improvement of the old system. I meant identifying N particular tasks that need replacing, for example user accounts, content management, front-end, etc. Then for each one, isolating that part of the old system, eg. using a proxying layer. A replacement can then be built alongside, with the proxy sending all requests to both. Results can be compared, and differences treated as test failures. When the replacement is working, the proxy can be switched to serve the new system's responses, and once everything's settled down, the old system can be turned off. Once all of the new system is in place and replicating the behaviour of the old system, the rewrite is finished. New features and improvements can be worked on as normal, as a separate phase. What's more, we only need to feature-freeze the component that's being migrated, rather than the whole thing (as long as coding standards for new features enforce our newly-imposed compoentn isolation).
curiously 12y ago
I rewrote my product 6 or 7 times in the past 6 years. 4 of those years were written in isolation by myself. 1.5 years I worked as a developer which I HATED and fired from (refusing to work excessive overtimes and as a result impossible to find a job in this competitive labor market) and working on my product again. I open sourced the old code that I "threw" out to start all over. Technically, never reached market until 5 months ago. I have of course, launched after each time the product was finished but would give up due to being burnt out etc. An entire year spent starting from scratch which would be my 7th attempt. 3 month in after launch I was going to rewrite it again, I decided, this would've been my last attempt in the software business. I decided that enough was enough, this is it. I ALMOST rewrote it again but having done that 7 times already, instead of doing that, I just ripped out a lot of code, made architectural changes and reused what I could, whatever was tested. 8 times a charm.... My lesson learned from past 6 years pursuing insanity, restarting from scratch means you will have to solve the same problems again and again. A blank canvas is NOT a solution. Don't do that, it's insane, I was insane but not anymore.
- segmondy 12y ago
  Stop chasing your tail. I have no idea what you are working on, but if you are not trying to solve a hard CS problem and you are trying to build a business ...
  - curiously 12y ago
    NLP from texts detected from the audio in video files. 100% solved? No. Others have attempted or marketing their own solution? Yes. I was just really crazy and stupid to pursue my own vision of what the solution would be when in reality I lacked the experience (I started with barely knowing how to code).
Fumzz 12y ago
To number 2 (Migrating the database is hard) I would also add that it is important to verify the integrity of the migrated database in some way afterwards. Even a simple assertion like "the number of user accounts (or whatever) is the same in both databases" will help greatly.
- antonpirker 12y ago
  Yes, right! I forgot about this in the blog post. We counted the rows of some tables. Unfortunately we could not do this with all tables because we had dead data in the old database (ex: child rows without parents). But with the new database we made sure to have all the database features and checks in place to guarantee referential integrity. We also did spot tests to make sure the data is as expected.
dschiptsov 12y ago
It is not that simple and comes back to the same old maxim - as long as the only documentation of what the system is doing and why is its source code, you are doomed. If you have a written detailed guidelines of what you are doing and why, which design decisions were made and why, which data representation is used and why and what corresponding algorithms are and why, then one could re-write parts of the system in different languages/frameworks just for fun or to explore and compare. Coding or rewriting is second to deep, hierarchical understanding of hows and whys. This is how the most of programming exercises are designed - once you have understood how it should be, which results each procedure should return (wishful thinking and tests before code, to define domain and range of a procedure - ideas as old as SICP) then coding is the most trivial part. Look at antirez - he is a rare example of a guy of deep understanding before code.
- antonpirker 12y ago
  'once you have understood how it should be, which results each procedure should return then coding is the most trivial part.' I wholeheartedly agree with this! The big challenge is that you have the patience (and the patience of the management) to understand the old system thoroughly before starting to write new code.
- thirdtruck 12y ago
  Uncle Bob made this point very salient in one of his videos: If there was some catastrophic, quirky failure of backup, and you could only save either (a) all your production code, sans test, or (b) only your tests, which would you rather save? Assuming you've written thorough tests, they win out over code every time. The tests remove all ambiguity. I'm reminded of a story that illustrates the point: A young woman always cuts the ends off of her pot roasts before cooking them, following her mother's example. One day, she asks her mother why they do that. The mother doesn't recall, so she asks the grandmother. The grandmother looks at them both, shrugs, and says, "When I was your age, we lived in an apartment with a very small oven. We had to cut the pot roast down to make it fit!"
  - rquirk 12y ago
Kiro 12y ago
Any drawbacks of #3?
- antonpirker 12y ago
  It's a really easy thing to do. But you need enough server resources for this. A mirror of the current production system would be the best thing. If you only have one server where the live system is running and there is not enough resources on this server you can not do this.
LeonidBugaev 12y ago
For testing performance i would recommend https://github.com/buger/gor https://github.com/buger/gor (i'm author).
- nkuttler 12y ago
  Nice, I was just going to point out that using an iframe really seems like a bad idea.
troels 12y ago
My experience from doing rewrites is that if at all possible (and it usually is, even if you don't think so), do the rewrite incrementally. For example, in the case described here, they wanted to switch rdbms, change up the schema, change the application level implementation/framework and change the interface all at once. That's a lot of risk to combine. Maybe they could have started by refactoring the schema in-place (That is, within mysql and webtek). Then change from mysql to pgsql - still behind webtek. Maybe separating the change of webtek -> django from the interface change is a bit harder, but you could still try to restrain your self as much as possible and then make a second iteration of the interface after the initial change. I've taken this approach before, when migrating a big product and it worked quite well. It seems as a lot of extra work at first, but I'm pretty sure that is not actually case.
- Demiurge 12y ago
  Doing it incrementally is definitely the best. However, in your scenario, the problem of a 'weird' codebase people have a hard time using is not addressed by requiring further development in it. Database layout is usually not the problem if the system was at all functional. Having done this a few times, when switching from old to new, the new framework would have a DB agnostic ORM. What works nicely is building the models, duplicate templating, getting a mirror working. Shut down the old site, and then it's possible to go on with normal development. So, it is important to consider rewrite and redesign as separate goals.
  - a8da6b0c91d 12y ago
    In this case porting from perl to python makes it much harder and seems pointless. Why not just keep it in perl but incrementally move to a modern perl framework?
    - Demiurge 12y ago
      Why do we see so many more posts about Django compared to 'modern Perl framework'? Python has killer features like simple syntax and developer familiarity. Django has GeoDjango. This isn't even a question to me, but to the blog author. My. Point is about order of steps in the rewrite. It is not harder to rewrite first, redesign second, it is easier. I've done it multiple times in c++/c#, perl/python, php/python, r/python.
adrianhoward 12y ago
For those who've not already come across it I would thoroughly recommend folk taking a gander at Michael Feathers book "Working Effectively with Legacy Code" on this topic. Lots of solid advice and patterns for approaching thing sort of thing.
ExpiredLink 12y ago
> The first thing I will do the next time is taking more time to analyze the old system thoroughly and write a technical spec what the new system should do. A simple "What the old system is doing now" as a spec for the rewrite does not cut it. No one knows what the old system is doing exactly so it is necessary to take the time to write a spec. I had the same experiences last year. There wasn't even a single person who knew the 'old system' on the whole. A receipt for many detours and much extra work.
fsniper 12y ago
I have an issue with iframe perf test. It is very reckless and a waste of user bandwidth. One can reroll user requests from web server access logs and it would be a controlled perf test with real user behaviour.
h1fra 12y ago
After spending 3months rewriting an huge module for our many websites, initially estimated for 3weeks at most, I can say that it's easy to make mistake but also incredibly good for the company. - Mixing refactoring and new feature. - Introducing new technology - Changing behavior of old code that were not directly concerned by the rewrite ... Many things that should have been done after that. But, all I can say is if the company can handle this amount of work, it will always be a very good way to improve the quality and durability of the website. Because if you simply rewrite code, there is a big chance that your company will assign to you, new task that are not related at all and you will never be able to introduce these feature.

75 points by antonpirker 12y ago

joosters 12y ago

Lesson 1: Don't ever attempt a big rewrite?

antonpirker 12y ago
Sure you can attempt a rewrite! But you must make sure that you know what are getting into. It is not an easy thing to do, but once the rewrite is done your live gets way better :)
- arethuza 12y ago
  Having had rewrites of large projects go spectacularly well and also spectacularly badly, I suspect one factor to be mindful about is if the rewrite facilitates the "second-system effect" - I've had that happen where the rewrite of a messy but successful suddenly allows the introduction of lots of cool features you've wanted to add for ages and you actually end up with a "simplified" system that is actually a lot more complex than the original. NB I've only had one project go badly wrong that way and I guess it was a useful learning experience :-)
chriswarbo 12y ago
I agree; instead of doing big rewrites, do a bunch of small rewrites. Especially try to avoid "big bang" switchovers.
- twic 12y ago
  The OP's big rewrite was a lot of work. But I suspect that incrementally improving the legacy system to the point where it was as good as the new one would have been even more work. Incremental improvement is the right way to make small improvements. But a rewrite can be the right way to achieve root-and-branch reform.
  - chriswarbo 12y ago
    I wasn't advocating a gradual improvement of the old system. I meant identifying N particular tasks that need replacing, for example user accounts, content management, front-end, etc. Then for each one, isolating that part of the old system, eg. using a proxying layer. A replacement can then be built alongside, with the proxy sending all requests to both. Results can be compared, and differences treated as test failures. When the replacement is working, the proxy can be switched to serve the new system's responses, and once everything's settled down, the old system can be turned off. Once all of the new system is in place and replicating the behaviour of the old system, the rewrite is finished. New features and improvements can be worked on as normal, as a separate phase. What's more, we only need to feature-freeze the component that's being migrated, rather than the whole thing (as long as coding standards for new features enforce our newly-imposed compoentn isolation).

curiously 12y ago

I rewrote my product 6 or 7 times in the past 6 years. 4 of those years were written in isolation by myself. 1.5 years I worked as a developer which I HATED and fired from (refusing to work excessive overtimes and as a result impossible to find a job in this competitive labor market) and working on my product again. I open sourced the old code that I "threw" out to start all over. Technically, never reached market until 5 months ago. I have of course, launched after each time the product was finished but would give up due to being burnt out etc. An entire year spent starting from scratch which would be my 7th attempt. 3 month in after launch I was going to rewrite it again, I decided, this would've been my last attempt in the software business. I decided that enough was enough, this is it. I ALMOST rewrote it again but having done that 7 times already, instead of doing that, I just ripped out a lot of code, made architectural changes and reused what I could, whatever was tested. 8 times a charm.... My lesson learned from past 6 years pursuing insanity, restarting from scratch means you will have to solve the same problems again and again. A blank canvas is NOT a solution. Don't do that, it's insane, I was insane but not anymore.

segmondy 12y ago
Stop chasing your tail. I have no idea what you are working on, but if you are not trying to solve a hard CS problem and you are trying to build a business ...
- curiously 12y ago
  NLP from texts detected from the audio in video files. 100% solved? No. Others have attempted or marketing their own solution? Yes. I was just really crazy and stupid to pursue my own vision of what the solution would be when in reality I lacked the experience (I started with barely knowing how to code).

Fumzz 12y ago

To number 2 (Migrating the database is hard) I would also add that it is important to verify the integrity of the migrated database in some way afterwards. Even a simple assertion like "the number of user accounts (or whatever) is the same in both databases" will help greatly.

antonpirker 12y ago
Yes, right! I forgot about this in the blog post. We counted the rows of some tables. Unfortunately we could not do this with all tables because we had dead data in the old database (ex: child rows without parents). But with the new database we made sure to have all the database features and checks in place to guarantee referential integrity. We also did spot tests to make sure the data is as expected.

dschiptsov 12y ago

It is not that simple and comes back to the same old maxim - as long as the only documentation of what the system is doing and why is its source code, you are doomed. If you have a written detailed guidelines of what you are doing and why, which design decisions were made and why, which data representation is used and why and what corresponding algorithms are and why, then one could re-write parts of the system in different languages/frameworks just for fun or to explore and compare. Coding or rewriting is second to deep, hierarchical understanding of hows and whys. This is how the most of programming exercises are designed - once you have understood how it should be, which results each procedure should return (wishful thinking and tests before code, to define domain and range of a procedure - ideas as old as SICP) then coding is the most trivial part. Look at antirez - he is a rare example of a guy of deep understanding before code.

antonpirker 12y ago
'once you have understood how it should be, which results each procedure should return then coding is the most trivial part.' I wholeheartedly agree with this! The big challenge is that you have the patience (and the patience of the management) to understand the old system thoroughly before starting to write new code.
thirdtruck 12y ago
Uncle Bob made this point very salient in one of his videos: If there was some catastrophic, quirky failure of backup, and you could only save either (a) all your production code, sans test, or (b) only your tests, which would you rather save? Assuming you've written thorough tests, they win out over code every time. The tests remove all ambiguity. I'm reminded of a story that illustrates the point: A young woman always cuts the ends off of her pot roasts before cooking them, following her mother's example. One day, she asks her mother why they do that. The mother doesn't recall, so she asks the grandmother. The grandmother looks at them both, shrugs, and says, "When I was your age, we lived in an apartment with a very small oven. We had to cut the pot roast down to make it fit!"
- rquirk 12y ago

Kiro 12y ago

Any drawbacks of #3?

antonpirker 12y ago
It's a really easy thing to do. But you need enough server resources for this. A mirror of the current production system would be the best thing. If you only have one server where the live system is running and there is not enough resources on this server you can not do this.

LeonidBugaev 12y ago

For testing performance i would recommend https://github.com/buger/gor https://github.com/buger/gor (i'm author).

nkuttler 12y ago
Nice, I was just going to point out that using an iframe really seems like a bad idea.

troels 12y ago

My experience from doing rewrites is that if at all possible (and it usually is, even if you don't think so), do the rewrite incrementally. For example, in the case described here, they wanted to switch rdbms, change up the schema, change the application level implementation/framework and change the interface all at once. That's a lot of risk to combine. Maybe they could have started by refactoring the schema in-place (That is, within mysql and webtek). Then change from mysql to pgsql - still behind webtek. Maybe separating the change of webtek -> django from the interface change is a bit harder, but you could still try to restrain your self as much as possible and then make a second iteration of the interface after the initial change. I've taken this approach before, when migrating a big product and it worked quite well. It seems as a lot of extra work at first, but I'm pretty sure that is not actually case.

Demiurge 12y ago
Doing it incrementally is definitely the best. However, in your scenario, the problem of a 'weird' codebase people have a hard time using is not addressed by requiring further development in it. Database layout is usually not the problem if the system was at all functional. Having done this a few times, when switching from old to new, the new framework would have a DB agnostic ORM. What works nicely is building the models, duplicate templating, getting a mirror working. Shut down the old site, and then it's possible to go on with normal development. So, it is important to consider rewrite and redesign as separate goals.
- a8da6b0c91d 12y ago
  In this case porting from perl to python makes it much harder and seems pointless. Why not just keep it in perl but incrementally move to a modern perl framework?
  - Demiurge 12y ago
    Why do we see so many more posts about Django compared to 'modern Perl framework'? Python has killer features like simple syntax and developer familiarity. Django has GeoDjango. This isn't even a question to me, but to the blog author. My. Point is about order of steps in the rewrite. It is not harder to rewrite first, redesign second, it is easier. I've done it multiple times in c++/c#, perl/python, php/python, r/python.

adrianhoward 12y ago

For those who've not already come across it I would thoroughly recommend folk taking a gander at Michael Feathers book "Working Effectively with Legacy Code" on this topic. Lots of solid advice and patterns for approaching thing sort of thing.

ExpiredLink 12y ago

> The first thing I will do the next time is taking more time to analyze the old system thoroughly and write a technical spec what the new system should do. A simple "What the old system is doing now" as a spec for the rewrite does not cut it. No one knows what the old system is doing exactly so it is necessary to take the time to write a spec. I had the same experiences last year. There wasn't even a single person who knew the 'old system' on the whole. A receipt for many detours and much extra work.

fsniper 12y ago

I have an issue with iframe perf test. It is very reckless and a waste of user bandwidth. One can reroll user requests from web server access logs and it would be a controlled perf test with real user behaviour.

h1fra 12y ago

After spending 3months rewriting an huge module for our many websites, initially estimated for 3weeks at most, I can say that it's easy to make mistake but also incredibly good for the company. - Mixing refactoring and new feature. - Introducing new technology - Changing behavior of old code that were not directly concerned by the rewrite ... Many things that should have been done after that. But, all I can say is if the company can handle this amount of work, it will always be a very good way to improve the quality and durability of the website. Because if you simply rewrite code, there is a big chance that your company will assign to you, new task that are not related at all and you will never be able to introduce these feature.