5 ms·

This is madness. There's no real advice about rewrites in this article. The reality of rewrites is that they often fail, cost a ridiculous amount of time and ef

by tinco 5y ago

This is madness. There's no real advice about rewrites in this article. The reality of rewrites is that they often fail, cost a ridiculous amount of time and effort, don't actually fix the original problem and/or bankrupt the company. You can't just wave away those realities with "think about the downsides". I have participated in a few (small) rewrites that were successful, and this would be my advice: Don't rewrite an entire project at once. Pick off smaller features first, get a feel for what your new approach will be like before you commit to the big stuff. This will often mean you have to migrate from a monolithic architecture to a services architecture first. This means your project will grow in complexity first. If you're confused about why a project first grows in complexity before shrinking, watch "All the little things" by Sandi Metz, it's the greatest programming talk ever recorded. If your rewrite gets stuck after adding all of that complexity, then you've played yourself. Worst thing that can happen is you rewrite your entire project, some users or customers start using the new project, but you've failed to rewrite everything and some chunk of your users are still using the old codebase. Now you've got twice as much code to maintain. Before you start a rewrite, consider refactoring and rearchitecting the old codebase, there might be a gem in there that just needs some love and attention. That said there definitely are famous successful rewrites, and I think web application backends are especially suitable for rewrites. Most famous are Twitter and LinkedIn, Twitter going from a relational database backend to a more appropriate fan out message queue based backend allowing them to scale beyond imagination. LinkedIn going from a big rails monolith to super fast node.js microservices reducing their hardware costs a hundred fold if stories are to be believed.

yaohackernews 5y ago
I just checked out "All the little things" from Sandi Metz. It's incredible! I need to get back here to say thank you for this information!
- tinco 5y ago
  No problem! I try to spread the word whenever I can.
eternityforest 5y ago
I'm convinced that bad rewrites happen because coders get bored and care more about the possibility of a little bit better code, then the whole future of the company. Necessary rewrites happen because they did too much YAGNI thinking in version 1.
- tinco 5y ago
  It annoys me so much that after honing my skill as an engineer for 15 years now I still fall into the trap of not enough YAGNI thinking. There's a precarious middle ground there for sure. Even though I still fall into the trap of over designing, I believe it's way worse than under designing. I don't remember a single instance where a 'YAGNI' under designing cost us much except designing and implementing the feature the right way, a cost which we'd incur anyway. In contrast I've caused my team to work with an overly complex codebase for years on multiple occasions.
  - eternityforest 5y ago
    The only time I've regretted overdesigning is when there's been significant original design work involved. I don't think I've ever regretted overdesigning to meet a common standard or match how everyone else does something. A lot of the YAGNI I see is in the form of "We don't need a framework, just do vanilla JS" or "I only need one of the features of this library, I'll just write it myself". Almost all of my own big tech regrets have to do with building something "innovative". Even when they've solved the problem better than existing solutions they often become regrets, just for the simple fact that they aren't better by enough to justify the nonstandardness.
    - tinco 5y ago
      Ah, I think I wouldn't qualify those as YAGNI decisions, more in the premature optimisation direction. YAGNI is preparing for future integrations that might never happen, or preparing for supporting multiple databases, special users, special circumstances. It is always a decision of doing more work now, versus more work later. The decision to not use a framework or a library is sort of the reverse in that you're making a decision that you can later reverse by reducing the complexity of your own code. Definitely agree with you that a part of being a very effective development team is owning as little code as possible, by using as many existing solutions as possible.
throwaway984393 5y ago
I just started writing up a formal process on how to improve the small bits gradually. It's actually way more complicated than you think at first... Presenting a user problem, identifying root causes, proposing a solution, mapping out the individual components, listing all of the steps needed to complete a solution, working on each step, and validating each resulted in an improvement based on measurements. There's all these little details to consider to get each one of those right. Maybe this is all well described in some software project management book, but I've not seen it.
tialaramex 5y ago
I have participated in a handful of rewrites. Most of them were indeed unnecessary or harmful. However, "X2" was definitely an example of an appropriate rewrite. It had the following conditions: * The existing code "X" was the product of an early-phase startup which hadn't yet figured out what it was going to do. "X" already existed at the point we were still asking questions like, "Who is the customer?" with possible answers like "The NSA and similar entities" even though it eventually became a consumer product - and so obviously it was very flexible but had also been greatly distorted from its original conception to produce a product we were actively selling. * The existing code was developed and maintained by a third party team. So the knowledge of how that system worked was something we'd need to absorb anyway to bring it in-house. Existing code reflected house style of that team, and X2 would be in our style. * It was modestly sized. I think our plan said our small team (less than half-a-dozen people, not all of them available for this full-time) would rewrite it in one calendar year, with an MVP in maybe six months and from there simultaneously copying less important features from "X" and also new want-to-have features to get the finished "X2", that's roughly the timeline we followed. * I was pretty confident we could execute from the initial description of the work to be attempted. I went into a meeting assuming (as I always do in all-hands meetings) that I'd get fired, and came out thinking I should brush up my Java (the chosen language for "X2") but it seemed very possible. * The non-technical leadership actually wanted a new system. "We need a new pig" is how it was explained back to me by the CEO. (From "Lipstick on a pig" the idea that superficial changes to a product can't really disguise its flaws) so there was buy-in of this idea and nobody yelling at us about why the "X2" MVP doesn't deliver most of the things "X" didn't have yet. [This software, and its predecessor, had code names but somebody let me name the successor software and I have no imagination so although it wasn't literally "X" and "X2" that's about how similar the code names were].
- aspaviento 5y ago
  May I ask you something? In your experience, how would you approach the rewriting of an application with a mix of front end and back end code, bloated code from years of patches without any planning for a maintainable design (full of unused/buggy classes/methods/features), lack of tests and barely any documentation explaining its processes?
tsimionescu 5y ago
I think the only way to end up with a successful re-write is to be extremely realistic with the whole organization about the scope. For a program that took 10 years to develop, a rewrite will probably take around 10 years as well to reach feature parity. Obviously, if you stop development on your successful app for 10 years waiting for the rewrite, you will fail. If you think 10 years worth of work can now be finished in 2 years with the benefit of hindsight, you will fail. But, if you start your new app by first attacking new markets that the old one couldn't, so that the new app is genuinely valuable in itself, and then you slowly consolidate it by adding features that the old app had, while still maintaining a crew developing the old app as needed to address the existing markets, you can justify the investment and then some. Of course, this should only be done if it has become clear that the old program is really not possible/worth it to extend and continue in the long run, which can happen for various reasons. There will be times where you will find massive shortcuts you can take, since odds are things you had to develop from scratch 10 years ago exist pre-packaged today. You may also be able to re-use parts of the old app that were actually in good shape and take them whole. But you should never rely on these things making your work shorter - they are nice boons, but the correct estimate is still in the order of magnitude of the old app.
agumonkey 5y ago
That's why proper module and interface matter (and tests). With that you can swap one bit with another implementation with less risk of failure. It's a fine grained migration path in a way.
pyrale 5y ago
> The reality of rewrites is that they often fail, cost a ridiculous amount of time and effort, don't actually fix the original problem and/or bankrupt the company. The reality of rewrites is that they're often blamed for all of that, while the issue is not the rewrite. The issue is that companies that write bad software will, if given an occasion, write bad software again. The issue with rewrites is actually that they're not large enough in scope. When you rewrite software, you should probably also rewrite contributors' interactions, the way the company handles power dynamics and, occasionally, the way accounting works at that company. Short of systemic changes, rewrites are likely to fail. But again, the company is also likely to fail other initiatives aswell. Rewrites that fail in an isolated way are few and far between. > Before you start a rewrite, consider refactoring and rearchitecting the old codebase, there might be a gem in there that just needs some love and attention. OR it might be a dumpster fire that drives your engineers away, one that is fueled by unreasonable deadlines and expectations. Also good luck refactoring your codebase at snail's pace with noncomitting approval, on your own unpaid overtime.
coldtea 5y ago
>The reality of rewrites is that they often fail, cost a ridiculous amount of time and effort, don't actually fix the original problem and/or bankrupt the company. Citation needed. All of these is based on random internet wisdom (e.g. Joel's condemnation, based on a different era, different delivery model, and different concerns than most of today's startup and enteprrise code), and anecdotal evidence. Well, talking of anecdotal evidence I've seen lots of succesful rewrites, that made the thing faster and better. Heck, most/all of the FAANG stacks are rewrites. Twitter, for example, as you've mentioned yourself, is not the same program that needed restart every day to avoid a memory leak back in the 00s - they rewrote it. Reddit is not the original Lisp, and so on. It's also about the scope. You immediately undestood "let's rewrite the whole company code" and advise "don't rewrite the whole project". Well, who said it's "a" project to begin with? Companies more often than not have tons of different services, and rewrites can be easily done for such isolated services (e.g. like the dl.google.com rewrite into Go). Tons of core services of YouTube, CloudFlare, Google, Facebook, and tons of others have been rewriten with languages as Go, Rust, Hack, etc. and they have written about the experience and results as a success. And it's not like survivor bias, e.g. the keep mum about failed cases, because we hear about failures all the time (just not concerning rewritings).
- tinco 5y ago
  I agree, not sure why you are down voted because you make valid points. I can't think of any big public disastrous rewrites except for maybe frontend ones like Digg or enterprise modernizations for example when the government wanted to get rid of their 60s mainframes and the effort cost billions and was ultimately unsuccessful.
ignoramous 5y ago
> This is madness. Pretty sure the article is tongue-in-cheek satire. > There's no real advice about rewrites in this article. The actual advice is in the top image which goes: - Todo: Add critical customer request, fix bugs, new features - In progress: Rewrite everything

by tinco 5y ago

yaohackernews 5y ago

I just checked out "All the little things" from Sandi Metz. It's incredible! I need to get back here to say thank you for this information!

tinco 5y ago
No problem! I try to spread the word whenever I can.

eternityforest 5y ago

I'm convinced that bad rewrites happen because coders get bored and care more about the possibility of a little bit better code, then the whole future of the company. Necessary rewrites happen because they did too much YAGNI thinking in version 1.

tinco 5y ago
It annoys me so much that after honing my skill as an engineer for 15 years now I still fall into the trap of not enough YAGNI thinking. There's a precarious middle ground there for sure. Even though I still fall into the trap of over designing, I believe it's way worse than under designing. I don't remember a single instance where a 'YAGNI' under designing cost us much except designing and implementing the feature the right way, a cost which we'd incur anyway. In contrast I've caused my team to work with an overly complex codebase for years on multiple occasions.
- eternityforest 5y ago
  The only time I've regretted overdesigning is when there's been significant original design work involved. I don't think I've ever regretted overdesigning to meet a common standard or match how everyone else does something. A lot of the YAGNI I see is in the form of "We don't need a framework, just do vanilla JS" or "I only need one of the features of this library, I'll just write it myself". Almost all of my own big tech regrets have to do with building something "innovative". Even when they've solved the problem better than existing solutions they often become regrets, just for the simple fact that they aren't better by enough to justify the nonstandardness.
  - tinco 5y ago
    Ah, I think I wouldn't qualify those as YAGNI decisions, more in the premature optimisation direction. YAGNI is preparing for future integrations that might never happen, or preparing for supporting multiple databases, special users, special circumstances. It is always a decision of doing more work now, versus more work later. The decision to not use a framework or a library is sort of the reverse in that you're making a decision that you can later reverse by reducing the complexity of your own code. Definitely agree with you that a part of being a very effective development team is owning as little code as possible, by using as many existing solutions as possible.

throwaway984393 5y ago

I just started writing up a formal process on how to improve the small bits gradually. It's actually way more complicated than you think at first... Presenting a user problem, identifying root causes, proposing a solution, mapping out the individual components, listing all of the steps needed to complete a solution, working on each step, and validating each resulted in an improvement based on measurements. There's all these little details to consider to get each one of those right. Maybe this is all well described in some software project management book, but I've not seen it.

tialaramex 5y ago

I have participated in a handful of rewrites. Most of them were indeed unnecessary or harmful. However, "X2" was definitely an example of an appropriate rewrite. It had the following conditions: * The existing code "X" was the product of an early-phase startup which hadn't yet figured out what it was going to do. "X" already existed at the point we were still asking questions like, "Who is the customer?" with possible answers like "The NSA and similar entities" even though it eventually became a consumer product - and so obviously it was very flexible but had also been greatly distorted from its original conception to produce a product we were actively selling. * The existing code was developed and maintained by a third party team. So the knowledge of how that system worked was something we'd need to absorb anyway to bring it in-house. Existing code reflected house style of that team, and X2 would be in our style. * It was modestly sized. I think our plan said our small team (less than half-a-dozen people, not all of them available for this full-time) would rewrite it in one calendar year, with an MVP in maybe six months and from there simultaneously copying less important features from "X" and also new want-to-have features to get the finished "X2", that's roughly the timeline we followed. * I was pretty confident we could execute from the initial description of the work to be attempted. I went into a meeting assuming (as I always do in all-hands meetings) that I'd get fired, and came out thinking I should brush up my Java (the chosen language for "X2") but it seemed very possible. * The non-technical leadership actually wanted a new system. "We need a new pig" is how it was explained back to me by the CEO. (From "Lipstick on a pig" the idea that superficial changes to a product can't really disguise its flaws) so there was buy-in of this idea and nobody yelling at us about why the "X2" MVP doesn't deliver most of the things "X" didn't have yet. [This software, and its predecessor, had code names but somebody let me name the successor software and I have no imagination so although it wasn't literally "X" and "X2" that's about how similar the code names were].

aspaviento 5y ago
May I ask you something? In your experience, how would you approach the rewriting of an application with a mix of front end and back end code, bloated code from years of patches without any planning for a maintainable design (full of unused/buggy classes/methods/features), lack of tests and barely any documentation explaining its processes?

tsimionescu 5y ago

I think the only way to end up with a successful re-write is to be extremely realistic with the whole organization about the scope. For a program that took 10 years to develop, a rewrite will probably take around 10 years as well to reach feature parity. Obviously, if you stop development on your successful app for 10 years waiting for the rewrite, you will fail. If you think 10 years worth of work can now be finished in 2 years with the benefit of hindsight, you will fail. But, if you start your new app by first attacking new markets that the old one couldn't, so that the new app is genuinely valuable in itself, and then you slowly consolidate it by adding features that the old app had, while still maintaining a crew developing the old app as needed to address the existing markets, you can justify the investment and then some. Of course, this should only be done if it has become clear that the old program is really not possible/worth it to extend and continue in the long run, which can happen for various reasons. There will be times where you will find massive shortcuts you can take, since odds are things you had to develop from scratch 10 years ago exist pre-packaged today. You may also be able to re-use parts of the old app that were actually in good shape and take them whole. But you should never rely on these things making your work shorter - they are nice boons, but the correct estimate is still in the order of magnitude of the old app.

agumonkey 5y ago

That's why proper module and interface matter (and tests). With that you can swap one bit with another implementation with less risk of failure. It's a fine grained migration path in a way.

pyrale 5y ago

> The reality of rewrites is that they often fail, cost a ridiculous amount of time and effort, don't actually fix the original problem and/or bankrupt the company. The reality of rewrites is that they're often blamed for all of that, while the issue is not the rewrite. The issue is that companies that write bad software will, if given an occasion, write bad software again. The issue with rewrites is actually that they're not large enough in scope. When you rewrite software, you should probably also rewrite contributors' interactions, the way the company handles power dynamics and, occasionally, the way accounting works at that company. Short of systemic changes, rewrites are likely to fail. But again, the company is also likely to fail other initiatives aswell. Rewrites that fail in an isolated way are few and far between. > Before you start a rewrite, consider refactoring and rearchitecting the old codebase, there might be a gem in there that just needs some love and attention. OR it might be a dumpster fire that drives your engineers away, one that is fueled by unreasonable deadlines and expectations. Also good luck refactoring your codebase at snail's pace with noncomitting approval, on your own unpaid overtime.

coldtea 5y ago

>The reality of rewrites is that they often fail, cost a ridiculous amount of time and effort, don't actually fix the original problem and/or bankrupt the company. Citation needed. All of these is based on random internet wisdom (e.g. Joel's condemnation, based on a different era, different delivery model, and different concerns than most of today's startup and enteprrise code), and anecdotal evidence. Well, talking of anecdotal evidence I've seen lots of succesful rewrites, that made the thing faster and better. Heck, most/all of the FAANG stacks are rewrites. Twitter, for example, as you've mentioned yourself, is not the same program that needed restart every day to avoid a memory leak back in the 00s - they rewrote it. Reddit is not the original Lisp, and so on. It's also about the scope. You immediately undestood "let's rewrite the whole company code" and advise "don't rewrite the whole project". Well, who said it's "a" project to begin with? Companies more often than not have tons of different services, and rewrites can be easily done for such isolated services (e.g. like the dl.google.com rewrite into Go). Tons of core services of YouTube, CloudFlare, Google, Facebook, and tons of others have been rewriten with languages as Go, Rust, Hack, etc. and they have written about the experience and results as a success. And it's not like survivor bias, e.g. the keep mum about failed cases, because we hear about failures all the time (just not concerning rewritings).

tinco 5y ago
I agree, not sure why you are down voted because you make valid points. I can't think of any big public disastrous rewrites except for maybe frontend ones like Digg or enterprise modernizations for example when the government wanted to get rid of their 60s mainframes and the effort cost billions and was ultimately unsuccessful.

ignoramous 5y ago

> This is madness. Pretty sure the article is tongue-in-cheek satire. > There's no real advice about rewrites in this article. The actual advice is in the top image which goes: - Todo: Add critical customer request, fix bugs, new features - In progress: Rewrite everything