4 ms·

I've pulled off several large, successful rewrites so I'm going to respectfully disagree, even though I agree I have seen exactly the failed rewrites that would

by fl0ki 3y ago

I've pulled off several large, successful rewrites so I'm going to respectfully disagree, even though I agree I have seen exactly the failed rewrites that would motivate someone to ban rewrites. The crux of my argument is that some projects really are costly enough to maintain that a rewrite is an overall lower cost (pricing in risk), while others are fixable in-place and a rewrite is an unnecessary cost, and it's rarely obvious which is which for all of the same reasons that project planning and cost estimation are infamously difficult. > The important part is that any improvement needs to be broken up into small enough pieces that each can be shipped separately. You don't want half done, unshipped work to be a continuing feature of your development process putting an overhead on everything you do. This is an ideal but it should not be a requirement. If you "need" this for any improvement, you're going to miss out on a lot of the biggest possible improvements because they have exactly the far-reaching impacts that make them harder to pull off but also worth much more when you do merge them. If you reject any such refactorings, you'll only improve in small local ways and never large global ones. Best case, you can try to get the best of both worlds by supporting both old & new interfaces to the same improved implementation, slowly migrating old edges to new ones. That's also just an ideal, and there are many ways it can prove impractical, e.g. subtle divergence between old and new types which is useful for the new implementation but makes it harder to interoperate with the old one. The fact is, sometimes a design has a big enough problem that a large change will pay off, and sometimes an implementation can be bad enough that this change cannot safely be made within the existing implementation. Usually when I've seen those, it's because the regression testing was so inadequate that you can't make changes with confidence, and yet the code isn't factored in a way to introduce the testing without the refactoring itself facing risk of regression, etc. and it's a total deadlock that destroys a project. Many, many real world projects end up in this state. Often it's because leadership prioritized deadlines more than quality, and promised to "fix it once it's launched", but then nobody was confident changing something that sorta kinda worked. If you don't invest in regression testing from the start, it'll always be too risky to add later. At that point, you may as well build a new project factored for safe maintainability including its own regression testing that will pay off forever, including testing that it does not regress on any use cases you can reproduce from the old project. You have to do something like this to break the deadlock, so it may as well fix other deficiencies as well. I have saved several mission-critical FAANG projects this way, and even my managers agreed that it was a huge success despite the general resistance to rewriting large projects. It even takes less time than people will assume, because once you factor the new project for confident maintenance without regressions, you become far more productive working on it than anyone could ever be on the old project. You get there sooner than you expect to, and it pays off more than anyone can imagine because they're so used to the problems of the old project. I'd also like to add that while a bad project can limp along for many years, it faces a different kind of problem good managers should fear. Only a very capable engineer can maintain such a project with a low defect rate, but they do it with great stress and frustration on their end, because truly poor maintainability hurts even the best engineers. The better the engineer, the more they feel the deficiencies of the project, and the more likely they are to want to leave. An engineer like that MAY be able to pull off a rewrite, but if you ban it, they're more likely to leave than play along.

majormajor 3y ago
> Often it's because leadership prioritized deadlines more than quality, and promised to "fix it once it's launched", but then nobody was confident changing something that sorta kinda worked. If you don't invest in regression testing from the start, it'll always be too risky to add later. At that point, you may as well build a new project factored for safe maintainability including its own regression testing that will pay off forever, including testing that it does not regress on any use cases you can reproduce from the old project. You have to do something like this to break the deadlock, so it may as well fix other deficiencies as well. I'm not sure what you mean about it being too risky to add regression testing later? It sounds like one of your first steps in the rewrite is building such a test suite for "any use cases you can reproduce from the old project" but in my experience it's hard to make the rewrite-or-not decision until you have that. Only after the investigation to understand the features and behavior do you have enough info to know if you can fix it in-place or not. (And "in-place" may still be effectively "complete rewrite", just one module at a time in a single codebase instead of in a new one.)
- fl0ki 3y ago
  > I'm not sure what you mean about it being too risky to add regression testing later? Sure, let me expand on that. It's uncontroversial that code has to be factored for testing, projects don't usually end up easily testable by accident. You can't necessarily write a testing harness around a project as a black box. How repeatable is its output at all, how easily can you add new inputs and expected outputs, how long does it take to run, how many other dependencies does it have and how easy are they to isolate, etc. Some types of projects are more amenable to this than others. In general, if it is a CLI with some input files and some output files, and the output is always deterministic from the input, then regression testing is as simple as collecting some representative inputs and outputs. Services with stateful APIs and multiple backend dependencies are just about the opposite of this, and require a lot of engineering to harness at all. I have inherited a number of HTTP and RPC services where the outputs were not repeatable at all, for a mix of good and bad reasons. For example, generating random IDs as part of the output sounds like "best practices", but it makes it much more difficult to test a chain of related requests (where the IDs have to be consistent throughout) and assert their responses. At best, it requires a lot of test driver code which is itself hard to maintain. Even including timestamps can interfere with this, though that's generally easier to mock out than random IDs are. That's to say nothing of how many projects think it's fine to create a bunch of global state, often including global "helper" threads that can't be explicitly controlled by anything else. These either have no effect on testing and thus aren't actually tested in any meaningful way, or they have strictly bad effects and make things less deterministic and hygienic. So if you want to refactor a project to be more deterministic, have fewer and simpler dependency edges to mock out, have no global state or background threads, etc. then that can add up to a huge refactor that, in itself, risks subtle regressions which you don't yet have regression tests to catch. That's what keeps happening. You can't have thorough regression testing until it's hygienic and deterministic, but refactoring it to be hygienic and deterministic risks regressions that you aren't ready to test for. Breaking this cycle requires creating deterministic regression tests that you can't yet run against your existing project because it's not deterministic yet. One way to do this is by manually capturing a bunch of real examples (using real data, to cover the edge cases that synthetic tests miss), making a deterministic version of each, and making that a test case for the new version of the code. Then you rewrite or refactor the code until it's deterministic enough to pass these tests. If the existing implementation is a tangled mess, and it usually is, a rewrite can easily be more practical than a refactor. I contend that it's not even enough to make it pass each case individually. To prove that the logic is actually hygienic, you have to engineer it to pass all cases as if they were real parallel transactions in a single instance of the system, which also limits the test case design because they have to be independent enough in that dimension while still being inter-dependent enough to prove the corresponding logic works. I've done all of that and more for a single complex backend with lots of mutation APIs, many having pages of business logic, operating on over a dozen different entity types with subtly different storage- and API-facing schemas, using real requests and responses running in parallel with real threads. It's a huge investment, but after building the test driver itself, all code added to the project has been extremely easy to test and not a single regression has made it past this testing. As much work as this was, it took less work than some individual small refactorings did to the original project, and now it just pays off every day forever. This is not an isolated example, but it's my best and most recent. > It sounds like one of your first steps in the rewrite is building such a test suite for "any use cases you can reproduce from the old project" Yeah, but one of my points it that you can't even run this against the old project if it's not deterministic and hygienic yet. It's a set of potential test cases for a future refactor/rewrite that's more testable. Once you start down this road, you have to push through to completion, because every change made to the untestable old project is another way the test cases can fall out of sync and risk the new project diverging. > but in my experience it's hard to make the rewrite-or-not decision until you have that. Only after the investigation to understand the features and behavior do you have enough info to know if you can fix it in-place or not. I totally agree, and that's why this might only be possible a year or more into the project. I don't think anybody should attempt this when they first take over a project. I do think people should be free to make this call once they're familiar with a project, and leadership outright banning rewrites out of principle cannot possibly help with that.

by fl0ki 3y ago

majormajor 3y ago

> Often it's because leadership prioritized deadlines more than quality, and promised to "fix it once it's launched", but then nobody was confident changing something that sorta kinda worked. If you don't invest in regression testing from the start, it'll always be too risky to add later. At that point, you may as well build a new project factored for safe maintainability including its own regression testing that will pay off forever, including testing that it does not regress on any use cases you can reproduce from the old project. You have to do something like this to break the deadlock, so it may as well fix other deficiencies as well. I'm not sure what you mean about it being too risky to add regression testing later? It sounds like one of your first steps in the rewrite is building such a test suite for "any use cases you can reproduce from the old project" but in my experience it's hard to make the rewrite-or-not decision until you have that. Only after the investigation to understand the features and behavior do you have enough info to know if you can fix it in-place or not. (And "in-place" may still be effectively "complete rewrite", just one module at a time in a single codebase instead of in a new one.)

fl0ki 3y ago
> I'm not sure what you mean about it being too risky to add regression testing later? Sure, let me expand on that. It's uncontroversial that code has to be factored for testing, projects don't usually end up easily testable by accident. You can't necessarily write a testing harness around a project as a black box. How repeatable is its output at all, how easily can you add new inputs and expected outputs, how long does it take to run, how many other dependencies does it have and how easy are they to isolate, etc. Some types of projects are more amenable to this than others. In general, if it is a CLI with some input files and some output files, and the output is always deterministic from the input, then regression testing is as simple as collecting some representative inputs and outputs. Services with stateful APIs and multiple backend dependencies are just about the opposite of this, and require a lot of engineering to harness at all. I have inherited a number of HTTP and RPC services where the outputs were not repeatable at all, for a mix of good and bad reasons. For example, generating random IDs as part of the output sounds like "best practices", but it makes it much more difficult to test a chain of related requests (where the IDs have to be consistent throughout) and assert their responses. At best, it requires a lot of test driver code which is itself hard to maintain. Even including timestamps can interfere with this, though that's generally easier to mock out than random IDs are. That's to say nothing of how many projects think it's fine to create a bunch of global state, often including global "helper" threads that can't be explicitly controlled by anything else. These either have no effect on testing and thus aren't actually tested in any meaningful way, or they have strictly bad effects and make things less deterministic and hygienic. So if you want to refactor a project to be more deterministic, have fewer and simpler dependency edges to mock out, have no global state or background threads, etc. then that can add up to a huge refactor that, in itself, risks subtle regressions which you don't yet have regression tests to catch. That's what keeps happening. You can't have thorough regression testing until it's hygienic and deterministic, but refactoring it to be hygienic and deterministic risks regressions that you aren't ready to test for. Breaking this cycle requires creating deterministic regression tests that you can't yet run against your existing project because it's not deterministic yet. One way to do this is by manually capturing a bunch of real examples (using real data, to cover the edge cases that synthetic tests miss), making a deterministic version of each, and making that a test case for the new version of the code. Then you rewrite or refactor the code until it's deterministic enough to pass these tests. If the existing implementation is a tangled mess, and it usually is, a rewrite can easily be more practical than a refactor. I contend that it's not even enough to make it pass each case individually. To prove that the logic is actually hygienic, you have to engineer it to pass all cases as if they were real parallel transactions in a single instance of the system, which also limits the test case design because they have to be independent enough in that dimension while still being inter-dependent enough to prove the corresponding logic works. I've done all of that and more for a single complex backend with lots of mutation APIs, many having pages of business logic, operating on over a dozen different entity types with subtly different storage- and API-facing schemas, using real requests and responses running in parallel with real threads. It's a huge investment, but after building the test driver itself, all code added to the project has been extremely easy to test and not a single regression has made it past this testing. As much work as this was, it took less work than some individual small refactorings did to the original project, and now it just pays off every day forever. This is not an isolated example, but it's my best and most recent. > It sounds like one of your first steps in the rewrite is building such a test suite for "any use cases you can reproduce from the old project" Yeah, but one of my points it that you can't even run this against the old project if it's not deterministic and hygienic yet. It's a set of potential test cases for a future refactor/rewrite that's more testable. Once you start down this road, you have to push through to completion, because every change made to the untestable old project is another way the test cases can fall out of sync and risk the new project diverging. > but in my experience it's hard to make the rewrite-or-not decision until you have that. Only after the investigation to understand the features and behavior do you have enough info to know if you can fix it in-place or not. I totally agree, and that's why this might only be possible a year or more into the project. I don't think anybody should attempt this when they first take over a project. I do think people should be free to make this call once they're familiar with a project, and leadership outright banning rewrites out of principle cannot possibly help with that.