6 ms·
I'll highlight something I've learned in both succeeding and failing at this metric: When rewriting something, you should generally strive for a drop-in replace
by jerf 11y ago
I'll highlight something I've learned in both succeeding and failing at this metric: When rewriting something, you should generally strive for a drop-in replacement that does the same thing, in some cases, even matching bug-for-bug, or, as in the article, taking a very close look at the new vs. the old bugs.
It's tempting to throw away the old thing and write a brand new bright shiny thing with a new API and a new data models and generally NEW ALL THE THINGS!, but that is a high-risk approach that is usually without correspondingly high payoffs. The closer you can get to drop-in replacement, the happier you will be. You can then separate the risks of deployment vs. the new shiny features/bug fixes you want to deploy, and since risks tend to multiply rather than add, anything you can do to cut risks into two halves is still almost always a big win even if the "total risk" is still in some sense the same.
Took me a lot of years to learn this. (Currently paying for the fact that I just sorta failed to do a correct drop-in replacement because I was drop-in replacing a system with no test coverage, official semantics, or even necessarily agreement by all consumers what it was and how it works, let alone how it should work.)
- humanrebar 11y ago> ...that is a high-risk approach that is usually without correspondingly high payoffs That's probably true, but it's also true that over a long enough timescale (100 years, to trigger a reductio ad absurdum) there is a very high risk that not replacing or rewriting that code will sink your technology and possibly your organization. Just because the risk will be realized in the long run doesn't mean it's not a risk. And if the worst-case scenario is death of the entire organization, then the math could very well add up to a full rewrite. Most business managers are not prepared to think strategically about long-term technical debt. It's the duty of engineers to let them know the difference between "not now" and "never". And the difference between "urgent" and "low priority".
- mentat 11y agoIt boils down to "don't change the implementation and the interface at the same time". It doesn't say "don't ever change the interface".
- Avshalom 11y agoI think jerf was trying to say that when re-implementing something, re-implement it first then improve it, don't do both at once.
- jerf 11y agoBingo. That means even if your new pretty-shiny is going to eventually have a new API, it is usually worth it to still offer an old shim in with the old interface. It may seem like extra useless work, but it reduces risk. Far be it from me to suggest rewrites are never useful. Half my career could be characterized as "rewriting the things nobody thought could be rewritten because they were too messy and entrenched". It's personally a bit risky (you end up owning not only the bugs you created, but the bugs you failed to reproduce... doubly-whammy! better be good at unit testing so at least the new code is defensible), but the payoff is pretty significant, too, both for your career and for your organization.
- humanrebar 11y ago> re-implement it first then improve it Right. I'm saying (poorly, I guess) that that doesn't always work. If the technical debt is sprinkled throughout your system or your organization is sunk deeply into a broken paradigm, you're in trouble. Sometimes you need to provide a completely new technology that serves many of the same use cases in a different way. I'm not saying we should reach for rewriting things as our first instinct. I am saying that your COM interface will need to be replaced some day. Or your COBOL business logic will become a liability. Or your frames-based web API will cost you business. Incremental change leads you to a new local maximum, but sometimes that local maximum stunts your growth or even proves deadly.
- azernik 11y agoOne strategy that I've seen work for this kind of deep architectural change is to write the new system, then write a shim that provides a compatibility layer (however hacky and ugly) to emulate the old system. This lets you test the new system without then having to also test everything that the system interacts with. And then, start replacing usages of the shim with direct interaction with the newer prettier system. Think the win32 to NPAPI transition.
- ksk 11y ago>but it's also true that over a long enough timescale (100 years, to trigger a reductio ad absurdum) there is a very high risk that not replacing or rewriting that code will sink your technology and possibly your organization. Could you elaborate on what basis you claim this as a truth?
- humanrebar 11y agoIt's largely conjecture, admittedly. But there are very few pieces of software that last 40 years. Most companies don't plan to be out of business in the next 40 years. And many tech companies have gone out of business in even the last ten years because they weren't agile enough to adapt quickly to new technological changes.
- ksk 11y agoWhile I agree that in the abstract, technical debt can catch up to you, I'm just not sure that its impact is necessarily such that it cannot be contained or mitigated. I mean UNIX is still around for 30 odd years. It hasn't sunk. It carries tremendous technical debt, in terms of bad design, in terms of implementation bugs that need to be carried forward, etc. My sense is that the more organizations that depend on a piece of tech, the more chance there is that it is going to age well, warts and all.
- bcbrown 11y agoI don't think that argument holds. Usually it's true, that if something hasn't lasted a long time, it likely won't last a long time in the future. But we live at a quasi-special time, close to a boundary condition. There exists no software that has existed for 100 years, because software was invented sometime in the past hundred years. You can reductio ad absurdum that argument by saying, the week after the first software was written, that no pieces of software have lasted more than a week, and therefore very few pieces of software will last more than a week. So very few pieces of software have lasted 40 years, because very few pieces of software were written at least 40 years ago.
- halayli 11y agoThis is as true as it can get. I'd like to reemphasis the bug-for-bug. Sometimes that's the only way you can drop-in replace a system.
- makecheck 11y agoI think what works even better is to have "permission" to gradually change both the old and the new. It can drastically simplify the process of creating a replacement, if you're only replacing a slightly more sane version of the original instead of the actual original.
- robrenaud 11y agoThis is probably very context dependent, because I've learned the opposite. For example, I was rewriting/consolidating a corner of the local search logic for Google that was spread throughout multiple servers in the stack. Some of the implementation decisions were clearly made because of the convenience of doing so in a particular server. But when consolidating the code into a single server, the data structures and partial results available were not the same, so re-producing the exact same logic and behavior would have been hard. Realizing which parts of the initial implementation were there because it was convenient, and which were there for product concerns let me implement something much simpler that still satisfied the product demands, even if the output was not bitwise identical.
- acveilleux 11y agoI didn't read the parent comment as reproducing the exact same logic perfectly. More as a definition of the interface between the external code and the part to replace and matching that interface closely with the replacement. This isn't always possible but seems like a reasonable objective given my experience.
- tedunangst 11y ago> It hardly seems worth even having a bug system if the frequency of from-scratch rewrites always outstrips the pace of bug fixing. Why not be honest and resign yourself to the fact that version 0.8 is followed by version 0.8, which is then followed by version 0.8? https://www.jwz.org/doc/cadt.html https://www.jwz.org/doc/cadt.html
- colanderman 11y agoYou can break this down even more. As we speak, I'm "replacing" old code by just writing a wrapper around it with the new API it should have. Then I'll rewrite it without the wrapper, bug-for-bug. And then I'll actually fix the bugs.
- derefr 11y agoThere's a simpler method than this that provides even more surety, used by e.g. LibreSSL: 1. Start writing your new implementation (or heavily refactoring your old implementation, whichever), but in parallel, for each legacy function you remove, write an equivalent "legacy wrapper" function that implements the old API (and ABI; you have to return the same structs and all) in terms of the new API. 2. As you develop the new code, continue to run the old code's tests. (This shouldn't require any work; as far as the tests can tell, the codebase containing all of {the new code, what's left of the old code, and the legacy wrapper} presents exactly the same ABI as the old codebase.) The old tests should still all pass, at every step. 3. Once you're finished developing the new code, and all the old code's tests are passing, rewrite the tests in terms of the new API. 4. Split off all the legacy-wrapper code into a new, second library project; give it the new "core" library as a dependency. Copy all the old tests—from a commit before you rewrote them—into this project, too. This wrapper library can now be consumed in place of the original legacy library. Keeping this wrapper library up-to-date acts to ensure that your new code remains ABI-compatible; the old tests are now regression-tests on whether a change to the new "core" library breaks the legacy-ABI-wrapper library. 5. Document and release your new core as a separate, new library, and encourage devs to adopt it in place of the legacy library; release the legacy-wrapper (with its new-core dependency) as the next major version of the old library. When all-or-nearly-all downstream devs have transitioned from the legacy wrapper to the new core, you can stop supporting/updating the legacy wrapper and stop worrying about your updates breaking it. You're free! In LibreSSL, if you're wondering, the "new core" from above is called libtls, and the "legacy wrapper" from above is called libssl—which is, of course, the same linker name as OpenSSL's library, with a new major version.
- colanderman 11y agoInteresting! It seems almost as if the order doesn't matter, so long as each step is incremental and maintains the invariants of the previous step.
- AnimalMuppet 11y agoThere's always more use cases for the existing code than you think there are. There's always more corner cases handled by the existing code than you think there are. There's always more bug fixes in the existing code than you think there are. Combine all of those, and writing a replacement is always much harder than you expect it to be.
- asuffield 11y agoI've learned a variation on this theme, which is more specific: You have some code that, for whatever reason, you think is not very good. There is a reason why you have ended up with code that is not very good. If your action is to sit down and write it all again, you should anticipate getting a very similar result (and this has been the outcome of every "big rewrite" effort I've ever seen: they successfully reproduced all the major problems of the old system). The reasons why this happen are probably something to do with the way you're doing development work, and not the codebase that you're stuck with. Until you learn how to address those problems, you should not anticipate a better outcome. Once you have learned how to address those problems, you are likely to be able to correct the problem without doing a "big rewrite" (most commonly by fixing them one piece at a time). Sometimes I see people attempt a "big rewrite" after replacing all of the people, thinking that they can do a better job. The outcome of this appears to me to invariably be that the second team who tried to build the system with no real experience end up following a very similar path to the first team that did the same thing (guided by the map that the first team left them, and again reproducing all the same problems). From these observations I draw one key conclusion: the important thing that you get from taking smaller steps is that you amplify your ability to learn from things that have already been done, and avoid repeating the mistakes that were made the last time. The smaller the step, the easier it becomes to really understand how this went wrong last time and what to do instead. Yes, the old codebase is terrible, but it still contains vitally important knowledge: how to not do that again. Neither writing new code from scratch without touching the old, nor petting the old code for years without attempting to fix it, are effective ways to extract that knowledge. The only approach I've ever really seen work is some form of "take it apart, one piece at at time, understand it, and then change it".
- zatkin 11y agoThe biggest issue with new things is unknown unknowns.