3 ms·

You can break this down even more. As we speak, I'm "replacing" old code by just writing a wrapper around it with the new API it should have. Then I'll rewrit

You can break this down even more. As we speak, I'm "replacing" old code by just writing a wrapper around it with the new API it should have. Then I'll rewrite it without the wrapper, bug-for-bug. And then I'll actually fix the bugs.

derefr 11y ago
There's a simpler method than this that provides even more surety, used by e.g. LibreSSL: 1. Start writing your new implementation (or heavily refactoring your old implementation, whichever), but in parallel, for each legacy function you remove, write an equivalent "legacy wrapper" function that implements the old API (and ABI; you have to return the same structs and all) in terms of the new API. 2. As you develop the new code, continue to run the old code's tests. (This shouldn't require any work; as far as the tests can tell, the codebase containing all of {the new code, what's left of the old code, and the legacy wrapper} presents exactly the same ABI as the old codebase.) The old tests should still all pass, at every step. 3. Once you're finished developing the new code, and all the old code's tests are passing, rewrite the tests in terms of the new API. 4. Split off all the legacy-wrapper code into a new, second library project; give it the new "core" library as a dependency. Copy all the old tests—from a commit before you rewrote them—into this project, too. This wrapper library can now be consumed in place of the original legacy library. Keeping this wrapper library up-to-date acts to ensure that your new code remains ABI-compatible; the old tests are now regression-tests on whether a change to the new "core" library breaks the legacy-ABI-wrapper library. 5. Document and release your new core as a separate, new library, and encourage devs to adopt it in place of the legacy library; release the legacy-wrapper (with its new-core dependency) as the next major version of the old library. When all-or-nearly-all downstream devs have transitioned from the legacy wrapper to the new core, you can stop supporting/updating the legacy wrapper and stop worrying about your updates breaking it. You're free! In LibreSSL, if you're wondering, the "new core" from above is called libtls, and the "legacy wrapper" from above is called libssl—which is, of course, the same linker name as OpenSSL's library, with a new major version.
- colanderman 11y ago
  Interesting! It seems almost as if the order doesn't matter, so long as each step is incremental and maintains the invariants of the previous step.
- EvanPlaice 11y ago
  This is a textbook example of 'How to Kill an OSS project' #1, #4, and #5 are completely unnecessary. If it's desirable to 'kill with fire' the old codebase, you could always create a fork and merge the breaking changes on the next major release. Creating a second library is a terrible idea for 2 reasons... You lose the legacy source control history which is (arguably) more valuable than the current source because it can be used to research solutions to old problems. You split the community which is devastating to the culture and survivability of an OSS project. Even something as a simple name change will have massive negative impacts on the level of contributions. Only the most popular and actively developed projects can get away with forking the community. LibreSSL will likely survive because everything in the world that requires crypto uses OpenSSL. Even then, that was the absolutely wrong way to go about things. The only solid justification for a rename and complete rewrite, is if there are license/copyright issues.
  - derefr 11y ago
    > You lose the legacy source control history which is (arguably) more valuable than the current source because it can be used to research solutions to old problems. No reason for that. Both projects—the wrapper and the new core—can branch off the original. Create a commit that removes one half of the files on one side, and the other half of the files on the other, and make each new commit "master" of its repo, and now you've got two projects with one ancestor. Project mitosis. > You split the community How so? I'm presuming a scenario here where either 1. you were the sole maintainer for the old code, and it's become such a Big Ball of Mud that nothing's getting done; or 2. the maintainer of the old code is someone else who is really, really bad at their job, and you're "forking to replace" with community buy-in (OpenSSL, Node.js leading to the io.js fork, gcc circa 1997 with the egcs fork, MySQL leading to MariaDB, etc.). In both scenarios, development of the old code has already basically slowed to a halt. There is no active community contributing to it; or if there is, it is with great disgust and trepidation, mostly just engineers at large companies that have to fix upstream bugs to get their own code working (i.e. "I'm doing it because they pay me.") There are a lot of privately-maintained forks internal to companies, too, sharing around patches the upstream just won't accept for some reason. The ecosystem around the project is unhealthy†. When you release the new legacy wrapper, it replaces the old library—the legacy wrapper is now the only supported "release" of the old library. It's there as a stopgap for Enterprise Clients with effectively-dead projects which have ossified around the old library's ABI, so these projects can continue to be kept current with security updates et al. It's not there for anyone to choose as a target for their new project! No new features will ever be added to the wrapper. It's a permanent Long-Term Support release, with (elegant, automatic) backporting of security updates, and that's it. Nobody starting a project would decide to build on it any more than they'd build on e.g. Apache 1.3, or Ubuntu 12.04. > Even something as a simple name change will have massive negative impacts on the level of contributions. Names are IP, obviously (so if you're a third party, you have to rename the project), but they're more than that—names are associated in our brains with reflexes and conventions for how we build things. The reason Perl 6, Python 3, etc. have so much trouble with adoption is that people come into them expecting to be able to reuse the muscle-memory of the APIs of Perl 5/Python 2. They'd have been much better off marketed as completely new languages, that happened to be package-ecosystem-compatible with the previous language, like Elixir is to Erlang or Clojure is to Java. If these releases were accompanied by their creators saying "Python/Perl is dead, long live _____!" then there'd have been a much more dramatic switchover to the new APIs. Managers understand "the upstream is dead and we have to switch" much more easily than they understand "the upstream has a new somewhat-incompatible major version with some great benefits." One good example: there's a reason Swift wasn't released as "Objective-C 3.0". As it is, ObjC is "obviously dead" (even though Apple hasn't said anything to that effect!) and Swift is "the thing everyone will be using from here on, so we'd better move over to it." In a parallel reality, we'd have this very slow shift from ObjC2 to ObjC3 that would never fully complete. --- † If the ecosystem were healthy, obviously you don't need the legacy wrapper. As you say, just release the new library as the new major version of the old library—or call the new library "foo2", as many projects have done—and tell people to switch, and they will. It's easy to find healthy projects like this when you live close enough to the cutting-edge that all your downstream consumers are still in active development, possibly pre-1.0 development. The Node, Elixir, Go and Rust communities look a lot like this right now; any project can just "restart" and that doesn't trouble anybody. Everyone rewrites bits and pieces of their code all the time to track their upstreams' fresh-new-hotness APIs. That's a lot of what people mean when they talk about using a "hip new language": the fact that they won't have to deal with stupid APIs for very long, because stupid APIs get replaced. But imagine trying to do the same thing to, say, C#, or Java, or any other language with Enterprise barnacles. Imagine trying to tell people consuming Java's DateTime library that "the version of DateTime in Java 9 is now JodaTime, and everyone has to rewrite their date-handling code to use the JodaTime API." While the end results would probably have 10x fewer bugs, because JodaTime is an excellent API whose UX makes the pertinent questions obvious and gives devs the right intuitions... a rewrite like that just ain't gonna happen. Java 9 needs a DateTime that looks and acts like DateTime.
fleitz 11y ago
Yup, this is how I do it too, generally there's an implementation that needs an API and a new implementation. I've generally found that do the smallest possible thing that results in an improvement to be the best way forward.
EvanPlaice 11y ago
Sounds like a perfectly reasonable deprecation strategy. To go a bit further... Document and mark the old endpoints for deletion on the next major release. Mark the old API endpoints as deprecated in minor releases when the new implementation is completed and the old API endpoint is wrapped. Isn't that the whole purpose of SemVer?

derefr 11y ago

There's a simpler method than this that provides even more surety, used by e.g. LibreSSL: 1. Start writing your new implementation (or heavily refactoring your old implementation, whichever), but in parallel, for each legacy function you remove, write an equivalent "legacy wrapper" function that implements the old API (and ABI; you have to return the same structs and all) in terms of the new API. 2. As you develop the new code, continue to run the old code's tests. (This shouldn't require any work; as far as the tests can tell, the codebase containing all of {the new code, what's left of the old code, and the legacy wrapper} presents exactly the same ABI as the old codebase.) The old tests should still all pass, at every step. 3. Once you're finished developing the new code, and all the old code's tests are passing, rewrite the tests in terms of the new API. 4. Split off all the legacy-wrapper code into a new, second library project; give it the new "core" library as a dependency. Copy all the old tests—from a commit before you rewrote them—into this project, too. This wrapper library can now be consumed in place of the original legacy library. Keeping this wrapper library up-to-date acts to ensure that your new code remains ABI-compatible; the old tests are now regression-tests on whether a change to the new "core" library breaks the legacy-ABI-wrapper library. 5. Document and release your new core as a separate, new library, and encourage devs to adopt it in place of the legacy library; release the legacy-wrapper (with its new-core dependency) as the next major version of the old library. When all-or-nearly-all downstream devs have transitioned from the legacy wrapper to the new core, you can stop supporting/updating the legacy wrapper and stop worrying about your updates breaking it. You're free! In LibreSSL, if you're wondering, the "new core" from above is called libtls, and the "legacy wrapper" from above is called libssl—which is, of course, the same linker name as OpenSSL's library, with a new major version.

colanderman 11y ago
Interesting! It seems almost as if the order doesn't matter, so long as each step is incremental and maintains the invariants of the previous step.
EvanPlaice 11y ago
This is a textbook example of 'How to Kill an OSS project' #1, #4, and #5 are completely unnecessary. If it's desirable to 'kill with fire' the old codebase, you could always create a fork and merge the breaking changes on the next major release. Creating a second library is a terrible idea for 2 reasons... You lose the legacy source control history which is (arguably) more valuable than the current source because it can be used to research solutions to old problems. You split the community which is devastating to the culture and survivability of an OSS project. Even something as a simple name change will have massive negative impacts on the level of contributions. Only the most popular and actively developed projects can get away with forking the community. LibreSSL will likely survive because everything in the world that requires crypto uses OpenSSL. Even then, that was the absolutely wrong way to go about things. The only solid justification for a rename and complete rewrite, is if there are license/copyright issues.
- derefr 11y ago
  > You lose the legacy source control history which is (arguably) more valuable than the current source because it can be used to research solutions to old problems. No reason for that. Both projects—the wrapper and the new core—can branch off the original. Create a commit that removes one half of the files on one side, and the other half of the files on the other, and make each new commit "master" of its repo, and now you've got two projects with one ancestor. Project mitosis. > You split the community How so? I'm presuming a scenario here where either 1. you were the sole maintainer for the old code, and it's become such a Big Ball of Mud that nothing's getting done; or 2. the maintainer of the old code is someone else who is really, really bad at their job, and you're "forking to replace" with community buy-in (OpenSSL, Node.js leading to the io.js fork, gcc circa 1997 with the egcs fork, MySQL leading to MariaDB, etc.). In both scenarios, development of the old code has already basically slowed to a halt. There is no active community contributing to it; or if there is, it is with great disgust and trepidation, mostly just engineers at large companies that have to fix upstream bugs to get their own code working (i.e. "I'm doing it because they pay me.") There are a lot of privately-maintained forks internal to companies, too, sharing around patches the upstream just won't accept for some reason. The ecosystem around the project is unhealthy†. When you release the new legacy wrapper, it replaces the old library—the legacy wrapper is now the only supported "release" of the old library. It's there as a stopgap for Enterprise Clients with effectively-dead projects which have ossified around the old library's ABI, so these projects can continue to be kept current with security updates et al. It's not there for anyone to choose as a target for their new project! No new features will ever be added to the wrapper. It's a permanent Long-Term Support release, with (elegant, automatic) backporting of security updates, and that's it. Nobody starting a project would decide to build on it any more than they'd build on e.g. Apache 1.3, or Ubuntu 12.04. > Even something as a simple name change will have massive negative impacts on the level of contributions. Names are IP, obviously (so if you're a third party, you have to rename the project), but they're more than that—names are associated in our brains with reflexes and conventions for how we build things. The reason Perl 6, Python 3, etc. have so much trouble with adoption is that people come into them expecting to be able to reuse the muscle-memory of the APIs of Perl 5/Python 2. They'd have been much better off marketed as completely new languages, that happened to be package-ecosystem-compatible with the previous language, like Elixir is to Erlang or Clojure is to Java. If these releases were accompanied by their creators saying "Python/Perl is dead, long live _____!" then there'd have been a much more dramatic switchover to the new APIs. Managers understand "the upstream is dead and we have to switch" much more easily than they understand "the upstream has a new somewhat-incompatible major version with some great benefits." One good example: there's a reason Swift wasn't released as "Objective-C 3.0". As it is, ObjC is "obviously dead" (even though Apple hasn't said anything to that effect!) and Swift is "the thing everyone will be using from here on, so we'd better move over to it." In a parallel reality, we'd have this very slow shift from ObjC2 to ObjC3 that would never fully complete. --- † If the ecosystem were healthy, obviously you don't need the legacy wrapper. As you say, just release the new library as the new major version of the old library—or call the new library "foo2", as many projects have done—and tell people to switch, and they will. It's easy to find healthy projects like this when you live close enough to the cutting-edge that all your downstream consumers are still in active development, possibly pre-1.0 development. The Node, Elixir, Go and Rust communities look a lot like this right now; any project can just "restart" and that doesn't trouble anybody. Everyone rewrites bits and pieces of their code all the time to track their upstreams' fresh-new-hotness APIs. That's a lot of what people mean when they talk about using a "hip new language": the fact that they won't have to deal with stupid APIs for very long, because stupid APIs get replaced. But imagine trying to do the same thing to, say, C#, or Java, or any other language with Enterprise barnacles. Imagine trying to tell people consuming Java's DateTime library that "the version of DateTime in Java 9 is now JodaTime, and everyone has to rewrite their date-handling code to use the JodaTime API." While the end results would probably have 10x fewer bugs, because JodaTime is an excellent API whose UX makes the pertinent questions obvious and gives devs the right intuitions... a rewrite like that just ain't gonna happen. Java 9 needs a DateTime that looks and acts like DateTime.

fleitz 11y ago

Yup, this is how I do it too, generally there's an implementation that needs an API and a new implementation. I've generally found that do the smallest possible thing that results in an improvement to be the best way forward.

EvanPlaice 11y ago

Sounds like a perfectly reasonable deprecation strategy. To go a bit further... Document and mark the old endpoints for deletion on the next major release. Mark the old API endpoints as deprecated in minor releases when the new implementation is completed and the old API endpoint is wrapped. Isn't that the whole purpose of SemVer?