5 ms·

The reality is that you have to be sensible and balanced. You can't keep BROKEN functions "because people rely on them", but you can't suddenly make a minor re

The reality is that you have to be sensible and balanced. You can't keep BROKEN functions "because people rely on them", but you can't suddenly make a minor release (e.g.: from 2.3.4 to 2.3.5) completely changing the semantic of a feature you expose. You need to: - Avoid changing APIs just for the sake of change. - Announce breaking changes. - Document how users are intended to migrate from your old API to the new one. Very large projects like Django do things like: - v2.0: Adds a new API, but keeps the old one - v3.0: Removes the old API. This gives consumers of the API plenty of time and margin to experiment with both and move on. If a single release changes too many things, it's very hard for a developer to upgrade, since they have a completely broken codebase, and need to fix multiple breakage at once.

MaxBarraclough 5y ago
> This gives consumers of the API plenty of time and margin to experiment with both and move on. It's not a silver bullet though, as we saw with the move to Python3. It annoyed people with large Python2 codebases, giving them the choice between: • Laboriously reworking their code to transition it to Python3 (a language with only limited advantages over Python2) • Using an unsupported language for a significant codebase
- WhyNotHugo 5y ago
  Python 2-3 is a great COUNTER example of what I'm saying. There was no intermediate release that could run both python2 and python3 code. It introduced new features AND removed the old ones all at once.
  - MaxBarraclough 5y ago
    > There was no intermediate release that could run both python2 and python3 code. Right, but would that have helped? Transitioning a large codebase from Python2 to Python3 is a significant undertaking, there's no way around it. As I understand it there are dedicated tools to help migrate Python code, but it's still no small thing if you're dealing with a large codebase. Perhaps it would have been helpful to have an interpreter that supported the union of both languages, but I don't see that this would have changed things from significantly costly down to no big deal. It would make the process more incremental, but it would still represent significant work. There are legitimate reasons people don't want to have to modify large stable codebases. (To be clear, I haven't had to do this personally, but that seems to be the sentiment of people who have.) C++, on the other hand, goes to considerable lengths to preserve backward compatibility, at the cost of accumulating complexity.
    - kevincox 5y ago
      It would have helped immensely. The main issues with the upgrade is that: 1. Libraries needed to maintain both versions (or code compatible with both. 2. Projects couldn't upgrade until their dependencies did. If Python 3 could run old code (for example you needed to write `import py3` at the top of a file to switch) then all projects could have updated their code in parallel, there would be no (or minimal) work to maintain support for old versions and everyone can update to py3 the day it came out. The workflow then looks like this: 1. Everyone updates to Py2.9 which supports the old features and the new features. 2. Everyone migrates their code to use the new features and remove deprecated features independently. 3. Py3 comes out and drops support for the old stuff. 1 is not trivial, but it is equivalent to any other time that you drop support for old versions, this happens regularly and isn't a huge event. Compare that to the nasty wave of updating dependencies and requirement to continue supporting old versions, especially since the code to support multiple versions was fairly awkward for a lot of projects.
    - jeltz 5y ago
      It helped for Ruby. Ruby released 1.8.6 and 1.9.1 which were both two intermediate steps between Ruby 1.8 and 2.0. 1.8.6 beibg similar to 1.8.4 but with many added 2.0 features and 1.9.1 which was almost 2.0 but with some deprecated features in it. I got to experience both and the Ruby 2 migration was much smoother than the Python 3 one. EDIT: To be clear, Ruby did not make it literally possible to run both 1.8 and 2.0 in the same code base but with just a few conditional statements and avoiding to use non-backported 2.0 features it was easy to write libraries which could run in both 1.8.6 and 1.9.
    - gwoplock 5y ago
      > Right, but would that have helped? Transitioning a large codebase from Python2 to Python3 is a significant undertaking, there's no way around it. I haven't migrated from python2->3 however I've worked on projects in both. I think a "python2+3" version would have helped. With both APIs in place new code can be written the new way and old code can be migrated as it is touched. Having the ability to incrementally change code I think encourages the transition since you won't end up with a pile of broken code.
  - PeterisP 5y ago
    You could transform your 2.x code so that it would run on both 3.x python and also on recent 2.x python versions, using new features with appropriate `from __future__ import` statements; so in that regard the latest 2.x versions were those intermediate releases that could also run 'new style' code. I had a bunch of internal code that would run on both 2.x and 3.x python, and many libraries also did just that.
    - jnxx 5y ago
      So why does the Python Wiki runs Python 2 with MoinMoin Wiki? You seem to imply it is easy and quick to rewrite software to match it to new versions, and resources for this are always available?
dgb23 5y ago
Depends on what you mean with "BROKEN". Breaking changes actually break other people's code, tutorials, documentation, tooling and developer knowledge often for very questionable gains. There are many projects that are relied on by thousands, but they keep pushing breaking changes on a regular basis without providing tangible improvements. This is especially true for tooling, frameworks etc. for web development.
- dTal 5y ago
  Hypothesis: this is especially true for corporate-sponsored open source where the devs also sell consulting services - i.e. "freemium" - because their incentive structure does not encourage making things easy to use without their services.
okamiueru 5y ago
Most of what you touch on is covered by semantic versioning. (https://semver.org https://semver.org), and IMO should be mandatory reading by anyone in software development. It sort of looks like you are sort of aware of it, but maybe not entirely. What you called a minor version is typically called a "patch" version. So it goes. MAJOR.MINOR.PATCH Major version bump can change anything. Minor versions are generally safe to upgrade, but will change the api in a backwards compatible way. Patch version updates make no change to the api. One of the worst things to deal with is libraries for which maintainers have broken the semver contract.
- ahartmetz 5y ago
  Qt is exemplary in that. There are ironclad rules that only MINOR adds new features and only MAJOR breaks API. And PATCH is strictly for bugfixes. Binary compatibility is guaranteed as well so that an application compiled against 5.0.0 will dynamically link against 5.15.2 and run fine unless it relied on clear bugs. There are exceptions for things like input method plugins, for which one needs to explicitly opt in to using internal API with build sysem flags. Without the opt-in, the corresponding private headers are not accessible.
- maweki 5y ago
  You have to define the notion of "backwards compatible" though, as this is the core of semver. To be strict, a bugfix is not backwards compatible. You explicitly change the behaviour of a function or interface. There's an interesting paper on checked semantic versioning [1] and you will get a new MAJOR version if you don't explicitly add the NOCOMPARE-pragma to a bugfix, as the prover (obviously) can not show the original function and the bugfixed function to be equal (usually they are not for some input). Backwards compatibility is hugely in the eye of the beholder (Hyrum's Law). So this is not a cut-and-dry issue. [1] https://drops.dagstuhl.de/opus/volltexte/2018/8456/ https://drops.dagstuhl.de/opus/volltexte/2018/8456/
  - okamiueru 5y ago
    I think the underlying assumption here is that this is versioning for humans (and package managers). It isn't 100% ironclad and fool proof. It really only tries to make the job better/easier for those who rely on other libraries by baking in an educated answer to the following question: > If I upgrade the library from X.Y.Z to A.B.C, what can I expect to have changed, and to what extent do I need to do anything after I make the upgrade to make my code work the same? I don't find it interesting that, technically, any observed change to the library can potentially break code. If I use reflection to branch on number of methods available, a bump in minor version can introduce a method to the API, and thus break my code. Making an observation of this is one thing, naming it after one self is a tad on the egotistical side. Other than that, I agree completely that the difficult part is defining what "backwards compatible" really means. However, to be clear, changing behavior is not the same as being backwards incompatible. I expect bugfixes to make my code function more correct and more reliably. I can reasonably expect minor versions to add functions or features that don't get in the way, and I can expect nothing from major version upgrades, so those will always need to be done with some extra care.

MaxBarraclough 5y ago

> This gives consumers of the API plenty of time and margin to experiment with both and move on. It's not a silver bullet though, as we saw with the move to Python3. It annoyed people with large Python2 codebases, giving them the choice between: • Laboriously reworking their code to transition it to Python3 (a language with only limited advantages over Python2) • Using an unsupported language for a significant codebase

WhyNotHugo 5y ago
Python 2-3 is a great COUNTER example of what I'm saying. There was no intermediate release that could run both python2 and python3 code. It introduced new features AND removed the old ones all at once.
- MaxBarraclough 5y ago
  > There was no intermediate release that could run both python2 and python3 code. Right, but would that have helped? Transitioning a large codebase from Python2 to Python3 is a significant undertaking, there's no way around it. As I understand it there are dedicated tools to help migrate Python code, but it's still no small thing if you're dealing with a large codebase. Perhaps it would have been helpful to have an interpreter that supported the union of both languages, but I don't see that this would have changed things from significantly costly down to no big deal. It would make the process more incremental, but it would still represent significant work. There are legitimate reasons people don't want to have to modify large stable codebases. (To be clear, I haven't had to do this personally, but that seems to be the sentiment of people who have.) C++, on the other hand, goes to considerable lengths to preserve backward compatibility, at the cost of accumulating complexity.
  - kevincox 5y ago
    It would have helped immensely. The main issues with the upgrade is that: 1. Libraries needed to maintain both versions (or code compatible with both. 2. Projects couldn't upgrade until their dependencies did. If Python 3 could run old code (for example you needed to write `import py3` at the top of a file to switch) then all projects could have updated their code in parallel, there would be no (or minimal) work to maintain support for old versions and everyone can update to py3 the day it came out. The workflow then looks like this: 1. Everyone updates to Py2.9 which supports the old features and the new features. 2. Everyone migrates their code to use the new features and remove deprecated features independently. 3. Py3 comes out and drops support for the old stuff. 1 is not trivial, but it is equivalent to any other time that you drop support for old versions, this happens regularly and isn't a huge event. Compare that to the nasty wave of updating dependencies and requirement to continue supporting old versions, especially since the code to support multiple versions was fairly awkward for a lot of projects.
  - jeltz 5y ago
    It helped for Ruby. Ruby released 1.8.6 and 1.9.1 which were both two intermediate steps between Ruby 1.8 and 2.0. 1.8.6 beibg similar to 1.8.4 but with many added 2.0 features and 1.9.1 which was almost 2.0 but with some deprecated features in it. I got to experience both and the Ruby 2 migration was much smoother than the Python 3 one. EDIT: To be clear, Ruby did not make it literally possible to run both 1.8 and 2.0 in the same code base but with just a few conditional statements and avoiding to use non-backported 2.0 features it was easy to write libraries which could run in both 1.8.6 and 1.9.
  - gwoplock 5y ago
    > Right, but would that have helped? Transitioning a large codebase from Python2 to Python3 is a significant undertaking, there's no way around it. I haven't migrated from python2->3 however I've worked on projects in both. I think a "python2+3" version would have helped. With both APIs in place new code can be written the new way and old code can be migrated as it is touched. Having the ability to incrementally change code I think encourages the transition since you won't end up with a pile of broken code.
- PeterisP 5y ago
  You could transform your 2.x code so that it would run on both 3.x python and also on recent 2.x python versions, using new features with appropriate `from __future__ import` statements; so in that regard the latest 2.x versions were those intermediate releases that could also run 'new style' code. I had a bunch of internal code that would run on both 2.x and 3.x python, and many libraries also did just that.
  - jnxx 5y ago
    So why does the Python Wiki runs Python 2 with MoinMoin Wiki? You seem to imply it is easy and quick to rewrite software to match it to new versions, and resources for this are always available?

dgb23 5y ago

Depends on what you mean with "BROKEN". Breaking changes actually break other people's code, tutorials, documentation, tooling and developer knowledge often for very questionable gains. There are many projects that are relied on by thousands, but they keep pushing breaking changes on a regular basis without providing tangible improvements. This is especially true for tooling, frameworks etc. for web development.

dTal 5y ago
Hypothesis: this is especially true for corporate-sponsored open source where the devs also sell consulting services - i.e. "freemium" - because their incentive structure does not encourage making things easy to use without their services.

okamiueru 5y ago

Most of what you touch on is covered by semantic versioning. (https://semver.org https://semver.org), and IMO should be mandatory reading by anyone in software development. It sort of looks like you are sort of aware of it, but maybe not entirely. What you called a minor version is typically called a "patch" version. So it goes. MAJOR.MINOR.PATCH Major version bump can change anything. Minor versions are generally safe to upgrade, but will change the api in a backwards compatible way. Patch version updates make no change to the api. One of the worst things to deal with is libraries for which maintainers have broken the semver contract.

ahartmetz 5y ago
Qt is exemplary in that. There are ironclad rules that only MINOR adds new features and only MAJOR breaks API. And PATCH is strictly for bugfixes. Binary compatibility is guaranteed as well so that an application compiled against 5.0.0 will dynamically link against 5.15.2 and run fine unless it relied on clear bugs. There are exceptions for things like input method plugins, for which one needs to explicitly opt in to using internal API with build sysem flags. Without the opt-in, the corresponding private headers are not accessible.
maweki 5y ago
You have to define the notion of "backwards compatible" though, as this is the core of semver. To be strict, a bugfix is not backwards compatible. You explicitly change the behaviour of a function or interface. There's an interesting paper on checked semantic versioning [1] and you will get a new MAJOR version if you don't explicitly add the NOCOMPARE-pragma to a bugfix, as the prover (obviously) can not show the original function and the bugfixed function to be equal (usually they are not for some input). Backwards compatibility is hugely in the eye of the beholder (Hyrum's Law). So this is not a cut-and-dry issue. [1] https://drops.dagstuhl.de/opus/volltexte/2018/8456/ https://drops.dagstuhl.de/opus/volltexte/2018/8456/
- okamiueru 5y ago
  I think the underlying assumption here is that this is versioning for humans (and package managers). It isn't 100% ironclad and fool proof. It really only tries to make the job better/easier for those who rely on other libraries by baking in an educated answer to the following question: > If I upgrade the library from X.Y.Z to A.B.C, what can I expect to have changed, and to what extent do I need to do anything after I make the upgrade to make my code work the same? I don't find it interesting that, technically, any observed change to the library can potentially break code. If I use reflection to branch on number of methods available, a bump in minor version can introduce a method to the API, and thus break my code. Making an observation of this is one thing, naming it after one self is a tad on the egotistical side. Other than that, I agree completely that the difficult part is defining what "backwards compatible" really means. However, to be clear, changing behavior is not the same as being backwards incompatible. I expect bugfixes to make my code function more correct and more reliably. I can reasonably expect minor versions to add functions or features that don't get in the way, and I can expect nothing from major version upgrades, so those will always need to be done with some extra care.