6 ms·

I'm waiting for the day when we will finally admit to ourselves that the Semantic Versioning experiment has failed. The purpose of Semver is to allow software

by p-e-w 3y ago

I'm waiting for the day when we will finally admit to ourselves that the Semantic Versioning experiment has failed. The purpose of Semver is to allow software and people to determine, from the version number alone, whether and what has changed. But even though many projects claim adherence to the Semver standard nowadays, the ecosystem as a whole isn't, and never has been, anywhere close to the point where a reasonable engineer would actually rely on version numbers to decide whether to upgrade. Imagine you see that one of your library dependencies, which professes adherence to Semver, has released version 1.2.3, up from version 1.2.2 which your project currently uses. Would you feel comfortable doing the update, without checking whether the project still compiles and passes its test suite afterwards? If you answered "no", then you are basically admitting that Semver is worthless in practice. If you are going to test everything anyway, version numbers might as well be a simple incrementing integer, or a date, or a random string. Upgrading will still consist of trying the new version, and, on failure, rolling back – which is exactly what you are probably already doing anyway. Semver doesn't work because you can't actually trust that all your dependencies have verified their claimed Semver compliance to the extent that would be necessary for your project to rely on it. You have to do that verification yourself, which makes the Semver version number at best an occasionally useful hint. And that's not even talking about all the other problems, such as different projects having different interpretations of what constitutes a "breaking" change, and almost none of the older, established libraries even claiming to follow Semver in the first place.

IanCal 3y ago
> Would you feel comfortable doing the update, without checking whether the project still compiles and passes its test suite afterwards? I don't trust saving a file that hasn't changed without checking if things compile/pass tests. > If you answered "no", then you are basically admitting that Semver is worthless in practice I hugely disagree. Version numbers give me a good idea about what level of changes to expect. Upgrading from 1 to 2 implies a larger review is needed, I'd not just check compile/tests. Upgrading from 1.4.656 to 1.4.657 and I don't expect to have to check the docs for major changes to how things work. Dates, pure increments etc don't have this. Two releases a day apart can be wildly different - a patch release fixing a logging format and a fundamental API change. It's a method of communication . Not being perfect doesn't make it useless.
- lifthrasiir 3y ago
  > Upgrading from 1 to 2 implies a larger review is needed, I'd not just check compile/tests. Upgrading from 1.4.656 to 1.4.657 and I don't expect to have to check the docs for major changes to how things work. What would you do for upgrading from 1.4 to 1.5? > Dates, pure increments etc don't have this. Right, but most versioning schemes are not pure increments and nor pure dates exactly for that reason. You don't plan to release twice a day when your version scheme is year-based; the second number is frequently used for when it's inevitable, making it a hybrid scheme. Still not compatible with the semantic versioning. Semantic versioning also effectively shadowed a previously widespread decimal-based versioning (e.g. 1.00 < 1.09 < 1.5 < 1.99 < 2.0). There are pros and cons for both formats, but decimal-based versioning makes authors to pick a suitable increment for communicating changes, and forbids them to perpetually delay the major version (which can be a pro or a con, I personally think it's a pro). If semantic versioning simply wanted to codify and mechanize the best practice, it should have instead adopted Debian's [1] which tried a lot to unify existing conventions. [1] https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-version https://www.debian.org/doc/debian-policy/ch-controlfields.ht...
  - simonw 3y ago
    "What would you do for upgrading from 1.4 to 1.5?" I'd read the release notes, and spend a little bit of time thinking about if they impact my project. For a 1.4.1 to 1.4.2 bump, if my (comprehensive) tests pass, I'll land the upgrade.
    - lifthrasiir 3y ago
      Yes, the minor version clearly doesn't have much information. Then why should we keep the three-part version number after all?
      - IanCal 3y ago
        It does have information about what's changed. It shouldn't be a patch release and it's not supposed to be a breaking change. Outside of new things it informs you about the work expected with downgrades. Going down in minor versions and you should check for breaking changes or removed functionality.
Nanana909 3y ago
I think you're looking for too much out of a versioning scheme. If you're seeing 1.2.3 -> 2.2.3, updates, i..e the dev is saying THIS 100% BREAKS SOMETHING IN MY OPINION, and you just do your normal set of analysis and walk away happy that's a bit scary. If you do anything other than your normal analysis, you've just gone ahead and used semver.
c7b 3y ago
I don't think semver is meant to condition users into blindly trusting updates. I always thought it was meant to condition developers into being mindful about what the changes they make will mean for their users. You can maybe never be sure that some change won't lead to something breaking, but it's often quite clear when a change almost certainly will lead to something breaking.
littlestymaar 3y ago
> Would you feel comfortable doing the update, without checking whether the project still compiles and passes its test suite afterwards? > If you answered "no", then you are basically admitting that Semver is worthless in practice. Because everything that is not perfect is worthless indeed… Some people should really learn about nuance.
pie_flavor 3y ago
You say the experiment has failed, but speak purely of theory. The actual experiment, the one running in practice, is an astounding success; the things you describe happening don't happen, or at least don't happen in Rust. In practice, a dependency I should not trust to have done diligence in semver management is only a couple steps past a dependency I should not trust to run on my machine. Ignore any dependency with less than 50k downloads and 'what if other people are dumb' concerns like this one just melt away.
- eviks 3y ago
  The tool author found out that at least 1 in 6 Rust creates broke semver that the tool detected. That's a big number for an "astounding success"
  - pie_flavor 3y ago
    What's the ratio among Rust crates with over 50k downloads? I manage a tool with literally seven hundred dependencies and not one of them has ever broken semver.
    - eviks 3y ago
      How are you so certain if it could be a break in a code path you don't use? Could you check it with this tool?
      - obi1kenobi 3y ago
        cargo-semver-checks is not dependent on whether code paths are used, because it doesn't analyze from the "use" side. It analyzes from the "what API is exposed" side. You can think of cargo-semver-checks as an automated way to answer a series of queries like "are there any public functions that existed in the previous version and don't exist in the new version?" Any results produced by any of those queries are examples of breaking changes. This is why our "more than 1 in 6" study found so many previously-unreported semver violations — in our study we were able to find issues in all sorts of dark corners of library APIs, then generate programs specifically designed to rely on the affected behavior in a way that works fine in one API version and causes a compilation error in the subsequent (non-major) version.
ekidd 3y ago
As other people have said, I normally run unit tests after changing comments. It's part of my check-in routine. That said, I maintain close to a dozen Rust utilities, some of which only get updated every two or three years. And when I update them, I try to update all their dependencies. We're talking 250-400 transitive dependencies in most cases, and typically two years of evolution of all of them. In other words, this is a worst-case scenario for semver. I'm often looking at 600 "library-years" of supposedly compatible changes. 4 times out of 5, the "semver-compatible" updates work flawlessly. The 5th time, I'll occasionally run into a minor problem that takes me 20 minutes to clean up. Once every several years, I'll run into a genuine headache. To me, this is stunningly successful. Rust semver isn't perfect, but it's a genuine joy from a maintenance perspective. And I expect things to get even better, thanks to tools like the one in the OP.
- cratermoon 3y ago
  > I normally run unit tests after changing comments. It's part of my check-in routine. I run the tests after running even a part of a file through the formatter. And no, I don't work in Python.
- kibwen 3y ago
  > I normally run unit tests after changing comments Thirded. Tests get run before I commit, period. The grandparent's apparent argument that semver has failed because you have to run tests when upgrading patch versions is utterly alien to me. Has any semver proponent ever claimed such a thing?
koverstreet 3y ago
No, the purpose of semvar is to be able to distinguish compatible updates from incompatible updates. As a concept it's relevant any time you have a protocol or a communications channel between agents that aren't rigidly kept in sync. Think network protocols, on disk formats... The alternative to semvar in the filesystem world has historically been feature bits, but feature bits suck because there was an order in which you rolled out features and feature bits do not preserve that ordering. Meaning users can (and will, because users do crazy things) end up in a state where they have all the modern feature bits enabled but not the one from 5 years ago and you never tested that configuration. Learn semvar. Use semvar. Semvar good.
cratermoon 3y ago
> the Semantic Versioning experiment has failed I'm going to take a wild guess that you're going off of experience from something like npm library versions. That ecosystem is a mess, and I don't blame you for mistrusting it. On the other hand, the Go language has maintained the Go Compatibility Promise remarkably well. https://go.dev/doc/go1compat https://go.dev/doc/go1compat In other words, whether or no semver is useful depends on how well its followed. The versioning compatibility would be a problem regardless of the version number system followed. Case in point: Windows 95 v Windows 10.
petters 3y ago
> If you are going to test everything anyway, version numbers might as well be a simple incrementing integer, or a date, or a random string. You are perhaps right when you say that semver had failed, but what you list above as alternatives are less expressive. Semver admits a tree of versions (weak ordering) but your alternatives are strictly ordered.
eviks 3y ago
Yeah, versions should just convey the "marketing" information (how much has changed, how important those changes are), and all the API compatibility should be left to automation like this tool

by p-e-w 3y ago

IanCal 3y ago

> Would you feel comfortable doing the update, without checking whether the project still compiles and passes its test suite afterwards? I don't trust saving a file that hasn't changed without checking if things compile/pass tests. > If you answered "no", then you are basically admitting that Semver is worthless in practice I hugely disagree. Version numbers give me a good idea about what level of changes to expect. Upgrading from 1 to 2 implies a larger review is needed, I'd not just check compile/tests. Upgrading from 1.4.656 to 1.4.657 and I don't expect to have to check the docs for major changes to how things work. Dates, pure increments etc don't have this. Two releases a day apart can be wildly different - a patch release fixing a logging format and a fundamental API change. It's a method of communication . Not being perfect doesn't make it useless.

lifthrasiir 3y ago
> Upgrading from 1 to 2 implies a larger review is needed, I'd not just check compile/tests. Upgrading from 1.4.656 to 1.4.657 and I don't expect to have to check the docs for major changes to how things work. What would you do for upgrading from 1.4 to 1.5? > Dates, pure increments etc don't have this. Right, but most versioning schemes are not pure increments and nor pure dates exactly for that reason. You don't plan to release twice a day when your version scheme is year-based; the second number is frequently used for when it's inevitable, making it a hybrid scheme. Still not compatible with the semantic versioning. Semantic versioning also effectively shadowed a previously widespread decimal-based versioning (e.g. 1.00 < 1.09 < 1.5 < 1.99 < 2.0). There are pros and cons for both formats, but decimal-based versioning makes authors to pick a suitable increment for communicating changes, and forbids them to perpetually delay the major version (which can be a pro or a con, I personally think it's a pro). If semantic versioning simply wanted to codify and mechanize the best practice, it should have instead adopted Debian's [1] which tried a lot to unify existing conventions. [1] https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-version https://www.debian.org/doc/debian-policy/ch-controlfields.ht...
- simonw 3y ago
  "What would you do for upgrading from 1.4 to 1.5?" I'd read the release notes, and spend a little bit of time thinking about if they impact my project. For a 1.4.1 to 1.4.2 bump, if my (comprehensive) tests pass, I'll land the upgrade.
  - lifthrasiir 3y ago
    Yes, the minor version clearly doesn't have much information. Then why should we keep the three-part version number after all?
    - IanCal 3y ago
      It does have information about what's changed. It shouldn't be a patch release and it's not supposed to be a breaking change. Outside of new things it informs you about the work expected with downgrades. Going down in minor versions and you should check for breaking changes or removed functionality.

Nanana909 3y ago

I think you're looking for too much out of a versioning scheme. If you're seeing 1.2.3 -> 2.2.3, updates, i..e the dev is saying THIS 100% BREAKS SOMETHING IN MY OPINION, and you just do your normal set of analysis and walk away happy that's a bit scary. If you do anything other than your normal analysis, you've just gone ahead and used semver.

c7b 3y ago

I don't think semver is meant to condition users into blindly trusting updates. I always thought it was meant to condition developers into being mindful about what the changes they make will mean for their users. You can maybe never be sure that some change won't lead to something breaking, but it's often quite clear when a change almost certainly will lead to something breaking.

littlestymaar 3y ago

> Would you feel comfortable doing the update, without checking whether the project still compiles and passes its test suite afterwards? > If you answered "no", then you are basically admitting that Semver is worthless in practice. Because everything that is not perfect is worthless indeed… Some people should really learn about nuance.

pie_flavor 3y ago

You say the experiment has failed, but speak purely of theory. The actual experiment, the one running in practice, is an astounding success; the things you describe happening don't happen, or at least don't happen in Rust. In practice, a dependency I should not trust to have done diligence in semver management is only a couple steps past a dependency I should not trust to run on my machine. Ignore any dependency with less than 50k downloads and 'what if other people are dumb' concerns like this one just melt away.

eviks 3y ago
The tool author found out that at least 1 in 6 Rust creates broke semver that the tool detected. That's a big number for an "astounding success"
- pie_flavor 3y ago
  What's the ratio among Rust crates with over 50k downloads? I manage a tool with literally seven hundred dependencies and not one of them has ever broken semver.
  - eviks 3y ago
    How are you so certain if it could be a break in a code path you don't use? Could you check it with this tool?
    - obi1kenobi 3y ago
      cargo-semver-checks is not dependent on whether code paths are used, because it doesn't analyze from the "use" side. It analyzes from the "what API is exposed" side. You can think of cargo-semver-checks as an automated way to answer a series of queries like "are there any public functions that existed in the previous version and don't exist in the new version?" Any results produced by any of those queries are examples of breaking changes. This is why our "more than 1 in 6" study found so many previously-unreported semver violations — in our study we were able to find issues in all sorts of dark corners of library APIs, then generate programs specifically designed to rely on the affected behavior in a way that works fine in one API version and causes a compilation error in the subsequent (non-major) version.

ekidd 3y ago

As other people have said, I normally run unit tests after changing comments. It's part of my check-in routine. That said, I maintain close to a dozen Rust utilities, some of which only get updated every two or three years. And when I update them, I try to update all their dependencies. We're talking 250-400 transitive dependencies in most cases, and typically two years of evolution of all of them. In other words, this is a worst-case scenario for semver. I'm often looking at 600 "library-years" of supposedly compatible changes. 4 times out of 5, the "semver-compatible" updates work flawlessly. The 5th time, I'll occasionally run into a minor problem that takes me 20 minutes to clean up. Once every several years, I'll run into a genuine headache. To me, this is stunningly successful. Rust semver isn't perfect, but it's a genuine joy from a maintenance perspective. And I expect things to get even better, thanks to tools like the one in the OP.

cratermoon 3y ago
> I normally run unit tests after changing comments. It's part of my check-in routine. I run the tests after running even a part of a file through the formatter. And no, I don't work in Python.
kibwen 3y ago
> I normally run unit tests after changing comments Thirded. Tests get run before I commit, period. The grandparent's apparent argument that semver has failed because you have to run tests when upgrading patch versions is utterly alien to me. Has any semver proponent ever claimed such a thing?

koverstreet 3y ago

No, the purpose of semvar is to be able to distinguish compatible updates from incompatible updates. As a concept it's relevant any time you have a protocol or a communications channel between agents that aren't rigidly kept in sync. Think network protocols, on disk formats... The alternative to semvar in the filesystem world has historically been feature bits, but feature bits suck because there was an order in which you rolled out features and feature bits do not preserve that ordering. Meaning users can (and will, because users do crazy things) end up in a state where they have all the modern feature bits enabled but not the one from 5 years ago and you never tested that configuration. Learn semvar. Use semvar. Semvar good.

cratermoon 3y ago

> the Semantic Versioning experiment has failed I'm going to take a wild guess that you're going off of experience from something like npm library versions. That ecosystem is a mess, and I don't blame you for mistrusting it. On the other hand, the Go language has maintained the Go Compatibility Promise remarkably well. https://go.dev/doc/go1compat https://go.dev/doc/go1compat In other words, whether or no semver is useful depends on how well its followed. The versioning compatibility would be a problem regardless of the version number system followed. Case in point: Windows 95 v Windows 10.

petters 3y ago

> If you are going to test everything anyway, version numbers might as well be a simple incrementing integer, or a date, or a random string. You are perhaps right when you say that semver had failed, but what you list above as alternatives are less expressive. Semver admits a tree of versions (weak ordering) but your alternatives are strictly ordered.

eviks 3y ago

Yeah, versions should just convey the "marketing" information (how much has changed, how important those changes are), and all the API compatibility should be left to automation like this tool