5 ms·

Semver violations are common, better tooling is the answer

77 points by steveklabnik 3y ago

o11c 3y ago
One thing that's often not mentioned is - how to deal with experimental APIs and such? You don't want to mark them as experimental indefinitely, and you don't want to force an upgrade if the experimental API didn't change when stabilizing. The best solution I've found is: each compiled library doesn't just have a single API, but rather has multiple. libfoo-1.2.3 might implement the following APIs: * libfoo-1.0 * libfoo-1.1 * libfoo-1.2 * libfoo-1.2.1 (because people don't actually use the "no API additions in patch releases" rule) * libfoo-1.2.2 * libfoo-1.2.3 * libfoo-experimental-1_2_3 (no semver in the version here) and if you enable experimental exports for the library, you gain a package-manager-level dependency on libfoo-experimental-1_2_3. Then later you can declare "libfoo-1.3 includes libfoo-experimental-1_2_3", so old programs can use newer libraries without change. That said, mutable metadata is still an important part of the ecosystem. Blacklisting is essential (since the symbol-list isn't actually the whole API and it is impossible to automatically detect violations), but easily abused for "that's insecure" and "I don't support that" purposes so there should be some friction. Note also that ABI (and sometimes API) is fundamentally platform-specific. And there's enough complexity in the world that the tooling shouldn't think in terms of semver, but rather arbitrary compatibility DAGs (but with sparseness when possible). Consider the GCC 4.7.0/4.7.1 std::list breakage that was reverted in 4.7.2, breaking software that was compiled in the mean time (the only reason this wasn't catastrophic is because nobody actually uses std::list) - that intermediate software needs to know it can't use new binaries.
- obi1kenobi 3y ago
  In Rust, this is often done by using optional features, and usually features named like `unstable-<something>`. They are a form of conditional compilation: people that don't opt in don't see that code, and people that do opt in know the code is unstable. `cargo-semver-checks` by default doesn't check for any semver obligations for code behind such features. (I'm its maintainer )
  - epage 3y ago
    And we'd like to have native support for these in cargo: https://github.com/rust-lang/cargo/issues/10881 https://github.com/rust-lang/cargo/issues/10881
  - o11c 3y ago
    The problem with the current approach is that it does not specify whether `libfoo-1.2.3[unstable-bar]` is compatible with `libfoo-1.3.4[unstable-bar]`, nor whether the latter is compatible with `libfoo-1.4.5[default]`. (also I'm not sure under what circumstances recompilation is required)
    - conradludgate 3y ago
      The rust std lib doesn't make any commitments to unstable (nightly) APIs, so this is the norm.
      - o11c 3y ago
        It doesn't formally make commitments. But in practice for most particular APIs there is stability, and it is the job of tooling to support that.
andrewstuart 3y ago
Forget the complexity, just use integer version numbers.
- Ar-Curunir 3y ago
  So every change is a breaking change? Not sure that helps anybody
  - SkyPuncher 3y ago
    I agree. Semver isn’t perfect, but it gives you a general idea of how hard an upgrade might be. That’s even more true with the most popular of libraries. For example, Rails and React are the two most important dependencies in our system. They are extremely good about semver with major versions including highly detailed patch notes.
  - Groxx 3y ago
    It definitely does not. v43 to v44: fixed "teh" in a comment v44 to v45: restructured the whole thing, steps for users to migrate are expected to be in the man-months for most projects Semver is a human convention that can also frequently help machines. It'd be nice if it were a machine system that also helped humans, but very very few compatibility systems achieve that. There are no other options currently to achieve this, and it works pretty well in practice. Use semver, don't make everyone else pay for your willful ignorance.
    - 3np 3y ago
      And now you need to update from version 42 to 67 after not paying attention for 4 months because there's an emergency bugfix or crucial feature that you can't backport trivially. Have fun.
      - Groxx 3y ago
        They do have a backport for version 56, but it's in a branch named v62-backport-v56-githubusername-2023-asdf-try-2 and you have no way to discover that. Instead of choosing v56.1, because that would be semver.
aurelianito 3y ago
Semver is based on a lie. It assumes that you can define what is the proper behavior of a library. That's not necessarily true, and can be subtlety false, and different depending on the use case. I can think on some situations where I trusted semantic versioning and it bited me when an update was made. I am not talking about new bug introductions. I assumed behavoir that the library owner assumed as not important.
- esafak 3y ago
  You should write tests to catch such changes.
- cmrdporcupine 3y ago
  In the end it's why Google insists on "vendoring" (checking in) third party dependencies and only having one version of a dependency for the entire (massive) mono repo and designated maintainers. You want a new version? Put a ring on it. Test it, and maintain it. At first I thought it was insanity. Now, buried in a maze of transitive third party and first party dependencies scattered across multiple git repositories and crates, I really really miss it. Of course like many other things there it depends on having lots of well-paid staff, without deadline guns to their head, committed to code quality and infrastructure.
- williamdclt 3y ago
  Because it’s not perfect doesn’t mean it’s useless. If semver correctly describes 90% of behaviour changes (random number), that’s a significant load off my plate as a lib user
- gpm 3y ago
  "All models are wrong, but some are useful". Yes, no matter how hard you try semver will sometimes get it wrong, but as long as it usually doesn't it's still useful to let my tools do their best at automatically selecting a version of the library that will be compatible with every part of my application that cares, and that has had as many bugs fixed as possible. That's what semver allows cargo to do, and 99.9% of the time it just works. 0.1% of the time you have to manually intervene, and that's ok. It's better than having to manually intervene 100% of the time, or the old-school C solution of "just pick whatever is lying around on the OS and hope it works/is compatible".
obi1kenobi 3y ago
Post co-author + `cargo-semver-checks` maintainer here, AMA. What we did: 1. Scan Rust's most popular 1000 crates with `cargo-semver-checks` 2. Triage & verify 3000+ semver violations 3. Build better tooling instead of blaming human error Around 1 in 31 releases had at least one semver violation. More than 1 in 6 crates violated semver in at least one release. These numbers aren't just "sum up everything `cargo-semver-checks` reported." We did a ton of validation through a combination of automated and manual means, and a big chunk of the blog post is dedicated to talking about that. Here's just one of those validation steps. For each breaking change, we constructed a "witness," a program that gets broken by it. We then verified that it: - fails to compile on the release with the semver-violating change - compiles fine on the previous version Along the way, we discovered multiple `rustc` and `cargo-semver-checks` bugs, and found out a lot of interesting edge cases about semver. Also, now you know another reason why it was so important to us to add those huge performance optimizations from a few months ago: https://predr.ag/blog/speeding-up-rust-semver-checking-by-over-2000x/ https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov...
- obi1kenobi 3y ago
  If you'd like to dig deeper, here are some links: - cargo-semver-checks: https://github.com/obi1kenobi/cargo-semver-checks https://github.com/obi1kenobi/cargo-semver-checks - Trustfall query engine, which powers cargo-semver-checks: https://github.com/obi1kenobi/trustfall https://github.com/obi1kenobi/trustfall - Trustfall playground, where you can query Rust library APIs in your browser -- for example, "which structs in `itertools` are importable by more than one path": https://play.predr.ag/rustdoc#?f=2&q=*3-Structs-importable-from-the-crate-via-multiple-paths*lquery---0Crate---2item---4*E-Struct---6name-*o*l*l--_6importable_path-*F*l--_m*T*9*p-**count***0*l--_m*f*9*p-***G*e***L-*v-*c***4path_count***j*0*l--_6--8path-*o*l--_6*J*l*l--_6span---8filename-*o*l--_8first_line*B-begin_line-*o*l--_6--*4--*2--*0*J*l*J&v=--0**path_count***B-2*l*J https://play.predr.ag/rustdoc#?f=2&q=*3-Structs-importable-f... - 10min conference talk on Trustfall: https://www.hytradboi.com/2022/how-to-query-almost-everything https://www.hytradboi.com/2022/how-to-query-almost-everythin... I'm also giving a talk at P99 CONF in a few months about how Trustfall's new optimizations API made cargo-semver-checks over 2300x faster: https://twitter.com/PredragGruevski/status/1689300249590800386 https://twitter.com/PredragGruevski/status/16893002495908003...
variaga 3y ago
Semver will never "solve" the library dependency problem, because it is not a problem that can be "solved". We want 2 things out of dependency management. 1. If a new version of a dependency contains a desirable/necessary improvement (however that is defined), we want to automatically pick up the new version. 2. If upgrading to the new version will make things worse (however that is defined), we do not want to pick up the new version. But there is no automated way of determining which of those is the case sort of actually trying the new version and fully testing it, even in the presence of "proper" semantic versioning (or any other kind of metadata you might provide). - you may already have a workaround for the library bug in your code, so a non- interface-breaking fix in the library may break your usage until you remove the workaround. Semver can't help you with this because the library maintainer doesn't know about your workaround - an update may have both properties (fixed one thing, broke something else). Semver can't make the judgement call about whether the tradeoff is a net improvement or not - the update may change a behavior that is not formally guaranteed, but which your use has an (unknown to you) dependency on. So the semver can "correctly" label it as a non-breaking change, but it still breaks your use case. - etc. This doesn't mean semver is worthless. Knowing that something is definitely a breaking change because an API was removed or modified in an incompatible way will still save you time. But you still need to actually test to know if updating a dependency is actually safe.
- obi1kenobi 3y ago
  Of course! `cargo-semver-checks` is not a replacement for testing. It's exactly the kind of tool you describe: > Knowing that something is _definitely_ a breaking change because an API was removed or modified in an incompatible way will still save you time. We want to help maintainers spot accidental breaking changes early — before they are released — instead of both maintainers and downstream projects having to scramble for fixes after an accidental breaking change is released.
- wnoise 3y ago
  > Knowing that something is definitely a breaking change because an API was removed or modified in an incompatible way will still save you time. Iff it's a part of the API you use directly or indirectly. Sometimes you're only using part of it, and changes to the rest don't matter.
HellsMaddy 3y ago
I'm curious: When updating packages, say NPM packages, what are your practices? For us, we audit every update of every package. We read the changelog or release notes for every version from the current version to the new version (some packages make this very cumbersome, looking at you Vercel). Then, we use our judgement about the order in which to update packages; usually we apply all patch-level updates, run our tests; then minor updates, re-run our tests; and then we apply major updates one-by-one and re-run tests after each of them. We've found plenty of semver violations in this process, to the point where we don't really trust the semver of NPM packages at all. Is this a common approach, or do you just blindly update everything at once and fix things if they break?
- obi1kenobi 3y ago
  Speaking for myself in Rust, I blindly update everything and fix what's broken. My projects tend to have substantial test suites, so if the compiler thinks it's fine and the tests think it's fine, it's probably fine. Sometimes it isn't fine, of course. Those cases fortunately are few enough and not damaging enough that this approach hasn't been that problematic thus far.
- contextnavidad 3y ago
  Upgrade everything you need with a commit per upgrade, then if something breaks `git bisect` to find the troublesome package. And read release notes for everything you upgrade too.
- stevepike 3y ago
  (disclaimer - I'm the founder of infield.ai; we're software for upgrading dependencies safely, even when there are breaking changes). I spent most of last year upgrading large Rails/React apps as a consultant so I have lots of thoughts on this. In general companies are in one of two states - deep in a hole of technical debt, or in a good place and trying to stay relatively up-to-date. Most established JS/Ruby codebases I've seen have dependency technical debt and they _know_ they're going to run into breaking changes trying to upgrade. These companies need to do the project management work to break their debt down into as many small incremental changes as they can. Any upgrade that has a breaking change gets done standalone (you need to read the changelog to figure out if there's a breaking change, you don't trust semver). Upgrades that are necessary but safe can be batched together. When you have to make code changes for breaking changes these should also get done incrementally ahead of time. Usually these changes are backwards compatible (e.g., if a deprecated method is going away in v2.0, you can switch to the new method in v1.X so you're not upgrading the package and changing your code in the same PR). Even if they're not trivially backwards compatible you can often make them so, and you should. I've done individual upgrades consisting of dozens of small pull requests. When you do it this way you don't get stuck with a long-running branch that gets abandoned, you roll back fewer deploys, and changes are easy to review. Once you're on the latest major version of all your packages then you want to get into an ongoing cadence of dependency maintenance. Good small-medium teams tend to do a maintenance rotation where you spend ~1 day/sprint of one developer's time on this upgrade work. Here project management is also important - ideally someone is looking at CVEs, stale packages, and other indicators of package risk, reading changelogs to identify effort, and breaking out maintenance tickets. Some of these tickets will be individual package upgrades, some will be batches of safe upgrades, and some (when a new major version of a framework comes out) will get treated like the previous paragraph. Larger teams might consolidate this work in a platform engineering team, but that has its own organizational challenges.
ekiauhce 3y ago
I skimmed through the API Compatibility section in the cargo semver docs https://doc.rust-lang.org/cargo/reference/semver.html#api-compatibility https://doc.rust-lang.org/cargo/reference/semver.html#api-co... and got excited about the list of rules they came up with for choosing right version for your release. Is there similar guidelines for other languages like Java or Python?
- kibwen 3y ago
  For Java it should be easy to create such a list, if it doesn't exist (even easier than for Rust, I assume). For Python, I think the dynamicity would make it hard to come up with anything other than a subjective set of basic, non-comprehensive guidelines (which still sounds useful, but not from the perspective of a tool like this).
  - toolslive 3y ago
    I'm still struggling with this: def do_it(*args, **kwargs) -> None: ... You might keep the signature constant, but let the behaviour evaluate across versions, but semver-wise, you could consider that API unchanged.
- obi1kenobi 3y ago
  All those rules are derived from a much smaller set of rules that are easier to understand. In Rust, those rules are more or less "breaking changes require a major version, unless the breaking change could have been avoided by the caller by adding disambiguation ahead of time." More details here: https://predr.ag/blog/some-rust-breaking-changes-do-not-require-major-version/ https://predr.ag/blog/some-rust-breaking-changes-do-not-requ... Rust generally has more exceptions than most languages about which breaking changes are technically considered non-major, so writing such guidelines for Python or Java shouldn't be too difficult. The same tech and ideas that power `cargo-semver-checks` could also be repurposed for those languages as well. If a company is interested in sponsoring such work, I'd be happy to help build something like that!
sjrd 3y ago
In the Scala ecosystem, MiMa [1] has been in widespread use for years. It automatically checks compatibility for the binary API of a library. Every library with any amount of success uses it. One could say it's the foundation of a stable ecosystem. We also have sbt-version-policy [2] to set it up with minimal configuration (and directly relate it to SemVer). More recently, we got tasty-mima [3], which checks compatibility at the type system level, rather than the binary level. [1] https://github.com/lightbend/mima https://github.com/lightbend/mima [2] https://github.com/scalacenter/sbt-version-policy https://github.com/scalacenter/sbt-version-policy [3] https://github.com/scalacenter/tasty-mima https://github.com/scalacenter/tasty-mima
BugsJustFindMe 3y ago
tldr: Semver dictates three version fields with specific definitions. At least one of those fields as defined is always useless to everyone except in a very rare circumstance. Another of those fields as defined is always risky to trust except in a very rare circumstance. Semver itself is silly. Better tooling is great but won't fix that. ... This article starts with the wrong assumption that semver is itself meaningfully correct and useful. Semver is neither of those things except in a very specific circumstance that almost nobody has. Semver violations are common because: 1. Semver as defined by semver.org is founded on the fundamentally false/flawed premise that you can predict whether a change will break compatibility. You can't. You should still try, but you should also recognize that sometimes you will guess wrong. And sometimes when you guess right you will still be wrong. The notion that just because you didn't add/remove an input or output parameter means you didn't change your API is extremely naive. But this isn't the important reason. The important reason is... 2. Even if that weren't the case, it still would only make sense for approximately 0% of all versioned software projects, because almost no projects in the wild continually preserve separate feature branches with backported bug fixes, and continually preserved feature branches with backported bug fixes is the only scenario where semver's separation of features that aren't expected to break compatibility and fixes that aren't expected to break compatibility makes any kind of sense, because there is no such thing as a clean separation between feature and fix. I hope that the truth of #1 is obvious to everyone, but if it isn't please refer to https://xkcd.com/1172/ https://xkcd.com/1172/ Less pithily, internal changes alter outward behavior literally all the time in unintentional ways, and no software project on the planet defines and constrains the expectations for its interface with sufficient clarity to avoid that. Not one. It's probably not even possible. As for #2, consider the following scenario... You have released version 1.0.0 of something. Then you add a feature and fix a bug unrelated to that feature. Are you at version 1.1.0 or 1.1.1? Well, it depends on the order you added your changes, doesn't it? If you fixed the bug first you'll go from 1.0.0 to 1.0.1 to 1.1.0, and if you add the feature first you'll go from 1.0.0 to 1.1.0 to 1.1.1. And if that difference doesn't matter, then the last digit doesn't matter. The problem here is that Semver treats feature and fix as fundamentally distinct from each other. But non-breaking is non-breaking. The user on the other side of the fence wants the best version of whatever you have that doesn't break compatibility with what they're doing. If they trust you to make that distinction, then they only care about your major versions. If they don't trust you to make that distinction, then they only care about strictly matching the whole string. If you trust yourself (lol), you can cater to both groups with two-part major.minor. If you don't trust yourself, you can just use a single value that increases over time. But semver has three fields, and one of those fields is basically always completely useless except to satisfy a contractual obligation. There is only one scenario where three version fields matter, and that's when a government defense contract forces you to fork development at the start of each project and then maintain separate code branches for separate projects, where they only get the features they ask for and you only fix the bugs they ask you to fix (this is exactly how defense contracts work), and you laboriously backport bug fixes to all of them, and the major and minor versions indicate the point of the fork, and the patch version is all the changes applied to that fork.
iraqmtpizza 3y ago
How about the tools read the flipping patch notes like they're supposed to? IT'S ALL THERE.
verdverm 3y ago
This is more a coordination problem, semver doesn't really matter here, except as a good enough standard or guardrail for most. It's going to get broken from time to time, and that is fine. The CUE Unity project is an interesting take on how to "solve" the bigger issue of breaking changes or regressions. https://github.com/cue-unity/unity https://github.com/cue-unity/unity Basically, the idea is that users can register their project, tests, or benchmarks in Unity. The project maintainers can then test new code against these projects before releasing, or even before pushing a commit, because a regression is found for example. This works by injecting the version (or local code) into the dependency management system. For CUE, this is a `go mod edit`, or setting up the environment with the cli at version / local. How this is set up and managed isn't as important as getting the process in place and making it an easy workflow for both sides. Hence why I call it a coordination problem. We have a similar project Harmony, inspired by Unity, powered by Dagger. https://github.com/hofstadter-io/hof/tree/_dev/ci/harmony https://github.com/hofstadter-io/hof/tree/_dev/ci/harmony (it's really just a dagger pipeline that has a few flags and loops over data in a file)
- zacmps 3y ago
  The rust compiler does something similar by testing on all (ish) of crates.io, the public package repository.
  - verdverm 3y ago
    yup, they reference it as an inspiration: https://github.com/rust-lang/crater https://github.com/rust-lang/crater it's probably impossible to automate an entire ecosystem, and there is value to enabling a tighter integration within a project ecosystem (a subset of the language ecosystem).

77 points by steveklabnik 3y ago

o11c 3y ago

One thing that's often not mentioned is - how to deal with experimental APIs and such? You don't want to mark them as experimental indefinitely, and you don't want to force an upgrade if the experimental API didn't change when stabilizing. The best solution I've found is: each compiled library doesn't just have a single API, but rather has multiple. libfoo-1.2.3 might implement the following APIs: * libfoo-1.0 * libfoo-1.1 * libfoo-1.2 * libfoo-1.2.1 (because people don't actually use the "no API additions in patch releases" rule) * libfoo-1.2.2 * libfoo-1.2.3 * libfoo-experimental-1_2_3 (no semver in the version here) and if you enable experimental exports for the library, you gain a package-manager-level dependency on libfoo-experimental-1_2_3. Then later you can declare "libfoo-1.3 includes libfoo-experimental-1_2_3", so old programs can use newer libraries without change. That said, mutable metadata is still an important part of the ecosystem. Blacklisting is essential (since the symbol-list isn't actually the whole API and it is impossible to automatically detect violations), but easily abused for "that's insecure" and "I don't support that" purposes so there should be some friction. Note also that ABI (and sometimes API) is fundamentally platform-specific. And there's enough complexity in the world that the tooling shouldn't think in terms of semver, but rather arbitrary compatibility DAGs (but with sparseness when possible). Consider the GCC 4.7.0/4.7.1 std::list breakage that was reverted in 4.7.2, breaking software that was compiled in the mean time (the only reason this wasn't catastrophic is because nobody actually uses std::list) - that intermediate software needs to know it can't use new binaries.

obi1kenobi 3y ago
In Rust, this is often done by using optional features, and usually features named like `unstable-<something>`. They are a form of conditional compilation: people that don't opt in don't see that code, and people that do opt in know the code is unstable. `cargo-semver-checks` by default doesn't check for any semver obligations for code behind such features. (I'm its maintainer )
- epage 3y ago
  And we'd like to have native support for these in cargo: https://github.com/rust-lang/cargo/issues/10881 https://github.com/rust-lang/cargo/issues/10881
- o11c 3y ago
  The problem with the current approach is that it does not specify whether `libfoo-1.2.3[unstable-bar]` is compatible with `libfoo-1.3.4[unstable-bar]`, nor whether the latter is compatible with `libfoo-1.4.5[default]`. (also I'm not sure under what circumstances recompilation is required)
  - conradludgate 3y ago
    The rust std lib doesn't make any commitments to unstable (nightly) APIs, so this is the norm.
    - o11c 3y ago
      It doesn't formally make commitments. But in practice for most particular APIs there is stability, and it is the job of tooling to support that.

andrewstuart 3y ago

Forget the complexity, just use integer version numbers.

Ar-Curunir 3y ago
So every change is a breaking change? Not sure that helps anybody
- SkyPuncher 3y ago
  I agree. Semver isn’t perfect, but it gives you a general idea of how hard an upgrade might be. That’s even more true with the most popular of libraries. For example, Rails and React are the two most important dependencies in our system. They are extremely good about semver with major versions including highly detailed patch notes.
- Groxx 3y ago
  It definitely does not. v43 to v44: fixed "teh" in a comment v44 to v45: restructured the whole thing, steps for users to migrate are expected to be in the man-months for most projects Semver is a human convention that can also frequently help machines. It'd be nice if it were a machine system that also helped humans, but very very few compatibility systems achieve that. There are no other options currently to achieve this, and it works pretty well in practice. Use semver, don't make everyone else pay for your willful ignorance.
  - 3np 3y ago
    And now you need to update from version 42 to 67 after not paying attention for 4 months because there's an emergency bugfix or crucial feature that you can't backport trivially. Have fun.
    - Groxx 3y ago
      They do have a backport for version 56, but it's in a branch named v62-backport-v56-githubusername-2023-asdf-try-2 and you have no way to discover that. Instead of choosing v56.1, because that would be semver.

aurelianito 3y ago

Semver is based on a lie. It assumes that you can define what is the proper behavior of a library. That's not necessarily true, and can be subtlety false, and different depending on the use case. I can think on some situations where I trusted semantic versioning and it bited me when an update was made. I am not talking about new bug introductions. I assumed behavoir that the library owner assumed as not important.

esafak 3y ago
You should write tests to catch such changes.
cmrdporcupine 3y ago
In the end it's why Google insists on "vendoring" (checking in) third party dependencies and only having one version of a dependency for the entire (massive) mono repo and designated maintainers. You want a new version? Put a ring on it. Test it, and maintain it. At first I thought it was insanity. Now, buried in a maze of transitive third party and first party dependencies scattered across multiple git repositories and crates, I really really miss it. Of course like many other things there it depends on having lots of well-paid staff, without deadline guns to their head, committed to code quality and infrastructure.
williamdclt 3y ago
Because it’s not perfect doesn’t mean it’s useless. If semver correctly describes 90% of behaviour changes (random number), that’s a significant load off my plate as a lib user
gpm 3y ago
"All models are wrong, but some are useful". Yes, no matter how hard you try semver will sometimes get it wrong, but as long as it usually doesn't it's still useful to let my tools do their best at automatically selecting a version of the library that will be compatible with every part of my application that cares, and that has had as many bugs fixed as possible. That's what semver allows cargo to do, and 99.9% of the time it just works. 0.1% of the time you have to manually intervene, and that's ok. It's better than having to manually intervene 100% of the time, or the old-school C solution of "just pick whatever is lying around on the OS and hope it works/is compatible".

obi1kenobi 3y ago

Post co-author + `cargo-semver-checks` maintainer here, AMA. What we did: 1. Scan Rust's most popular 1000 crates with `cargo-semver-checks` 2. Triage & verify 3000+ semver violations 3. Build better tooling instead of blaming human error Around 1 in 31 releases had at least one semver violation. More than 1 in 6 crates violated semver in at least one release. These numbers aren't just "sum up everything `cargo-semver-checks` reported." We did a ton of validation through a combination of automated and manual means, and a big chunk of the blog post is dedicated to talking about that. Here's just one of those validation steps. For each breaking change, we constructed a "witness," a program that gets broken by it. We then verified that it: - fails to compile on the release with the semver-violating change - compiles fine on the previous version Along the way, we discovered multiple `rustc` and `cargo-semver-checks` bugs, and found out a lot of interesting edge cases about semver. Also, now you know another reason why it was so important to us to add those huge performance optimizations from a few months ago: https://predr.ag/blog/speeding-up-rust-semver-checking-by-over-2000x/ https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov...

obi1kenobi 3y ago
If you'd like to dig deeper, here are some links: - cargo-semver-checks: https://github.com/obi1kenobi/cargo-semver-checks https://github.com/obi1kenobi/cargo-semver-checks - Trustfall query engine, which powers cargo-semver-checks: https://github.com/obi1kenobi/trustfall https://github.com/obi1kenobi/trustfall - Trustfall playground, where you can query Rust library APIs in your browser -- for example, "which structs in `itertools` are importable by more than one path": https://play.predr.ag/rustdoc#?f=2&q=*3-Structs-importable-from-the-crate-via-multiple-paths*lquery---0Crate---2item---4*E-Struct---6name-*o*l*l--_6importable_path-*F*l--_m*T*9*p-**count***0*l--_m*f*9*p-***G*e***L-*v-*c***4path_count***j*0*l--_6--8path-*o*l--_6*J*l*l--_6span---8filename-*o*l--_8first_line*B-begin_line-*o*l--_6--*4--*2--*0*J*l*J&v=--0**path_count***B-2*l*J https://play.predr.ag/rustdoc#?f=2&q=*3-Structs-importable-f... - 10min conference talk on Trustfall: https://www.hytradboi.com/2022/how-to-query-almost-everything https://www.hytradboi.com/2022/how-to-query-almost-everythin... I'm also giving a talk at P99 CONF in a few months about how Trustfall's new optimizations API made cargo-semver-checks over 2300x faster: https://twitter.com/PredragGruevski/status/1689300249590800386 https://twitter.com/PredragGruevski/status/16893002495908003...

variaga 3y ago

Semver will never "solve" the library dependency problem, because it is not a problem that can be "solved". We want 2 things out of dependency management. 1. If a new version of a dependency contains a desirable/necessary improvement (however that is defined), we want to automatically pick up the new version. 2. If upgrading to the new version will make things worse (however that is defined), we do not want to pick up the new version. But there is no automated way of determining which of those is the case sort of actually trying the new version and fully testing it, even in the presence of "proper" semantic versioning (or any other kind of metadata you might provide). - you may already have a workaround for the library bug in your code, so a non- interface-breaking fix in the library may break your usage until you remove the workaround. Semver can't help you with this because the library maintainer doesn't know about your workaround - an update may have both properties (fixed one thing, broke something else). Semver can't make the judgement call about whether the tradeoff is a net improvement or not - the update may change a behavior that is not formally guaranteed, but which your use has an (unknown to you) dependency on. So the semver can "correctly" label it as a non-breaking change, but it still breaks your use case. - etc. This doesn't mean semver is worthless. Knowing that something is definitely a breaking change because an API was removed or modified in an incompatible way will still save you time. But you still need to actually test to know if updating a dependency is actually safe.

obi1kenobi 3y ago
Of course! `cargo-semver-checks` is not a replacement for testing. It's exactly the kind of tool you describe: > Knowing that something is _definitely_ a breaking change because an API was removed or modified in an incompatible way will still save you time. We want to help maintainers spot accidental breaking changes early — before they are released — instead of both maintainers and downstream projects having to scramble for fixes after an accidental breaking change is released.
wnoise 3y ago
> Knowing that something is definitely a breaking change because an API was removed or modified in an incompatible way will still save you time. Iff it's a part of the API you use directly or indirectly. Sometimes you're only using part of it, and changes to the rest don't matter.

HellsMaddy 3y ago

I'm curious: When updating packages, say NPM packages, what are your practices? For us, we audit every update of every package. We read the changelog or release notes for every version from the current version to the new version (some packages make this very cumbersome, looking at you Vercel). Then, we use our judgement about the order in which to update packages; usually we apply all patch-level updates, run our tests; then minor updates, re-run our tests; and then we apply major updates one-by-one and re-run tests after each of them. We've found plenty of semver violations in this process, to the point where we don't really trust the semver of NPM packages at all. Is this a common approach, or do you just blindly update everything at once and fix things if they break?

obi1kenobi 3y ago
Speaking for myself in Rust, I blindly update everything and fix what's broken. My projects tend to have substantial test suites, so if the compiler thinks it's fine and the tests think it's fine, it's probably fine. Sometimes it isn't fine, of course. Those cases fortunately are few enough and not damaging enough that this approach hasn't been that problematic thus far.
contextnavidad 3y ago
Upgrade everything you need with a commit per upgrade, then if something breaks `git bisect` to find the troublesome package. And read release notes for everything you upgrade too.
stevepike 3y ago
(disclaimer - I'm the founder of infield.ai; we're software for upgrading dependencies safely, even when there are breaking changes). I spent most of last year upgrading large Rails/React apps as a consultant so I have lots of thoughts on this. In general companies are in one of two states - deep in a hole of technical debt, or in a good place and trying to stay relatively up-to-date. Most established JS/Ruby codebases I've seen have dependency technical debt and they _know_ they're going to run into breaking changes trying to upgrade. These companies need to do the project management work to break their debt down into as many small incremental changes as they can. Any upgrade that has a breaking change gets done standalone (you need to read the changelog to figure out if there's a breaking change, you don't trust semver). Upgrades that are necessary but safe can be batched together. When you have to make code changes for breaking changes these should also get done incrementally ahead of time. Usually these changes are backwards compatible (e.g., if a deprecated method is going away in v2.0, you can switch to the new method in v1.X so you're not upgrading the package and changing your code in the same PR). Even if they're not trivially backwards compatible you can often make them so, and you should. I've done individual upgrades consisting of dozens of small pull requests. When you do it this way you don't get stuck with a long-running branch that gets abandoned, you roll back fewer deploys, and changes are easy to review. Once you're on the latest major version of all your packages then you want to get into an ongoing cadence of dependency maintenance. Good small-medium teams tend to do a maintenance rotation where you spend ~1 day/sprint of one developer's time on this upgrade work. Here project management is also important - ideally someone is looking at CVEs, stale packages, and other indicators of package risk, reading changelogs to identify effort, and breaking out maintenance tickets. Some of these tickets will be individual package upgrades, some will be batches of safe upgrades, and some (when a new major version of a framework comes out) will get treated like the previous paragraph. Larger teams might consolidate this work in a platform engineering team, but that has its own organizational challenges.

ekiauhce 3y ago

I skimmed through the API Compatibility section in the cargo semver docs https://doc.rust-lang.org/cargo/reference/semver.html#api-compatibility https://doc.rust-lang.org/cargo/reference/semver.html#api-co... and got excited about the list of rules they came up with for choosing right version for your release. Is there similar guidelines for other languages like Java or Python?

kibwen 3y ago
For Java it should be easy to create such a list, if it doesn't exist (even easier than for Rust, I assume). For Python, I think the dynamicity would make it hard to come up with anything other than a subjective set of basic, non-comprehensive guidelines (which still sounds useful, but not from the perspective of a tool like this).
- toolslive 3y ago
  I'm still struggling with this: def do_it(*args, **kwargs) -> None: ... You might keep the signature constant, but let the behaviour evaluate across versions, but semver-wise, you could consider that API unchanged.
obi1kenobi 3y ago
All those rules are derived from a much smaller set of rules that are easier to understand. In Rust, those rules are more or less "breaking changes require a major version, unless the breaking change could have been avoided by the caller by adding disambiguation ahead of time." More details here: https://predr.ag/blog/some-rust-breaking-changes-do-not-require-major-version/ https://predr.ag/blog/some-rust-breaking-changes-do-not-requ... Rust generally has more exceptions than most languages about which breaking changes are technically considered non-major, so writing such guidelines for Python or Java shouldn't be too difficult. The same tech and ideas that power `cargo-semver-checks` could also be repurposed for those languages as well. If a company is interested in sponsoring such work, I'd be happy to help build something like that!

sjrd 3y ago

In the Scala ecosystem, MiMa [1] has been in widespread use for years. It automatically checks compatibility for the binary API of a library. Every library with any amount of success uses it. One could say it's the foundation of a stable ecosystem. We also have sbt-version-policy [2] to set it up with minimal configuration (and directly relate it to SemVer). More recently, we got tasty-mima [3], which checks compatibility at the type system level, rather than the binary level. [1] https://github.com/lightbend/mima https://github.com/lightbend/mima [2] https://github.com/scalacenter/sbt-version-policy https://github.com/scalacenter/sbt-version-policy [3] https://github.com/scalacenter/tasty-mima https://github.com/scalacenter/tasty-mima

BugsJustFindMe 3y ago

tldr: Semver dictates three version fields with specific definitions. At least one of those fields as defined is always useless to everyone except in a very rare circumstance. Another of those fields as defined is always risky to trust except in a very rare circumstance. Semver itself is silly. Better tooling is great but won't fix that. ... This article starts with the wrong assumption that semver is itself meaningfully correct and useful. Semver is neither of those things except in a very specific circumstance that almost nobody has. Semver violations are common because: 1. Semver as defined by semver.org is founded on the fundamentally false/flawed premise that you can predict whether a change will break compatibility. You can't. You should still try, but you should also recognize that sometimes you will guess wrong. And sometimes when you guess right you will still be wrong. The notion that just because you didn't add/remove an input or output parameter means you didn't change your API is extremely naive. But this isn't the important reason. The important reason is... 2. Even if that weren't the case, it still would only make sense for approximately 0% of all versioned software projects, because almost no projects in the wild continually preserve separate feature branches with backported bug fixes, and continually preserved feature branches with backported bug fixes is the only scenario where semver's separation of features that aren't expected to break compatibility and fixes that aren't expected to break compatibility makes any kind of sense, because there is no such thing as a clean separation between feature and fix. I hope that the truth of #1 is obvious to everyone, but if it isn't please refer to https://xkcd.com/1172/ https://xkcd.com/1172/ Less pithily, internal changes alter outward behavior literally all the time in unintentional ways, and no software project on the planet defines and constrains the expectations for its interface with sufficient clarity to avoid that. Not one. It's probably not even possible. As for #2, consider the following scenario... You have released version 1.0.0 of something. Then you add a feature and fix a bug unrelated to that feature. Are you at version 1.1.0 or 1.1.1? Well, it depends on the order you added your changes, doesn't it? If you fixed the bug first you'll go from 1.0.0 to 1.0.1 to 1.1.0, and if you add the feature first you'll go from 1.0.0 to 1.1.0 to 1.1.1. And if that difference doesn't matter, then the last digit doesn't matter. The problem here is that Semver treats feature and fix as fundamentally distinct from each other. But non-breaking is non-breaking. The user on the other side of the fence wants the best version of whatever you have that doesn't break compatibility with what they're doing. If they trust you to make that distinction, then they only care about your major versions. If they don't trust you to make that distinction, then they only care about strictly matching the whole string. If you trust yourself (lol), you can cater to both groups with two-part major.minor. If you don't trust yourself, you can just use a single value that increases over time. But semver has three fields, and one of those fields is basically always completely useless except to satisfy a contractual obligation. There is only one scenario where three version fields matter, and that's when a government defense contract forces you to fork development at the start of each project and then maintain separate code branches for separate projects, where they only get the features they ask for and you only fix the bugs they ask you to fix (this is exactly how defense contracts work), and you laboriously backport bug fixes to all of them, and the major and minor versions indicate the point of the fork, and the patch version is all the changes applied to that fork.

iraqmtpizza 3y ago

How about the tools read the flipping patch notes like they're supposed to? IT'S ALL THERE.

verdverm 3y ago

This is more a coordination problem, semver doesn't really matter here, except as a good enough standard or guardrail for most. It's going to get broken from time to time, and that is fine. The CUE Unity project is an interesting take on how to "solve" the bigger issue of breaking changes or regressions. https://github.com/cue-unity/unity https://github.com/cue-unity/unity Basically, the idea is that users can register their project, tests, or benchmarks in Unity. The project maintainers can then test new code against these projects before releasing, or even before pushing a commit, because a regression is found for example. This works by injecting the version (or local code) into the dependency management system. For CUE, this is a `go mod edit`, or setting up the environment with the cli at version / local. How this is set up and managed isn't as important as getting the process in place and making it an easy workflow for both sides. Hence why I call it a coordination problem. We have a similar project Harmony, inspired by Unity, powered by Dagger. https://github.com/hofstadter-io/hof/tree/_dev/ci/harmony https://github.com/hofstadter-io/hof/tree/_dev/ci/harmony (it's really just a dagger pipeline that has a few flags and loops over data in a file)

zacmps 3y ago
The rust compiler does something similar by testing on all (ish) of crates.io, the public package repository.
- verdverm 3y ago
  yup, they reference it as an inspiration: https://github.com/rust-lang/crater https://github.com/rust-lang/crater it's probably impossible to automate an entire ecosystem, and there is value to enabling a tighter integration within a project ecosystem (a subset of the language ecosystem).