4 ms·

You say the experiment has failed, but speak purely of theory. The actual experiment, the one running in practice, is an astounding success; the things you desc

You say the experiment has failed, but speak purely of theory. The actual experiment, the one running in practice, is an astounding success; the things you describe happening don't happen, or at least don't happen in Rust. In practice, a dependency I should not trust to have done diligence in semver management is only a couple steps past a dependency I should not trust to run on my machine. Ignore any dependency with less than 50k downloads and 'what if other people are dumb' concerns like this one just melt away.

eviks 3y ago
The tool author found out that at least 1 in 6 Rust creates broke semver that the tool detected. That's a big number for an "astounding success"
- pie_flavor 3y ago
  What's the ratio among Rust crates with over 50k downloads? I manage a tool with literally seven hundred dependencies and not one of them has ever broken semver.
  - eviks 3y ago
    How are you so certain if it could be a break in a code path you don't use? Could you check it with this tool?
    - obi1kenobi 3y ago
      cargo-semver-checks is not dependent on whether code paths are used, because it doesn't analyze from the "use" side. It analyzes from the "what API is exposed" side. You can think of cargo-semver-checks as an automated way to answer a series of queries like "are there any public functions that existed in the previous version and don't exist in the new version?" Any results produced by any of those queries are examples of breaking changes. This is why our "more than 1 in 6" study found so many previously-unreported semver violations — in our study we were able to find issues in all sorts of dark corners of library APIs, then generate programs specifically designed to rely on the affected behavior in a way that works fine in one API version and causes a compilation error in the subsequent (non-major) version.
      - eviks 3y ago
        > cargo-semver-checks is not dependent on whether code paths are used that's exactly my point, without this great tool he can't know whether semver is broken from his practical use since his use is not exhaustive
  - obi1kenobi 3y ago
    Tool and study coauthor here. The "more than 1 in 6" ratio is purely in the top 1000 most-downloaded crates on crates.io, which are the only ones we tested. I'm glad to hear of your positive experience with semver. Unfortunately, the issue with semver violations is that their impact has extremely high variance, easily ranging between "nobody noticed" and "ecosystem-wide pain": - Most of the semver violations we found appear to have never been reported or noticed. After talking to a representative subset of the maintainers of those crates, they were generally surprised to find out a semver violation had happened. - At the same time, a quick search for "violated semver" in Rust repos on GitHub will surface hundreds of issues, many of them heavily linked-to by issues in other projects. Every such issue represents dozens or hundreds of cumulative engineer-hours of triage and remediation in both the project where the semver accident happened and in affected downstream projects. All this work is regrettable, in the sense that both maintainers and downstream consumers would have preferred if it could have been avoided. Most semver violations aren't intentional -- they are accidents that happen to all of us regardless of skill level, carefulness, experience, etc. Most of the time, those accidents go unnoticed or have minimal impact. Every so often, they trigger nearly ecosystem-wide consequences for a few days or weeks. (Some examples on my blog: https://predr.ag/blog/toward-fearless-cargo-update/ https://predr.ag/blog/toward-fearless-cargo-update/ and https://predr.ag/tags/semver https://predr.ag/tags/semver ) And sometimes, breaking semver on purpose can be the correct thing to do! cargo-semver-checks aims to be a tool such that maintainers never have to say "if I had known this API change had happened, I wouldn't have published this version." Its biggest impact should be at minimizing or altogether preventing those ecosystem-wide semver issues that happen from time to time. Our day-to-day experience with semver should remain unchanged otherwise.
- obi1kenobi 3y ago
  Both can be true at the same time, and I believe that to be the case. The issue in Rust isn't that semver isn't doing the right thing for us. It's that every so often (maybe as rarely as 2-3 times per year for the entire Rust community!) there's a spectacularly painful and regrettable accidental semver break that causes tons of effort to be spent by both maintainers and many downstream users. For every such incident, there are probably thousands of little semver violations here and there that we don't notice and where nobody is affected by them. cargo-semver-checks is about letting maintainers have actionable information at the right time so they can engage in well-informed decision-making. Without this tool, the alternative is: every time you publish a new version, roll a d100; on a roll of 2 or 3, you broke semver but nobody noticed; on a roll of 1, you wake up to a GitHub issue with 1000 other issues linked to it that says "you broke semver, please fix ASAP." This is the part that sucks! (The numbers are approximately accurate based on our study of the top 1000 most-downloaded crates.) The remaining 97 times, everything is great -- you publish a new version, everyone happily uses it, astounding success all around.

eviks 3y ago

The tool author found out that at least 1 in 6 Rust creates broke semver that the tool detected. That's a big number for an "astounding success"

pie_flavor 3y ago
What's the ratio among Rust crates with over 50k downloads? I manage a tool with literally seven hundred dependencies and not one of them has ever broken semver.
- eviks 3y ago
  How are you so certain if it could be a break in a code path you don't use? Could you check it with this tool?
  - obi1kenobi 3y ago
    cargo-semver-checks is not dependent on whether code paths are used, because it doesn't analyze from the "use" side. It analyzes from the "what API is exposed" side. You can think of cargo-semver-checks as an automated way to answer a series of queries like "are there any public functions that existed in the previous version and don't exist in the new version?" Any results produced by any of those queries are examples of breaking changes. This is why our "more than 1 in 6" study found so many previously-unreported semver violations — in our study we were able to find issues in all sorts of dark corners of library APIs, then generate programs specifically designed to rely on the affected behavior in a way that works fine in one API version and causes a compilation error in the subsequent (non-major) version.
    - eviks 3y ago
      > cargo-semver-checks is not dependent on whether code paths are used that's exactly my point, without this great tool he can't know whether semver is broken from his practical use since his use is not exhaustive
- obi1kenobi 3y ago
  Tool and study coauthor here. The "more than 1 in 6" ratio is purely in the top 1000 most-downloaded crates on crates.io, which are the only ones we tested. I'm glad to hear of your positive experience with semver. Unfortunately, the issue with semver violations is that their impact has extremely high variance, easily ranging between "nobody noticed" and "ecosystem-wide pain": - Most of the semver violations we found appear to have never been reported or noticed. After talking to a representative subset of the maintainers of those crates, they were generally surprised to find out a semver violation had happened. - At the same time, a quick search for "violated semver" in Rust repos on GitHub will surface hundreds of issues, many of them heavily linked-to by issues in other projects. Every such issue represents dozens or hundreds of cumulative engineer-hours of triage and remediation in both the project where the semver accident happened and in affected downstream projects. All this work is regrettable, in the sense that both maintainers and downstream consumers would have preferred if it could have been avoided. Most semver violations aren't intentional -- they are accidents that happen to all of us regardless of skill level, carefulness, experience, etc. Most of the time, those accidents go unnoticed or have minimal impact. Every so often, they trigger nearly ecosystem-wide consequences for a few days or weeks. (Some examples on my blog: https://predr.ag/blog/toward-fearless-cargo-update/ https://predr.ag/blog/toward-fearless-cargo-update/ and https://predr.ag/tags/semver https://predr.ag/tags/semver ) And sometimes, breaking semver on purpose can be the correct thing to do! cargo-semver-checks aims to be a tool such that maintainers never have to say "if I had known this API change had happened, I wouldn't have published this version." Its biggest impact should be at minimizing or altogether preventing those ecosystem-wide semver issues that happen from time to time. Our day-to-day experience with semver should remain unchanged otherwise.
obi1kenobi 3y ago
Both can be true at the same time, and I believe that to be the case. The issue in Rust isn't that semver isn't doing the right thing for us. It's that every so often (maybe as rarely as 2-3 times per year for the entire Rust community!) there's a spectacularly painful and regrettable accidental semver break that causes tons of effort to be spent by both maintainers and many downstream users. For every such incident, there are probably thousands of little semver violations here and there that we don't notice and where nobody is affected by them. cargo-semver-checks is about letting maintainers have actionable information at the right time so they can engage in well-informed decision-making. Without this tool, the alternative is: every time you publish a new version, roll a d100; on a roll of 2 or 3, you broke semver but nobody noticed; on a roll of 1, you wake up to a GitHub issue with 1000 other issues linked to it that says "you broke semver, please fix ASAP." This is the part that sucks! (The numbers are approximately accurate based on our study of the top 1000 most-downloaded crates.) The remaining 97 times, everything is great -- you publish a new version, everyone happily uses it, astounding success all around.