4 ms·

What's the ratio among Rust crates with over 50k downloads? I manage a tool with literally seven hundred dependencies and not one of them has ever broken semver

What's the ratio among Rust crates with over 50k downloads? I manage a tool with literally seven hundred dependencies and not one of them has ever broken semver.

eviks 3y ago
How are you so certain if it could be a break in a code path you don't use? Could you check it with this tool?
- obi1kenobi 3y ago
  cargo-semver-checks is not dependent on whether code paths are used, because it doesn't analyze from the "use" side. It analyzes from the "what API is exposed" side. You can think of cargo-semver-checks as an automated way to answer a series of queries like "are there any public functions that existed in the previous version and don't exist in the new version?" Any results produced by any of those queries are examples of breaking changes. This is why our "more than 1 in 6" study found so many previously-unreported semver violations — in our study we were able to find issues in all sorts of dark corners of library APIs, then generate programs specifically designed to rely on the affected behavior in a way that works fine in one API version and causes a compilation error in the subsequent (non-major) version.
  - eviks 3y ago
    > cargo-semver-checks is not dependent on whether code paths are used that's exactly my point, without this great tool he can't know whether semver is broken from his practical use since his use is not exhaustive
obi1kenobi 3y ago
Tool and study coauthor here. The "more than 1 in 6" ratio is purely in the top 1000 most-downloaded crates on crates.io, which are the only ones we tested. I'm glad to hear of your positive experience with semver. Unfortunately, the issue with semver violations is that their impact has extremely high variance, easily ranging between "nobody noticed" and "ecosystem-wide pain": - Most of the semver violations we found appear to have never been reported or noticed. After talking to a representative subset of the maintainers of those crates, they were generally surprised to find out a semver violation had happened. - At the same time, a quick search for "violated semver" in Rust repos on GitHub will surface hundreds of issues, many of them heavily linked-to by issues in other projects. Every such issue represents dozens or hundreds of cumulative engineer-hours of triage and remediation in both the project where the semver accident happened and in affected downstream projects. All this work is regrettable, in the sense that both maintainers and downstream consumers would have preferred if it could have been avoided. Most semver violations aren't intentional -- they are accidents that happen to all of us regardless of skill level, carefulness, experience, etc. Most of the time, those accidents go unnoticed or have minimal impact. Every so often, they trigger nearly ecosystem-wide consequences for a few days or weeks. (Some examples on my blog: https://predr.ag/blog/toward-fearless-cargo-update/ https://predr.ag/blog/toward-fearless-cargo-update/ and https://predr.ag/tags/semver https://predr.ag/tags/semver ) And sometimes, breaking semver on purpose can be the correct thing to do! cargo-semver-checks aims to be a tool such that maintainers never have to say "if I had known this API change had happened, I wouldn't have published this version." Its biggest impact should be at minimizing or altogether preventing those ecosystem-wide semver issues that happen from time to time. Our day-to-day experience with semver should remain unchanged otherwise.

What's the ratio among Rust crates with over 50k downloads? I manage a tool with literally seven hundred dependencies and not one of them has ever broken semver.

eviks 3y ago

How are you so certain if it could be a break in a code path you don't use? Could you check it with this tool?

obi1kenobi 3y ago
cargo-semver-checks is not dependent on whether code paths are used, because it doesn't analyze from the "use" side. It analyzes from the "what API is exposed" side. You can think of cargo-semver-checks as an automated way to answer a series of queries like "are there any public functions that existed in the previous version and don't exist in the new version?" Any results produced by any of those queries are examples of breaking changes. This is why our "more than 1 in 6" study found so many previously-unreported semver violations — in our study we were able to find issues in all sorts of dark corners of library APIs, then generate programs specifically designed to rely on the affected behavior in a way that works fine in one API version and causes a compilation error in the subsequent (non-major) version.
- eviks 3y ago
  > cargo-semver-checks is not dependent on whether code paths are used that's exactly my point, without this great tool he can't know whether semver is broken from his practical use since his use is not exhaustive

obi1kenobi 3y ago

Tool and study coauthor here. The "more than 1 in 6" ratio is purely in the top 1000 most-downloaded crates on crates.io, which are the only ones we tested. I'm glad to hear of your positive experience with semver. Unfortunately, the issue with semver violations is that their impact has extremely high variance, easily ranging between "nobody noticed" and "ecosystem-wide pain": - Most of the semver violations we found appear to have never been reported or noticed. After talking to a representative subset of the maintainers of those crates, they were generally surprised to find out a semver violation had happened. - At the same time, a quick search for "violated semver" in Rust repos on GitHub will surface hundreds of issues, many of them heavily linked-to by issues in other projects. Every such issue represents dozens or hundreds of cumulative engineer-hours of triage and remediation in both the project where the semver accident happened and in affected downstream projects. All this work is regrettable, in the sense that both maintainers and downstream consumers would have preferred if it could have been avoided. Most semver violations aren't intentional -- they are accidents that happen to all of us regardless of skill level, carefulness, experience, etc. Most of the time, those accidents go unnoticed or have minimal impact. Every so often, they trigger nearly ecosystem-wide consequences for a few days or weeks. (Some examples on my blog: https://predr.ag/blog/toward-fearless-cargo-update/ https://predr.ag/blog/toward-fearless-cargo-update/ and https://predr.ag/tags/semver https://predr.ag/tags/semver ) And sometimes, breaking semver on purpose can be the correct thing to do! cargo-semver-checks aims to be a tool such that maintainers never have to say "if I had known this API change had happened, I wouldn't have published this version." Its biggest impact should be at minimizing or altogether preventing those ecosystem-wide semver issues that happen from time to time. Our day-to-day experience with semver should remain unchanged otherwise.