10 ms·
Cargo-semver-checks: Scan your Rust crate for semver violations
- mgaunard 3y agoA build system based on people never breaking semantic versioning is a bad idea. Not only will people break it, it's inconvenient when doing active development, tweaking things here and there. Just autodetect whatever you need to in the first place.
- jopsen 3y agoWhether something is breaking or not can be subjective. You can change the behavior of an API in breaking ways without changing the interface. There are so many edge cases.
- tialaramex 3y agoSure, Hyrum's law applies. For example one of the game or 3D libraries blew up people's stuff a while back because Rust's structure layout behaviour changed. The library cared about type layout, but it neither asked its users to ensure layout (which you can do with a Rust attribute) nor checked the layout (which you could also do with a macro if you put some work in) - it just blindly assumed it knew how things are laid out, even though Rust specifically tells you that's not the default behaviour - and so it broke. This tool is a linter, with I believe no false positives. So if it says you've broken semver constraints, you can decide "Oh, whoops, I knew there's a reason I didn't change that, I'll put it back" or "Time for a new version" or, I guess, YOLO although in that case I wonder why you bother running the tool.
- mgaunard 3y agoI'm not a Rust developer, but my understanding is that if you break semver, which is very easy to do by mistake, you're in the same situation as in C++ with ODR issues, which are undefined behaviour. For a language that touts itself as a safer C++, that's definitely bad.
- lifthrasiir 3y agoSemantic versioning of Rust crates is just a convention and doesn't affect the correctness. Compiler (rustc) in particular has no notion of crates at all and will error whenever users try to use whatever API incorrectly.
- mgaunard 3y agoIt affects what gets rebuilt.
- lifthrasiir 3y agoThat's not even true for C/C++. A patch version can still update header files and C/C++ build systems can't detect whether the update is semantic or not. In Rust the serialized crate metadata can be used to detect if the recompilation is needed, and it is completely orthogonal to the semantic versioning.
- steveklabnik 3y ago(Rustc does know what crates are. You pass a crate root to rustc. You pass —extern to point at other crates. Rustc has no notion of packages.)
- tialaramex 3y agoC++ ODR is not just "Undefined Behaviour" it's an example of IFNDR, a get-out in which the language standard just washes its hands of difficult problems by saying in that case what you've written is silently not a C++ program, so its behaviour is arbitrary but you don't get any compiler diagnostics. Rust doesn't have this get-out. To some extent that's because it's not a formal standard, but mostly it's because IFNDR is diametrically opposed to correctness. Many C++ programs (perhaps even the vast majority) technically have no meaning whatsoever due to IFNDR which is bad -- it's not that the C++ compiler judges them to be correct when they aren't, but that it has no opinion either way, ever and programmers mistake "it compiled" for success. Thus, in Rust you don't get "ODR errors" here, your program won't compile. Here's a nice easy Rust semver break example: I make a library about US vacation booking, it has a public enumerated type FederalHoliday but initially that enum doesn't mention Juneteenth - whoops. Somebody points that out to me, and I correct the omission, but I forgot it's in a public enum. So that's a semver break. Why? If somebody writes code using FederalHoliday from my library, they're entitled to exhaustively match against the enumerated type. So they can make code to handle MLK Day and Thanksgiving and so on, and of course they had no reason to handle Juneteenth because it wasn't in the list - then one day they take my update. Their program no longer builds, it complains that their FederalHoliday code isn't handling Juneteenth, I made a semver breaking change. Now, in this type of case the correct fix (which you need to do before release, as introducing it later is a semver break) is to mark the enumeration as #[non_exhaustive] which tells Rust I know what's in this enumeration, so I can use it as an exhaustive match in my own code, but outside this crate, you can't match it exhaustively, you need to write a default case to handle new additions.
- nrabulinski 3y agoExcept it’s not really subjective. Does it make the code which previously compiled not compile? Yes - Breaking No - then, does it make the code which previously worked as expected do something unexpected? Yes - Breaking No - Not breaking I agree that the second part is tricky as you need to cover all usecases that used to be considered valid, but it’s not impossible, especially with a well defined API
- mgaunard 3y agoAPI breakage and ABI breakage are two different things.
- nrabulinski 3y agoRust has no stable ABI (simplification for the sake of discussing crates) and instead sources of all dependencies are fetched and compiled along with your program hence we’re talking about crate’s API and not a library’s ABI And yes, I’d say that if a program did not error at runtime but after upgrading your library it started to - that’s still a breaking change (even worse than one that you can detect at build time)
- mgaunard 3y agoThe build system needs to know what source files it needs to rebuild and for which ones it can just reuse previously-generated build artifacts, typically by detecting what changed in the source. What changed in the source outside of your project is driven by whether the semver changed. Semver not changing means that the source didn't change and you can link existing binaries as-is. Then there is the whole business of dynamic linking.
- tialaramex 3y agoHow is this different from equivalent claims about Continuous Integration or about Version Control? Automation allows humans to more easily produce good quality software.
- mgaunard 3y agoWhat I'm saying is that you could just integrate some tooling which would detect when changes are or are not made in a compatible way, instead of relying on the human making a correct annotation.
- obi1kenobi 3y agocargo-semver-checks is the tooling you describe. It doesn't currently catch every possible semver violation, so it won't suggest version numbers for you yet, but that's the eventual goal!
- mgaunard 3y agoIt isn't. You apparently didn't even read the README.
- cratermoon 3y agoAre you telling the author of the tool that he didn't even read his own README?
- obi1kenobi 3y agoxD
- mgaunard 3y agoDoesn't change the fact that the README is sufficient to see the tool doesn't do what I stated. It's a workflow problem, things are done the wrong way around.
- Qwuke 3y agoI've not had much pain with semver violations in the Rust ecosystem (compared to say people abusing the yank functionality of crates.io), but semver is easy enough to get wrong where I might have just been lucky. This is such a cool project that I hope it goes further, since verifying dependency distribution for correctness on top of all of Rust's other guarantees makes the ecosystem that much easier to justify investing in.
- obi1kenobi 3y agocargo-semver-checks author here, thanks for the kind words :) I've written several blog posts about how semver in Rust is particularly tricky and has unexpected edge cases, if you'd like to see specific examples of things non-obviously going wrong: https://predr.ag/tags/semver/ https://predr.ag/tags/semver/ Also, I've been working with the cargo team and the plan is to merge cargo-semver-checks into cargo itself, so it runs automatically on `cargo publish`. This would be similar to how cargo checks for uncommitted git changes: it will alert if it finds anything but you can override it if you think that's warranted.
- kibwen 3y ago> I've not had much pain with semver violations in the Rust ecosystem The Rust ecosystem is pretty good about crate authors not knowingly violating semver. This tool is mostly to keep crate authors from unknowingly violating semver. It's possible that the reason that you haven't noticed is because even if a crate accidentally violates semver by making an unanticipated breaking change to a given API, either you're not using that particular API (who uses 100% of any given library?) or else that the breakage doesn't affect your specific usage (e.g. maybe some type no longer implements a trait, but you weren't using that trait to begin with).
- obi1kenobi 3y agoExactly right. Most accidental semver violations aren't noticed by anyone, let alone most people. But every so often, an accidental semver violation causes ecosystem-wide breakage and lots of pain for both maintainers and users. The goal of this tool is to aid in preventing regrettable accidents like that, by informing maintainers about semver issues before they are published.
- danpalmer 3y agoElm does this. While developing Elm libraries it caught violations for me a few times even though I was trying hard to stick to semver. I also found that the average version of an Elm package was much higher than I expected given the ecosystem’s relative infancy when I was working with it (~5 years ago), when comparing to the JavaScript ecosystem where popular packages theoretically follow semver. I suspect that most packages regularly violate semver in small ways, and I’d recommend giving automated checks for violations a go if only to have the experience and be more aware of the issue.
- obi1kenobi 3y agoTool author here. Your hunch is absolutely right. An analysis I and a few contributors did for Rust showed that over 1 in 6 of the top 1000 most downloaded Rust crates has broken semver at least once in a way our tool can detect & where we can produce a program that suffers a compile error due to the semver break. This is obviously a lower bound, and yet it's quite a high number already. I plan to write more about this on my blog in the coming weeks: https://predr.ag/blog https://predr.ag/blog In the meantime I put some more info (including a link to slides from a recent talk I gave on this) in another comment: https://news.ycombinator.com/item?id=36653954 https://news.ycombinator.com/item?id=36653954
- danpalmer 3y agoNice one, great analysis.
- obi1kenobi 3y agoThank you!
- tiziano88 3y agoBy definition, if a library is updated (even at the same semver version), something in its behaviour must have changed. By Hyrum's law, that's a breaking change at least for someone. It makes no sense to try and enforce versioning by only looking at the types of an API. In fact that's the least useful way, since the compiler will already trivially catch such mismatches.
- skitter 3y agoThere might just be new functionality added with no change to existing behavior. And sure, someone might rely on undocumented behavior[1], but that's a bug in their program – nobody can reasonably complain that you've improved performance. Also, the compiler won't necessarily catch this – you might, say, change a Foo parameter to impl Into<Foo> and have it work fine for all your tests, but it might break type inference for some of your users. [1] which may hint at deficiencies in your library
- nayuki 3y ago> There might just be new functionality added with no change to existing behavior. Even this can be a breaking change for you. Suppose you're writing a bootloader that must fit in 512 bytes and you import a library. If the size of the library plus your code exceeds 512 B, then your program doesn't fit the target and will not compile. Hyrum's law is real.
- iudqnolq 3y agoIf your program has to fit in 512 bytes you're probably not pulling in a heap of external libraries?
- nayuki 3y agoIt doesn't matter. Pick a number. Maybe you have a 650 MiB cap because you're releasing software on CD. Maybe you have a 25 GB cap because you're releasing on BD. Or 100 MB for some app store. Yet, semver only considers API compatibility, not other factors such as binary size. Any minor change could push the size beyond an acceptable limit. The point still stands.
- p-e-w 3y agoI'm waiting for the day when we will finally admit to ourselves that the Semantic Versioning experiment has failed. The purpose of Semver is to allow software and people to determine, from the version number alone, whether and what has changed. But even though many projects claim adherence to the Semver standard nowadays, the ecosystem as a whole isn't, and never has been, anywhere close to the point where a reasonable engineer would actually rely on version numbers to decide whether to upgrade. Imagine you see that one of your library dependencies, which professes adherence to Semver, has released version 1.2.3, up from version 1.2.2 which your project currently uses. Would you feel comfortable doing the update, without checking whether the project still compiles and passes its test suite afterwards? If you answered "no", then you are basically admitting that Semver is worthless in practice. If you are going to test everything anyway, version numbers might as well be a simple incrementing integer, or a date, or a random string. Upgrading will still consist of trying the new version, and, on failure, rolling back – which is exactly what you are probably already doing anyway. Semver doesn't work because you can't actually trust that all your dependencies have verified their claimed Semver compliance to the extent that would be necessary for your project to rely on it. You have to do that verification yourself, which makes the Semver version number at best an occasionally useful hint. And that's not even talking about all the other problems, such as different projects having different interpretations of what constitutes a "breaking" change, and almost none of the older, established libraries even claiming to follow Semver in the first place.
- IanCal 3y ago> Would you feel comfortable doing the update, without checking whether the project still compiles and passes its test suite afterwards? I don't trust saving a file that hasn't changed without checking if things compile/pass tests. > If you answered "no", then you are basically admitting that Semver is worthless in practice I hugely disagree. Version numbers give me a good idea about what level of changes to expect. Upgrading from 1 to 2 implies a larger review is needed, I'd not just check compile/tests. Upgrading from 1.4.656 to 1.4.657 and I don't expect to have to check the docs for major changes to how things work. Dates, pure increments etc don't have this. Two releases a day apart can be wildly different - a patch release fixing a logging format and a fundamental API change. It's a method of communication . Not being perfect doesn't make it useless.
- KingOfCoders 3y agoIs there something like this for Go?
- obi1kenobi 3y agoTool author here. I think the same overall approach I described here should work for Go as well: https://news.ycombinator.com/item?id=36653954 https://news.ycombinator.com/item?id=36653954 One would just need a machine readable data source for Go package APIs, and would plug it into the Trustfall query engine. I'm not a Go expert but most languages I've worked with have something like that. If you might be interested in building this for Go, ping me and I'd be happy to help you hit the ground running!
- KingOfCoders 3y ago<3
- xh-dude 3y agohttps://pkg.go.dev/golang.org/x/exp/apidiff https://pkg.go.dev/golang.org/x/exp/apidiff might be of interest
- KingOfCoders 3y agoThanks will take a look!
- obi1kenobi 3y agoAuthor here! Excited to see this on the front page, ping me with any questions! I recently gave a talk about the tool, addressing: - why semver in Rust is important and harder than it looks - how the tool works - why it took until 2022-2023 to solve this problem There's no recording but here are the slides: https://docs.google.com/presentation/d/1gVYM9-YLrBmRYSuHqW8roE5Rt2FLAS0cijjgFU7ztxY/edit?usp=drivesdk https://docs.google.com/presentation/d/1gVYM9-YLrBmRYSuHqW8r... A lot of semver is just queries over APIs. For example: "find public functions that existed in the last version but don't exist in the new one." Any such function we find is a breaking change. Cargo-semver-checks uses an engine designed to query any data sources called Trustfall, and uses rustdoc JSON (which has an unstable format that changes ~once a month). Here's a query playground where you can try running queries over rustdoc: https://play.predr.ag/rustdoc https://play.predr.ag/rustdoc More info about all this on my blog: https://predr.ag/blog https://predr.ag/blog
- lifthrasiir 3y agoYou did much more wonderful job than just enforcing the semantic versioning---you gave a general database query tool that may drastically change how rustdoc can be used! (Sorry for my general displeasure with semver below, that's nothing to do with this kind of tools which does the actual job semver is supposed to do.)
- obi1kenobi 3y agoNo worries! Trustfall is even more general than semver actually, one can use it to query any data source -- even HN itself: https://play.predr.ag/hackernews https://play.predr.ag/hackernews Here's a link to that project if you'd like to check it out: https://github.com/obi1kenobi/trustfall https://github.com/obi1kenobi/trustfall I gave a 10min conference talk on it last year titled "How to query (almost) everything" https://www.hytradboi.com/2022/how-to-query-almost-everything https://www.hytradboi.com/2022/how-to-query-almost-everythin...
- bloopernova 3y agoI'm not familiar with rust+semver problems. Can someone please explain what the issue is?
- obi1kenobi 3y agoTool author here! I wrote a blog post that explains the motivation for building cargo-semver-checks here: https://predr.ag/blog/toward-fearless-cargo-update/ https://predr.ag/blog/toward-fearless-cargo-update/ TL;DR: Semver is particularly tricky in Rust and can be broken in many far-from-obvious ways. When a library breaks semver it can cause compile errors in every downstream bit of code that uses it. This sucks for everyone on all sides.
- bloopernova 3y agooh interesting, thank you! So semver is something that Rust always uses and enforces as opposed to stuff I've seen where it's a manual number added by a human. Cool, I can definitely see how that would cause issues, and your examples+blog were helpful. Much appreciated! Could there be a workflow in Rust compilation or packaging that uses your tool and says "oops, looks like the semver needs to be updated, can I do that for you?"
- obi1kenobi 3y agocargo-semver-checks is in the process of being merged into cargo itself, which would make the default publishing process into essentially exactly what you describe! Running `cargo publish` would check for semver issues and alert if it finds anything. Users would be able to override it, of course, but most often they'd probably accept the version change or revert the semver break if it wasn't intended.
- mjw1007 3y agoUnlike the way some languages' package systems work, Rust crates don't cleanly separate the public interface from the private implementation. In practice a crate's public interface is effectively scattered throughout a number of modules. There's no separate textual interface definition that you could diff to see what you changed without also seeing changes to the implementation. Clients importing your library compile against the whole source. So in practice if you want to check whether you changed your public interface unintentionally, your best bet is to compare the old and new output of Rust's automated documentation system, which by default is describing only the public interface. Cargo-semver-checks attempts to automate that.
- waffletower 3y agoWhat a great linting concept! Wow, just imagine a world where semver violation scanning was used by PyPi in all of its hosted pip packages.
- obi1kenobi 3y agoAuthor here. The same overall approach should work for Python and most other languages as well! If you or anyone else is interested in building such a tool, ping me and I'd be happy to help you get started.
- techdragon 3y agoI've looked into this! The issue with python is dynamic typing... It could be a tractable problem for fully type hinted code, but when your object type can be meta-programmed from external data, something I've seen SOAP client libraries do (point client class object at WSDL -> call virtual method name on client class -> get arbitrary tree of data objects which have no code representation with specific class types as appropriate)... it can be nigh impossible to fully resolve this. I'd love to have tools to scan what I could, but its remarkably challenging due to the level of meta-programming that Python provides. I admit I'm clearly not as well versed on the topic as you are at this based on looking at your work, so its entirely possible I've missed some obvious tricks that make it easier, but I'm not sure I've got the time any more to tackle a fun new project.
- zdimension 3y agoAbout a year ago, I worked on a similar tool, using the same rustdoc-based idea, [cargo-breaking](https://github.com/iomentum/cargo-breaking/tree/zdimension/rustdoc-backend https://github.com/iomentum/cargo-breaking/tree/zdimension/r...), but since cargo-semver-checks was released around the same time and used a better general approach to change detection, we never bothered actually releasing it and ended up just archiving the repo
- obi1kenobi 3y agocargo-breaking is cool! I only learned of it well after I had published cargo-semver-checks, but your logo is awesome and the tool was definitely a good idea, obviously. If you ever feel like doing more work in the space, I'd love to work together on cargo-semver-checks!