4 ms·
To elaborate, we use a tool called Crater to estimate the potential impact of breaking changes. It attempts to compile all of the packages on crates.io with a n
by kibwen 11y ago
To elaborate, we use a tool called Crater to estimate the potential impact of breaking changes. It attempts to compile all of the packages on crates.io with a new version of the compiler to determine any regressions, effectively using Rust's public package repository as an extended test suite.
Here's Crater's regression report for the linked soundness fix: https://gist.github.com/nikomatsakis/2f851e2accfa7ba2830d#root-regressions-sorted-by-rank https://gist.github.com/nikomatsakis/2f851e2accfa7ba2830d#ro... . It detected four root regressions, which means that there were four packages that were relying on unsound behavior. This isn't necessarily the full extent of the regressions, however, because if we ship a compiler that breaks those packages, then any other packages that have the formerly-unsound packages as dependencies will obviously also break.
Having a concrete list of regressions also allows us to go through the ecosystem and submit PRs ourselves to bring the affected packages back to building, which is usually quite easy. Crater is a really, really great tool.
- Terribledactyl 11y ago> ...effectively using Rust's public package repository as an extended test suite. This is one of the coolest and most practical things I've read in a while, seeing what happens to stable(ish) real wild code. So many practical applications and analytics are coming to mind in many areas. Thank you. Sparks my interest in Rust again.
- cies 11y agoShould also have a look at http://stackage.org http://stackage.org (quite similar but in Haskell, yet curated -- a "try to compile all" feature also exists IIRC). Not sure how much this is used in order to test impact language changes.
- ibotty 11y agoIt regularly is.
- sjrd 11y agoScala also has something similar, called the community build: https://github.com/scala/community-builds https://github.com/scala/community-builds It does not contain everything, but (open source) authors are encourage to add their library to the mix.
- aidenn0 11y agoThis has been one of the many benefits of quicklisp (a repository for common lisp packages); there is a test-grid maintained by Xach and he reports any breakages when there is a release candidate of a new sbcl.
- d4mi3n 11y agoI believe Perl and CPAN originally coined this approach. I'm glad to see from the parent and other comments that this approach seems to have caught on in other ecosystems as well.
- clarus 11y agoCoq has that too! :) https://coq-bench.github.io/clean/Linux-x86_64-4.02.3-1.2.2/released/ https://coq-bench.github.io/clean/Linux-x86_64-4.02.3-1.2.2/...
- berdario 11y agoI think that something similar is also done by the Chicken Scheme folks: http://wiki.call-cc.org/eggref/4/salmonella#introduction http://wiki.call-cc.org/eggref/4/salmonella#introduction http://tests.call-cc.org/ http://tests.call-cc.org/
- wyldfire 11y agoDoes Crater ever find unstable tests that don't pass consistently on a good baseline? If so, do you just blacklist those packages and/or feedback improvements? NM: after reading more carefully, it's clear that at least in the case where rust causes the "regression" you feedback the improvements, so likely the same for unstable tests.
- nightpool 11y agoWell, the "tests" here aren't actually tests—they're whether the package compiles or not. If the compiler is being inconsistent, that's a pretty big bug ;)
- wyldfire 11y agoAh, I thought it was doing "cargo test" to see if the compiled code had a functional regression. But that would be challenging to investigate regressions (especially in the face of unstable tests).