3 ms·
Does Crater ever find unstable tests that don't pass consistently on a good baseline? If so, do you just blacklist those packages and/or feedback improvements?
by wyldfire 11y ago
Does Crater ever find unstable tests that don't pass consistently on a good baseline? If so, do you just blacklist those packages and/or feedback improvements?
NM: after reading more carefully, it's clear that at least in the case where rust causes the "regression" you feedback the improvements, so likely the same for unstable tests.
- nightpool 11y agoWell, the "tests" here aren't actually tests—they're whether the package compiles or not. If the compiler is being inconsistent, that's a pretty big bug ;)
- wyldfire 11y agoAh, I thought it was doing "cargo test" to see if the compiled code had a functional regression. But that would be challenging to investigate regressions (especially in the face of unstable tests).