4 ms·
Nix is mostly deterministic due to lots of effort by the community but it’s hard to maintain full determinism in your own tests.
by fire_lake 2y ago
Nix is mostly deterministic due to lots of effort by the community but it’s hard to maintain full determinism in your own tests.
- IshKebab 2y agoBasically the only sane answer is Bazel (or similar). We currently use path based test running because I couldn't convince people it was wrong, and it regularly results in `master` being broken. I don't really have a great answer for when that stops scaling - especially for low level things e.g. when Android changes bionic do they run every test because everything depends on it? Technically they should. The only other cool technique I know of is coverage-based test ranking, but as far as I know it's only used in silicon verification. Basically if you have more tests than you can run, only run the top N based on coverage metrics.
- turboponyy 2y agoThere's always the escape hatch of marking a derivation as impure. However, most tests should really have no reason to be impure by design. And tests that do depend on side effects might as well be re-evaluated every time.
- fire_lake 2y agoThe problem is it’s hard to know which tests are impure.