4 ms·
Nix solves this as a byproduct (as it does with many things) of its design. You can have your tests be a "build", where the build succeeds if your tests pass. A
by turboponyy 2y ago
Nix solves this as a byproduct (as it does with many things) of its design. You can have your tests be a "build", where the build succeeds if your tests pass. Any builds can be cached, which means you're essentially caching tests you've already run. Since Nix is deterministic, you never have to rerun any tests until anything about them that could change the evaluation changes.
- fire_lake 2y agoNix is mostly deterministic due to lots of effort by the community but it’s hard to maintain full determinism in your own tests.
- IshKebab 2y agoBasically the only sane answer is Bazel (or similar). We currently use path based test running because I couldn't convince people it was wrong, and it regularly results in `master` being broken. I don't really have a great answer for when that stops scaling - especially for low level things e.g. when Android changes bionic do they run every test because everything depends on it? Technically they should. The only other cool technique I know of is coverage-based test ranking, but as far as I know it's only used in silicon verification. Basically if you have more tests than you can run, only run the top N based on coverage metrics.
- turboponyy 2y agoThere's always the escape hatch of marking a derivation as impure. However, most tests should really have no reason to be impure by design. And tests that do depend on side effects might as well be re-evaluated every time.
- fire_lake 2y agoThe problem is it’s hard to know which tests are impure.