2 ms·
This is almost right except for two points: For fakes, spin up the real thing. If you’re not able to model your database transactions deterministically, then y
by vlovich123 5y ago
This is almost right except for two points:
For fakes, spin up the real thing. If you’re not able to model your database transactions deterministically, then your transactions could themselves be flawed and tests are great way to catch that.
Deterministic tests are not a goal in and of themselves. Controlled non determinism is valuable. This is popularized in various frameworks under the names of property checking and fuzzing which will let you know the seed to use for the failure for example so that while the runs don’t have the exact same input/output for every invocation, you get better coverage of your test space and can revisit the problematic points at any time. If you’re doing numeric simulation, make sure you are using a PRNG that’s seedable and that you log the seed at the start if you’re using a seed (and make sure time is an input parameter). Why is this technique valuable? You transitively get increasing code coverage for free through CI/coworkers running the tests AND you have a way to investigate issues sanely.
- blt 5y ago"Non-deterministic tests" usually refers to a test whose output depends on the execution environment in a way that cannot be controlled. For example, a multithreaded test with a race condition, or a test that uses the real-time clock. This is fundamentally different from a test that uses random numbers. Another way to look at it: Deterministic tests can be used in `git blame`. A unit test that uses a specific PRNG algorithm and sets the seed is fine. A test with a race condition is not. Fuzzing is very useful, but it's not unit testing. If you want to run a fuzzer or some other kind of endless randomized testing in CI, it should be a separate job from the unit tests (IMO).