3 ms·

One (misguided) counter argument I've heard from otherwise fantastic devs it's the notion of adding randomness to unit tests in the hopes that if there's a bug,

by mabbo 9y ago

One (misguided) counter argument I've heard from otherwise fantastic devs it's the notion of adding randomness to unit tests in the hopes that if there's a bug, at least some builds will fail. In practice, I've seen those builds and developers saying "yeah, sometimes you need to build it twice". I think the solution is to give those devs who favor such techniques a separate but easy to use fuzzing tool set that they can run just like their unit tests, separate from their usual 'build' command. Give them their ability to discover new bugs, but make it separate from the real build.

sitkack 9y ago
Non-determinism should be on a leaf of the DAG.
closeparen 9y ago
Why would the randomness in unit tests affect the binary? RNGs are invoked when the tests are run, not when they're built, and anyway, test code shouldn't be part of the final binary.
- davexunit 9y ago
  The test suite is used to validate the build. Intermittently failing test suite -> intermittently failing build. You can always disable running a problematic test suite, but that doesn't exactly inspire confidence in the result.
regularfry 9y ago
My take on that is that those devs are probably fooling themselves about the state of their test suite. I'm working on a codebase like that right now - it's in generally a relatively good state, but just this week I've run into a moderately nasty bug involving global state not getting properly reset which the tests didn't cover because their specific execution order happened to work.
kobeya 9y ago
Down-voted because that's not an argument against reproducible builds. The outputs can be bit-for-bit identical but still use an RNG for fuzz testing.
- davexunit 9y ago
  A "build" often includes running the unit test suite to verify that the software works in the build environment, so randomness in the test suite can indeed lead to non-reproducible output: when a bad RNG seed causes a test failure and thus a failed build. I've spent my fair share of time debugging issues like this.
  - kobeya 9y ago
    Use a PRNG seeded with the hash of the build manifest.
    - davexunit 9y ago
      I don't know what a build manifest is but yeah, that sounds great. However, the test suite needs to support this. Plenty of times they do not.
vfaronov 9y ago
Or just seed the PRNG with some digest of the relevant source code (perhaps as simple as the version number). Doesn’t solve the problem that your tests can suddenly break on unrelated changes, but does solve the problem that your tests can break without any changes.

by mabbo 9y ago

sitkack 9y ago

Non-determinism should be on a leaf of the DAG.

closeparen 9y ago

Why would the randomness in unit tests affect the binary? RNGs are invoked when the tests are run, not when they're built, and anyway, test code shouldn't be part of the final binary.

davexunit 9y ago
The test suite is used to validate the build. Intermittently failing test suite -> intermittently failing build. You can always disable running a problematic test suite, but that doesn't exactly inspire confidence in the result.

regularfry 9y ago

My take on that is that those devs are probably fooling themselves about the state of their test suite. I'm working on a codebase like that right now - it's in generally a relatively good state, but just this week I've run into a moderately nasty bug involving global state not getting properly reset which the tests didn't cover because their specific execution order happened to work.

kobeya 9y ago

Down-voted because that's not an argument against reproducible builds. The outputs can be bit-for-bit identical but still use an RNG for fuzz testing.

davexunit 9y ago
A "build" often includes running the unit test suite to verify that the software works in the build environment, so randomness in the test suite can indeed lead to non-reproducible output: when a bad RNG seed causes a test failure and thus a failed build. I've spent my fair share of time debugging issues like this.
- kobeya 9y ago
  Use a PRNG seeded with the hash of the build manifest.
  - davexunit 9y ago
    I don't know what a build manifest is but yeah, that sounds great. However, the test suite needs to support this. Plenty of times they do not.

vfaronov 9y ago

Or just seed the PRNG with some digest of the relevant source code (perhaps as simple as the version number). Doesn’t solve the problem that your tests can suddenly break on unrelated changes, but does solve the problem that your tests can break without any changes.