6 ms·

I never really understood the arguments against "non-deterministic" testing. A property based test will check more cases, and therefore detect more bugs, than t

I never really understood the arguments against "non-deterministic" testing. A property based test will check more cases, and therefore detect more bugs, than the corresponding hand-chosen unit tests (in which you probably forgot at least 2 of the important corner cases). All determinism guarantees is that you'll either reliably detect a bug or reliably not detect a bug, and hence get a nice aesthetic line of green checkmarks followed by red crosses in your test history. Or just get a nice looking line of green and not even know the bug is there. This seems like a small thing to sacrifice to detect more bugs. Of course, regular unit tests have their place too, when your specification can't really be expressed as a "forall" property.

moomin 9y ago
It's the second case that's more important: if you find a bug in production, and the test that covers that functionality is green, then there's a problem with your test suite. Truth is, there's no need for an either/or holy war about this. The two styles are easily mixable: property based testing for broad coverage, specific runs for specific coverage.
IanCal 9y ago
> All determinism guarantees is that you'll either reliably detect a bug or reliably not detect a bug, and hence get a nice aesthetic line of green checkmarks followed by red crosses in your test history. Or just get a nice looking line of green and not even know the bug is there. This seems like a small thing to sacrifice to detect more bugs. The situation you want to avoid is this: Run tests, find bug. Try and fix bug. Rerun tests, tests pass. Deploy fixes to production. Later find out that the bug is not fixed, your second run of tests simply didn't hit the right combination.
- sethammons 9y ago
  We print the rand seed and allow tests to take a set seed. This allows us to use random generated data and still have deterministic results for reproducing failed cases.
  - henrik_w 9y ago
    I've used the same technique when running randomly generated traffic when testing telecom systems. Very useful to be able to re-run the same random sequence!
  - IanCal 9y ago
    Yep, this is a really useful approach, and I highly recommend anyone here who is thinking of trying this stuff out to take a note of the seed (or ensure it's printed) as the first time you realise you want to do this it's often too late!
zmonx 9y ago
A few good points have already been made in the other responses, and I want to add an additional observation: To me, the core of the issue is not in choosing between "non-deterministic" testing vs. a tiny set of hand-chosen examples. Rather, to me, the issue is exhaustive testing vs. non-exhaustive testing. Typically, critical issues in complex systems only arise extremely rarely. Consequently, randomized testing typically does not bring them to light. There may definitely be cases where it is useful, and it can certainly easily be more exhaustive than a small hand-picked set of unit tests. But the core issue is still that I typically need exhaustive tests to be more certain about relevant properties.

moomin 9y ago

It's the second case that's more important: if you find a bug in production, and the test that covers that functionality is green, then there's a problem with your test suite. Truth is, there's no need for an either/or holy war about this. The two styles are easily mixable: property based testing for broad coverage, specific runs for specific coverage.

IanCal 9y ago

> All determinism guarantees is that you'll either reliably detect a bug or reliably not detect a bug, and hence get a nice aesthetic line of green checkmarks followed by red crosses in your test history. Or just get a nice looking line of green and not even know the bug is there. This seems like a small thing to sacrifice to detect more bugs. The situation you want to avoid is this: Run tests, find bug. Try and fix bug. Rerun tests, tests pass. Deploy fixes to production. Later find out that the bug is not fixed, your second run of tests simply didn't hit the right combination.

sethammons 9y ago
We print the rand seed and allow tests to take a set seed. This allows us to use random generated data and still have deterministic results for reproducing failed cases.
- henrik_w 9y ago
  I've used the same technique when running randomly generated traffic when testing telecom systems. Very useful to be able to re-run the same random sequence!
- IanCal 9y ago
  Yep, this is a really useful approach, and I highly recommend anyone here who is thinking of trying this stuff out to take a note of the seed (or ensure it's printed) as the first time you realise you want to do this it's often too late!

zmonx 9y ago

A few good points have already been made in the other responses, and I want to add an additional observation: To me, the core of the issue is not in choosing between "non-deterministic" testing vs. a tiny set of hand-chosen examples. Rather, to me, the issue is exhaustive testing vs. non-exhaustive testing. Typically, critical issues in complex systems only arise extremely rarely. Consequently, randomized testing typically does not bring them to light. There may definitely be cases where it is useful, and it can certainly easily be more exhaustive than a small hand-picked set of unit tests. But the core issue is still that I typically need exhaustive tests to be more certain about relevant properties.