5 ms·

> You shouldn't be able to run it 10 times and get 9 pass and one failure. It's either 10 passes or 10 failures. With property based testing, it actually CAN b

> You shouldn't be able to run it 10 times and get 9 pass and one failure. It's either 10 passes or 10 failures. With property based testing, it actually CAN be 9 passes and 1 failure, because that one single fail can be hitting an edge case the others just aren't. In fact, only a few failures are more likely than it being all failures

bluGill 1y ago
That is the one thing about PBT that worries me. I can write code and all tests pass, then next week the edge case I missed randomly is hit by a coworker who now has to figure out why their change broke my code (it didn't). I can tell you from experience that random failures cause loss of trust. People learn to ignore failures and just keep hitting rebuild until the tests pass. People will not investigate test failures in code that they don't understand.
- Akronymus 1y ago
  That's a fair concern. I can only really suggest upping the amount of test cases that are ran when merging so that you get a much more extensive run for that time, and later dial it back. Along with having the seed included in the failure case, so that you can bisect to check what actually broke that test. Also, implementing a standard test alongside the property based on, for all bugs you encounter over time (basically "hard coding" one of the failure cases) But yeah, probabilistic testing isn't perfect.
  - aswerty 1y ago
    Potentially using the git hash as a seed would make sense, so for a given snapshot of code it is always going to be deterministic. When the git hash changes (i.e. your code) then that would result in a different set of test inputs running. Allowing reproducibility for a given change set.
    - Akronymus 1y ago
      That's a pretty good idea, actually. Using the git hash as a seed to seed the rng for the different runs of the PB tests. Didn't even enter my mind.
    - chriswarbo 1y ago
      Using a git hash still has the problem of a co-worker's changes (which alter the git commit) causing an unrelated property to fail. Hypothesis has a nice option, to pick the seed for each property by hashing that property's code. It's a nice idea, but relies on Python's highly dynamic nature; so may not be easy/possible in other languages (especially compiled ones).
- jononor 1y ago
  So lock the seed in that case?
  - bluGill 1y ago
    If you lock your seed you are worse than unit tests - odds are you are not testing some interesting cases ever. The whole point of PBT is there are some properties that hold in all cases (well technically in the domain of the function inputs), so try random examples to see if I missed something. Generally the domain of all possible function inputs is a very large set such that exhaustive testing is impossible. The more different random seeds we try the better the odds we eventually catch some case that we didn't handle correctly.
- IanCal 1y ago
  This is fair, but also a tooling or process problem. The co-worker should add the test to a new branch / test it on main. If it fails that's a new ticket (with the great side effect of having a failing test). If that passes it's a problem in their branch. If not it's the same as having a broken main which happens anyway and you deal with that as you usually do.
- zelphirkalt 1y ago
  Usually though the failure case inputs should be stored, so that once the test fails, it will fail again, no matter how often you hit the run or rebuild button.
alkonaut 1y ago
When would that be preferable to have nondeterministic random?
- Akronymus 1y ago
  I don't think I have said anything about it being nondeterministic. You can have a prng seeded from something like the commit hash and have prngs generate the test cases. That still can fail on 1/10 tests for a particular run.
- Jtsummers 1y ago
  With PBT you generate an arbitrary number of inputs, ideally they all pass. However it's entirely possible that you have an error in your program and the property only holds for some inputs, but not all. In that case, since each execution starts with a different seed (unless you provide a specific seed, in which case the generated inputs should be exactly the same each time), you may have some executions that always pass, some that always fail, and others that are in between (have a mix of passes and failures). This is expected.

bluGill 1y ago

That is the one thing about PBT that worries me. I can write code and all tests pass, then next week the edge case I missed randomly is hit by a coworker who now has to figure out why their change broke my code (it didn't). I can tell you from experience that random failures cause loss of trust. People learn to ignore failures and just keep hitting rebuild until the tests pass. People will not investigate test failures in code that they don't understand.

Akronymus 1y ago
That's a fair concern. I can only really suggest upping the amount of test cases that are ran when merging so that you get a much more extensive run for that time, and later dial it back. Along with having the seed included in the failure case, so that you can bisect to check what actually broke that test. Also, implementing a standard test alongside the property based on, for all bugs you encounter over time (basically "hard coding" one of the failure cases) But yeah, probabilistic testing isn't perfect.
- aswerty 1y ago
  Potentially using the git hash as a seed would make sense, so for a given snapshot of code it is always going to be deterministic. When the git hash changes (i.e. your code) then that would result in a different set of test inputs running. Allowing reproducibility for a given change set.
  - Akronymus 1y ago
    That's a pretty good idea, actually. Using the git hash as a seed to seed the rng for the different runs of the PB tests. Didn't even enter my mind.
  - chriswarbo 1y ago
    Using a git hash still has the problem of a co-worker's changes (which alter the git commit) causing an unrelated property to fail. Hypothesis has a nice option, to pick the seed for each property by hashing that property's code. It's a nice idea, but relies on Python's highly dynamic nature; so may not be easy/possible in other languages (especially compiled ones).
jononor 1y ago
So lock the seed in that case?
- bluGill 1y ago
  If you lock your seed you are worse than unit tests - odds are you are not testing some interesting cases ever. The whole point of PBT is there are some properties that hold in all cases (well technically in the domain of the function inputs), so try random examples to see if I missed something. Generally the domain of all possible function inputs is a very large set such that exhaustive testing is impossible. The more different random seeds we try the better the odds we eventually catch some case that we didn't handle correctly.
IanCal 1y ago
This is fair, but also a tooling or process problem. The co-worker should add the test to a new branch / test it on main. If it fails that's a new ticket (with the great side effect of having a failing test). If that passes it's a problem in their branch. If not it's the same as having a broken main which happens anyway and you deal with that as you usually do.
zelphirkalt 1y ago
Usually though the failure case inputs should be stored, so that once the test fails, it will fail again, no matter how often you hit the run or rebuild button.

alkonaut 1y ago

When would that be preferable to have nondeterministic random?

Akronymus 1y ago
I don't think I have said anything about it being nondeterministic. You can have a prng seeded from something like the commit hash and have prngs generate the test cases. That still can fail on 1/10 tests for a particular run.
Jtsummers 1y ago
With PBT you generate an arbitrary number of inputs, ideally they all pass. However it's entirely possible that you have an error in your program and the property only holds for some inputs, but not all. In that case, since each execution starts with a different seed (unless you provide a specific seed, in which case the generated inputs should be exactly the same each time), you may have some executions that always pass, some that always fail, and others that are in between (have a mix of passes and failures). This is expected.