4 ms·
That's a fair concern. I can only really suggest upping the amount of test cases that are ran when merging so that you get a much more extensive run for that ti
by Akronymus 1y ago
That's a fair concern. I can only really suggest upping the amount of test cases that are ran when merging so that you get a much more extensive run for that time, and later dial it back. Along with having the seed included in the failure case, so that you can bisect to check what actually broke that test. Also, implementing a standard test alongside the property based on, for all bugs you encounter over time (basically "hard coding" one of the failure cases)
But yeah, probabilistic testing isn't perfect.
- aswerty 1y agoPotentially using the git hash as a seed would make sense, so for a given snapshot of code it is always going to be deterministic. When the git hash changes (i.e. your code) then that would result in a different set of test inputs running. Allowing reproducibility for a given change set.
- Akronymus 1y agoThat's a pretty good idea, actually. Using the git hash as a seed to seed the rng for the different runs of the PB tests. Didn't even enter my mind.
- chriswarbo 1y agoUsing a git hash still has the problem of a co-worker's changes (which alter the git commit) causing an unrelated property to fail. Hypothesis has a nice option, to pick the seed for each property by hashing that property's code. It's a nice idea, but relies on Python's highly dynamic nature; so may not be easy/possible in other languages (especially compiled ones).