3 ms·
PBT is nice in place of (some) unit tests in that you can describe immediately the properties you expect without needing to produce a collection of examples (or
by Jtsummers 6y ago
PBT is nice in place of (some) unit tests in that you can describe immediately the properties you expect without needing to produce a collection of examples (or write a custom generator that produces a limited set of examples, at which point you're halfway to PBT anyways).
It's also helpful to use it in a piece wise fashion if you're doing TDD. An illustrative example (though perhaps not stellar as it is a synthetic, non-real-world example) uses the diamond kata, TDD, and PBT together [0]. None of the tests on their own fully specify the system, but in total they do.
If you're doing TDD (or attempting to) I think this is an interesting case. Many TDD methods have you start off with an example case like (to stick with this kata, and using Pythonesque pseudocode because I'm still not awake this Saturday morning):
diamond-kata-test-a:
assert(diamond('A') == 'A')
Great, so now someone makes that absolute simplest solution:
diamond(c):
return 'A'
Now repeat with a second test case:
diamond-kata-test-b:
assert(diamond('B') == ' A \nB B\n A ')
And the function is duly complicated:
diamond(c):
switch c:
case 'A': return 'A'
case 'B': return ' A \nB B\n A '
default: return 'blah' // or error, doesn't matter it's not tested
But not actually generalized to reflect the intent of the system. By focusing on properties, I've found, the progression of the UUT is a bit better/more natural.
Another interesting thing to do with PBT is model-based testing [1]. The useful thing here is that sometimes the errors are triggered by a peculiar, though plausible, sequence of commands to your system. We've all worked with that one guy who somehow manages to find exactly the right sequence that triggers weird edge cases and errors, but unless we're him having a system which will generate arbitrary execution traces for you. (I actually used FsCheck for this last year in trying to sell PBT to my colleagues and was able to identify where a known issue originated as well as several other problems that hadn't been found by users or testers yet.)
In the end, when these failures are found you can always turn them into distinct unit tests in order to preserve them and prevent regressions. The two modes of testing fit well together.
[0] http://christophethibaut.com/programming/2020/03/18/Diamond-Kata-PBT-TDD.html http://christophethibaut.com/programming/2020/03/18/Diamond-...
[1] https://fscheck.github.io/FsCheck/StatefulTesting.html https://fscheck.github.io/FsCheck/StatefulTesting.html