5 ms·
To me property based testing is just checking that a function obeys a certain law by throwing lots of random data at it. That means you have to define the law
by bbcbasic 10y ago
To me property based testing is just checking that a function obeys a certain law by throwing lots of random data at it.
That means you have to define the law in terms of the input and output. Rather than fix the input like you'd do on a regular test and just assert the output.
The hard part is trying to figure out what laws you should expect from your functions.
- michaelfeathers 10y agoI look at property-based testing the same way I look at Design by Contract. It can be used to find errors in existing systems, but it's also valuable when you are designing. Just asking yourself what laws a function should adhere to as you are considering writing it function puts you in a different frame of mind and you can end up with simpler software.
- nickpsecurity 10y agoYeah, fuzzing is just throwing random data. Property-based is supposed to be more like DbC where it's guided to test the righg things.
- paulddraper 10y agoMaybe you should write a function that does the right thing, and then use it to tell you the correct output :/
- hyperpape 10y agoYou may be joking, but this can be a good technique for cases where you have a simple but inefficient solution and a complex optimized solution. Ditto for anything that has a special case that can be easily answered.
- platz 10y agoAlso known as an 'oracle'
- paulddraper 10y agoHalf-joking. It can be helpful to verify an optimized implementation with a naive one. That's a small minority of testing though. Special casing is effectively traditional testing: input is 1, output is 2.
- nickpsecurity 10y agoI did that for high-assurance systems as an equivalence check to ensure optimizing compiler didn't break things. It's also standard in hardware verification where they equivalence check each step from highest-level form to lowest-level form.
- dbcurtis 10y agoNot totally silly. Just last Friday I was writing a Python hypothesis-based test using constrained-random stimulus generation. The target function under test is rather complicated because it is coded to be space and time efficient. It is not so easy to read and reason about, surprise, surprise. But there is a straight-forward, easy to ready, while-loops-and-simple-if's implementation that is slow and inefficient, but it's not too hard to convince yourself it produces the correct answer. So I use that in my test driver to generated expected results. It's certainly possible for both implementations to contain bugs, but extremely unlikely for them both to report the same wrong answer for the same stimulus.
- llimllib 10y agoHere's an example I wrote where I test the fast API against the (probably correct) brute-force version: https://github.com/llimllib/ckmeans/blob/master/ckmeans.py#L147 https://github.com/llimllib/ckmeans/blob/master/ckmeans.py#L...
- di4na 10y agoJust nitpicking about "extremely unlikely for them both to report the same wrong answer for the same stimulus". This assumption is what are "blind implementation voting systems" were based on, and it was proved to not survive that well in practice. It may totally work in your case, but you may be interested in looking at the work on Nancy Leveson at the MIT (all links are from her website) http://sunnyday.mit.edu/papers/nver-tse.pdf http://sunnyday.mit.edu/papers/nver-tse.pdf http://sunnyday.mit.edu/critics.pdf http://sunnyday.mit.edu/critics.pdf http://sunnyday.mit.edu/papers/nver2.pdf http://sunnyday.mit.edu/papers/nver2.pdf http://sunnyday.mit.edu/papers/consistent-comp.pdf http://sunnyday.mit.edu/papers/consistent-comp.pdf This body of work launched quite a controversy, but well, it is still here...