3 ms·
referencing fuzzing so heavily in the definition places a lot of faith in understanding that there are those subtle varieties of fuzzers that can generate struc
by platz 10y ago
referencing fuzzing so heavily in the definition places a lot of faith in understanding that there are those subtle varieties of fuzzers that can generate structured data/inputs.
I didn't know fuzzers did that, so I'd guess that others would miss the nuance there too; and it's a rather large part of property based testing
This is a definition that aims to be more "categorically correct" without providing a lot of intuitive explanatory power. (E.g. A monad is a monoid in the category of endofunctors). Maybe that's ok in that was the aim and PBT can't be reduced to a single sentence like a lot of abstract concepts
It also assumes a well-defined definition of a fuzzer
I was also surprised to not see the word 'invariants' mentioned once. Maybe that's very close to "property", but property has other connotations so it still seems useful
- bbcbasic 10y agoTo me property based testing is just checking that a function obeys a certain law by throwing lots of random data at it. That means you have to define the law in terms of the input and output. Rather than fix the input like you'd do on a regular test and just assert the output. The hard part is trying to figure out what laws you should expect from your functions.
- michaelfeathers 10y agoI look at property-based testing the same way I look at Design by Contract. It can be used to find errors in existing systems, but it's also valuable when you are designing. Just asking yourself what laws a function should adhere to as you are considering writing it function puts you in a different frame of mind and you can end up with simpler software.
- nickpsecurity 10y agoYeah, fuzzing is just throwing random data. Property-based is supposed to be more like DbC where it's guided to test the righg things.
- paulddraper 10y agoMaybe you should write a function that does the right thing, and then use it to tell you the correct output :/
- hyperpape 10y agoYou may be joking, but this can be a good technique for cases where you have a simple but inefficient solution and a complex optimized solution. Ditto for anything that has a special case that can be easily answered.
- platz 10y agoAlso known as an 'oracle'
- paulddraper 10y agoHalf-joking. It can be helpful to verify an optimized implementation with a naive one. That's a small minority of testing though. Special casing is effectively traditional testing: input is 1, output is 2.
- nickpsecurity 10y agoI did that for high-assurance systems as an equivalence check to ensure optimizing compiler didn't break things. It's also standard in hardware verification where they equivalence check each step from highest-level form to lowest-level form.
- dbcurtis 10y agoNot totally silly. Just last Friday I was writing a Python hypothesis-based test using constrained-random stimulus generation. The target function under test is rather complicated because it is coded to be space and time efficient. It is not so easy to read and reason about, surprise, surprise. But there is a straight-forward, easy to ready, while-loops-and-simple-if's implementation that is slow and inefficient, but it's not too hard to convince yourself it produces the correct answer. So I use that in my test driver to generated expected results. It's certainly possible for both implementations to contain bugs, but extremely unlikely for them both to report the same wrong answer for the same stimulus.