4 ms·
"property testing" is closer to fuzzing: it runs a parametric test with randomly generated inputs (following an input specification). Usefully, good property te
by masklinn 6y ago
"property testing" is closer to fuzzing: it runs a parametric test with randomly generated inputs (following an input specification). Usefully, good property testing systems will try to reduce the input when they find a failure (simplify possibly extremely complex input to try and minimise them to the smallest set of operations triggering them).
Mutation testing alters the system under test to identify under-specified (under-tested) portions, rather than explore the input space.
- feanaro 6y agoThe system under test could be considered as an input to the specification / test suite. Mutation testing could then be viewed as fuzzing of the test suite, with every pass result after a mutation being considered a failure of the test suite.
- pfdietz 6y agoAnd with the property-based approach, test inputs can be generated that automatically increase coverage or that kill mutants. The bane of mutation testing is equivalent mutants: mutations that aren't actually bugs, because they leave the program correct.
- nullc 6y ago> The bane of mutation testing is equivalent mutants: mutations that aren't actually bugs, because they leave the program correct. That's one reason I check the hash of the optimized binaries-- often the compiler manages to convert different but equivalent code into the same binary. It's still not enough, but it's an easy filter to apply with no false exclusions.
- pfdietz 6y agoGood idea.
- dllthomas 6y agoArguably fuzzing is property testing with a constrained set of properties, although in practice the communities differ.
- dnautics 6y agoI don't think fuzzing usually has a concept of automatic shrinking.
- dllthomas 6y agoI think shrinking is very important, but I guess I don't see it as part of the definition. Both sides have some particulars of their approaches that aren't well reflected on the other side, but I still say it's reasonable to think of them as doing fundamentally the same thing.