53 ms·
I've deployed mutation testing extensively in libsecp256k1 for the past five years or so, to good ends. Though it's turned up some testing inadequacies here an
by nullc 6y ago
I've deployed mutation testing extensively in libsecp256k1 for the past five years or so, to good ends.
Though it's turned up some testing inadequacies here and there and a substantial performance improvement ( https://twitter.com/pwuille/status/1348835954396516353 https://twitter.com/pwuille/status/1348835954396516353 ), I don't believe it's yet caught a bug there, but I do feel a lot more confident in the tests as a result.
I've also deployed it to a lesser degree in the Bitcoin codebase and turned up some minor bugs as a result of the tests being improved to pass mutation testing.
The biggest challenge I've seen for most parties to use mutation testing is that to begin with you must have 100% branch coverage of the code you might mutate, and very few ordinary pieces of software reach that level of coverage.
The next issue is that in C/C++ there really aren't any great tools that I'm aware of-- so every effort needs to be homebrewed.
My process is to have some a harness script that:
1. makes a modification (e.g. a python script that does small search and replacements one at a time line by line, or just doing it by hand).
2. attempts to compile the code (if it fails, move on to the next change)
3. Compares the hash of the optimized binary to a collection of already tested hashes and moves onto the next if it's already been seen.
4. Runs the tests and if the tests pass save off the diff.
5. Goto 1.
Then I go back trough the diffs and toss ones that are obviously no meaningful effect, and lob the remaining diffs over to other contributors to figure out if they're false positives or to improve the tests.
- Agentlien 6y agoThis automated approach reminds me of fuzzing.
- stevekemp 6y agoFuzz-testing can be amazingly useful at finding bugs, crashes, and security issues. I'm looking forward to seeing more people use it, once it has been added to the golang testing tools: https://go.googlesource.com/proposal/+/master/design/draft-fuzzing.md https://go.googlesource.com/proposal/+/master/design/draft-f... In several of my projects I thought I had good testing, but still found issues within 20 minutes of fuzzing. Wonderful to be able to find them so easily.
- isbvhodnvemrwvn 6y agoThat's pretty much it, but it fuzzes the code rather than data.
- masklinn 6y ago"property testing" is closer to fuzzing: it runs a parametric test with randomly generated inputs (following an input specification). Usefully, good property testing systems will try to reduce the input when they find a failure (simplify possibly extremely complex input to try and minimise them to the smallest set of operations triggering them). Mutation testing alters the system under test to identify under-specified (under-tested) portions, rather than explore the input space.
- feanaro 6y agoThe system under test could be considered as an input to the specification / test suite. Mutation testing could then be viewed as fuzzing of the test suite, with every pass result after a mutation being considered a failure of the test suite.
- pfdietz 6y agoAnd with the property-based approach, test inputs can be generated that automatically increase coverage or that kill mutants. The bane of mutation testing is equivalent mutants: mutations that aren't actually bugs, because they leave the program correct.
- nullc 6y ago> The bane of mutation testing is equivalent mutants: mutations that aren't actually bugs, because they leave the program correct. That's one reason I check the hash of the optimized binaries-- often the compiler manages to convert different but equivalent code into the same binary. It's still not enough, but it's an easy filter to apply with no false exclusions.
- pfdietz 6y agoGood idea.
- dllthomas 6y agoArguably fuzzing is property testing with a constrained set of properties, although in practice the communities differ.
- AlexDenisov 6y agore: absence of C++ tools Have you looked at Mull[1] or Dextool[2]? [1] https://github.com/mull-project/mull https://github.com/mull-project/mull [2] https://github.com/joakim-brannstrom/dextool/tree/master/plugin/mutate https://github.com/joakim-brannstrom/dextool/tree/master/plu...
- pfdietz 6y agoAlso, it's possible to mutate binaries.