4 ms·

Fuzzing vs. Property Testing

62 points by chwolfe 7y ago

ehsankia 7y ago
Related, I've been having a lot of success with Hypothesis [0], the Python property-based testing library. [0] https://hypothesis.readthedocs.io/en/latest/ https://hypothesis.readthedocs.io/en/latest/
- SloopJon 7y ago
  Hypothesis is my favorite. I also use TestCheck.js [0] for JavaScript and RapidCheck for C++ [1], but the Hypothesis generators seem the smartest, and they're easy to customize. [0] http://leebyron.com/testcheck-js/api http://leebyron.com/testcheck-js/api [1] https://github.com/emil-e/rapidcheck https://github.com/emil-e/rapidcheck
kcsrk 7y ago
Of interest is Crowbar[0,1], a testing tool for OCaml that combines Quickcheck like property testing with AFL fuzz. [0] https://github.com/stedolan/crowbar https://github.com/stedolan/crowbar [1] https://ocaml.org/meetings/ocaml/2017/extended-abstract__2017__stephen-dolan_mindy-preston__testing-with-crowbar.pdf https://ocaml.org/meetings/ocaml/2017/extended-abstract__201...
stevekemp 7y ago
One of the things that I like about go is that the standard-library really encourages the use of writing test-cases. Most of the bigger projects have good coverage, and I try hard to get my own. But nothing beats the sheer tenacity of running your code through a fuzz-tester. I've written a simple virtual machine, which interprets bytecode, and a BASIC interpreter amongst other things recently. Both of these projects benefited hugely from fuzz-testing, despite having high coverage via manually-written test-cases. Fuzz testing is cheap and largely automatic, so if it takes a few hours or a few days to find an interesting result that's not a problem. I remember the first time I tested my interpreter when I had ~90% coverage of the code with my test-cases and it crashed via fuzzing within seconds! Magic!
dnautics 7y ago
A good property testing framework has two properties, 1) inputs increase in complexity over generations, and 2) properties can shrink, enabling you to find "minimum error conditions", so property tests can help you identify the source of the error too.
carlmr 7y ago
Is there something like a good proptest in C++?
- patrec 7y ago
  https://github.com/emil-e/rapidcheck https://github.com/emil-e/rapidcheck is not bad.

62 points by chwolfe 7y ago

Related, I've been having a lot of success with Hypothesis [0], the Python property-based testing library. [0] https://hypothesis.readthedocs.io/en/latest/ https://hypothesis.readthedocs.io/en/latest/

SloopJon 7y ago
Hypothesis is my favorite. I also use TestCheck.js [0] for JavaScript and RapidCheck for C++ [1], but the Hypothesis generators seem the smartest, and they're easy to customize. [0] http://leebyron.com/testcheck-js/api http://leebyron.com/testcheck-js/api [1] https://github.com/emil-e/rapidcheck https://github.com/emil-e/rapidcheck

kcsrk 7y ago

Of interest is Crowbar[0,1], a testing tool for OCaml that combines Quickcheck like property testing with AFL fuzz. [0] https://github.com/stedolan/crowbar https://github.com/stedolan/crowbar [1] https://ocaml.org/meetings/ocaml/2017/extended-abstract__2017__stephen-dolan_mindy-preston__testing-with-crowbar.pdf https://ocaml.org/meetings/ocaml/2017/extended-abstract__201...

stevekemp 7y ago

One of the things that I like about go is that the standard-library really encourages the use of writing test-cases. Most of the bigger projects have good coverage, and I try hard to get my own. But nothing beats the sheer tenacity of running your code through a fuzz-tester. I've written a simple virtual machine, which interprets bytecode, and a BASIC interpreter amongst other things recently. Both of these projects benefited hugely from fuzz-testing, despite having high coverage via manually-written test-cases. Fuzz testing is cheap and largely automatic, so if it takes a few hours or a few days to find an interesting result that's not a problem. I remember the first time I tested my interpreter when I had ~90% coverage of the code with my test-cases and it crashed via fuzzing within seconds! Magic!

dnautics 7y ago

A good property testing framework has two properties, 1) inputs increase in complexity over generations, and 2) properties can shrink, enabling you to find "minimum error conditions", so property tests can help you identify the source of the error too.

carlmr 7y ago

Is there something like a good proptest in C++?

patrec 7y ago
https://github.com/emil-e/rapidcheck https://github.com/emil-e/rapidcheck is not bad.