4 ms·

I think Prop testing is great, but stuff like AFL (with instrumentation to basically find your conditionals for you, and working backwards to modify data for it

by rtpg 5y ago

I think Prop testing is great, but stuff like AFL (with instrumentation to basically find your conditionals for you, and working backwards to modify data for it) is very different from property testing. My experience (real world, for actually existing code) is that property tests often require a lot of fiddling with the data generation, in order to actually stress your system in interesting ways. If you just throw "totally random" data into your system, you won't be testing very interesting properties. Amazing assurances and payoff from doing it, of course! Just like... "just generate random arbitraries" works a lot less when you're working with 100 field structs and only 10 or so "matter" in senses you care about. To my knowledge I have not seen a property testing toolkit that leverages code coverage in the way fuzzing does.

eru 5y ago
Python's hypothesis did coverage guided property-based testing for a bit. They turned it off (not sure if they turned it on again): the problem is that property based tests of the kind implemented with hypothesis are mainly supposed to be run as part of your CI/CD test suit. So they should be fast. Coverage guided fuzzing like AFL does usually takes longer than people are prepared to wait for in their build/test server.
- zhd 5y ago
  I've recently launched HypoFuzz - https://hypofuzz.com/ https://hypofuzz.com/ - which solves this by running it on a different server. You run the tests briefly in CI to check for regressions, and then leave them running permanently on a fuzz server to search for new bugs. Nelson Elhage has a good writeup of this approach at https://blog.nelhage.com/post/two-kinds-of-testing/ https://blog.nelhage.com/post/two-kinds-of-testing/
  - eru 5y ago
    Awesome!
gsg 5y ago
There are some examples, for example Crowbar is an OCaml tool that uses AFL to drive property based tests.
deleted 5y ago
[deleted]

by rtpg 5y ago

eru 5y ago

Python's hypothesis did coverage guided property-based testing for a bit. They turned it off (not sure if they turned it on again): the problem is that property based tests of the kind implemented with hypothesis are mainly supposed to be run as part of your CI/CD test suit. So they should be fast. Coverage guided fuzzing like AFL does usually takes longer than people are prepared to wait for in their build/test server.

zhd 5y ago
I've recently launched HypoFuzz - https://hypofuzz.com/ https://hypofuzz.com/ - which solves this by running it on a different server. You run the tests briefly in CI to check for regressions, and then leave them running permanently on a fuzz server to search for new bugs. Nelson Elhage has a good writeup of this approach at https://blog.nelhage.com/post/two-kinds-of-testing/ https://blog.nelhage.com/post/two-kinds-of-testing/
- eru 5y ago
  Awesome!

gsg 5y ago

There are some examples, for example Crowbar is an OCaml tool that uses AFL to drive property based tests.

deleted 5y ago

[deleted]