5 ms·

Someone (Hillel Wayne?) observed that fuzz testing and property testing are basically the same thing, but the communities are almost entirely disjoint so the to

by moomin 5y ago

Someone (Hillel Wayne?) observed that fuzz testing and property testing are basically the same thing, but the communities are almost entirely disjoint so the tools are completely separate.

Milner08 5y ago
Reading about fuzz testing all I could think of was 'is this not property testing?'... That is strange. It sounds like both communities could learn a lot from each other unless there is something I am missing (there probably is..)
- masklinn 5y ago
  In my mind fuzz testing is external, backboxed, and often profile-guided, while proptesting is more structured, internal (language aware) but generally less guided or entirely unguided. This here is closer to what I see as property testing than fuzzing, although it looks like they plan on coverage feedback (so guided generation).
  - ahelwer 5y ago
    Property-based testing requires you to define the condition of success or failure (the "property" you're testing for) too, right? Whereas fuzzing just looks for crashes?
    - masklinn 5y ago
      > Property-based testing requires you to define the condition of success or failure (the "property" you're testing for) too, right? That property can be "does not crash". And I'd say this is the structured / language-awareness part: with fuzzing you can't generally build an oracle. And if you can it's of course trivial: just have a wrapper script check the result against the oracle, and trigger whatever the fuzzer looks for indicating "failure" whether it's a return code or a segfault or…
- bottled_poe 5y ago
  Surely the coverage of fuzz testing is a superset of property testing?
  - amw-zero 5y ago
    Right. That’s the point - they are both conceptually the same thing, testing via automatically generated inputs.
- Kototama 5y ago
  Fuzzing is more to ensure safety and property checking to ensure correctness IMHO. Both are related but not similar.
rtpg 5y ago
I think Prop testing is great, but stuff like AFL (with instrumentation to basically find your conditionals for you, and working backwards to modify data for it) is very different from property testing. My experience (real world, for actually existing code) is that property tests often require a lot of fiddling with the data generation, in order to actually stress your system in interesting ways. If you just throw "totally random" data into your system, you won't be testing very interesting properties. Amazing assurances and payoff from doing it, of course! Just like... "just generate random arbitraries" works a lot less when you're working with 100 field structs and only 10 or so "matter" in senses you care about. To my knowledge I have not seen a property testing toolkit that leverages code coverage in the way fuzzing does.
- eru 5y ago
  Python's hypothesis did coverage guided property-based testing for a bit. They turned it off (not sure if they turned it on again): the problem is that property based tests of the kind implemented with hypothesis are mainly supposed to be run as part of your CI/CD test suit. So they should be fast. Coverage guided fuzzing like AFL does usually takes longer than people are prepared to wait for in their build/test server.
  - zhd 5y ago
    I've recently launched HypoFuzz - https://hypofuzz.com/ https://hypofuzz.com/ - which solves this by running it on a different server. You run the tests briefly in CI to check for regressions, and then leave them running permanently on a fuzz server to search for new bugs. Nelson Elhage has a good writeup of this approach at https://blog.nelhage.com/post/two-kinds-of-testing/ https://blog.nelhage.com/post/two-kinds-of-testing/
    - eru 5y ago
      Awesome!
- gsg 5y ago
  There are some examples, for example Crowbar is an OCaml tool that uses AFL to drive property based tests.
typical182 5y ago
People can have different definitions and still communicate usefully, and I think there is not 100% agreement on the exact boundaries between the two. That said, for me: they are distinct but related, and that distinction is useful. For example, Hypothesis[1] is a popular property testing framework. The authors have more recently created HypoFuzz[2], which includes this sentence in the introduction: “HypoFuzz runs your property-based test suite, using cutting-edge fuzzing techniques and coverage instrumentation to find even the rarest inputs which trigger an error.” Being able to talk about fuzzing and property testing as distinct things seems useful — saying something like “We added fuzzing techniques to our property testing framework” is more meaningful than “We added property testing techniques to our property testing framework” ;-) My personal hope is there will be more convergence, and work to add convenient first-class fuzzing support in a popular language like Go will hopefully help move the primary use case for fuzzing to be about correctness, with security moving to an important but secondary use case. [1] https://hypothesis.works https://hypothesis.works [2] https://hypofuzz.com https://hypofuzz.com
agumonkey 5y ago
the common point being, covering the input space right ?
Drup 5y ago
My understanding is that the difference between fuzz testing and property testing is how the input is crafted. Both can be viewed as a pair of things: a function to generate series of bits as input, and a way to turn these bits into the appropriate data under test. Property testing generates these bits using a specified distribution, and that's about it. Fuzz testing generates these bits by looking at how the program is executed, and uses a black box to try to explore all paths in the program. Most libraries for property testing comes with very convenient ways to craft the "input to data" part. Fuzz tools come with an almost magically effective way to craft interesting inputs. The two combines very well (and have been combined in several libraries).
- jlouis 5y ago
  Yep, but in the dual, they have the same goal: produce interesting inputs to the program which might exhibit trouble. This is why you can use one of the approaches to help the other side of the approach. The 3rd solution is concolic testing: use an SMT/SAT solver to flip branches. The path down to the branch imposes a formula. By inverting the formula, we can pick a certain branch path. Now you ask the SMT solver to check there there's no way to go down that branch. If it finds satisfiability, you have a counterexample and can explore that path.
  - UncleMeat 5y ago
    Being thematically similar is not that interesting, if one works better than the others. Coverage guided fuzzing is eating symbolic execution and concolic testing for breakfast. It isn't even close. As much as I love these more principled approaches, the strategy of throwing bits at programs and applying careful coverage metrics is just way more effective, even for cases that seem to be hand picked for SMT-based approaches to win.
- dnautics 5y ago
  > Property testing generates these bits using a specified distribution, and that's about it I think most property testing frameworks also come with the concept of "shrinkage", which is a way to walk back a failed condition to try to find the "minimum requirement of failure". Though I am sure there are PT frameworks that haven't implemented this.
px43 5y ago
I've heard that a bit recently too. I think it's more that people selling property testing tools are trying to sell them as fuzzing tools to unsuspecting suckers.
pag 5y ago
DeepState [1] is a tool that lets you write Google Test-style unit tests, as well as property tests, in either C or C++, and plug in fuzzers and symbolic executors. That is, DeepState bridges this gap between fuzz testing and property testing. [1] https://github.com/trailofbits/deepstate https://github.com/trailofbits/deepstate
staticassertion 5y ago
They are fundamentally the same in a lot of important ways. Any fuzz test is almost certainly (if not entirely certainly) a property test. In Rust the driver for property and fuzz testing can be shared, which is nice[0]. https://docs.rs/arbitrary/1.0.1/arbitrary/trait.Arbitrary.html https://docs.rs/arbitrary/1.0.1/arbitrary/trait.Arbitrary.ht... By describing my data with arbitrary I've written programs that had traditional property testing and fuzz-testing as well, without any additional effort. It's only really post-AFL where fuzzing has suddenly become synonymous with instrumentation guided data generation, which is a totally fine distinction to make, but they're still fundamentally equivalent. The first fuzzers were virtually identical to prop test frameworks today. I'd be fine with dropping the "fuzzing" name entirely and instead just having us use the term property testing, with "data generation" being the thing we start differentiating ie: "random prop testing" or "instrumentation guided prop testing" or "type based prop testing" etc.
dllthomas 5y ago
I expect the observation has been made many times, but one particular example of note is https://danluu.com/testing/ https://danluu.com/testing/

by moomin 5y ago

Someone (Hillel Wayne?) observed that fuzz testing and property testing are basically the same thing, but the communities are almost entirely disjoint so the tools are completely separate.

Milner08 5y ago

Reading about fuzz testing all I could think of was 'is this not property testing?'... That is strange. It sounds like both communities could learn a lot from each other unless there is something I am missing (there probably is..)

masklinn 5y ago
In my mind fuzz testing is external, backboxed, and often profile-guided, while proptesting is more structured, internal (language aware) but generally less guided or entirely unguided. This here is closer to what I see as property testing than fuzzing, although it looks like they plan on coverage feedback (so guided generation).
- ahelwer 5y ago
  Property-based testing requires you to define the condition of success or failure (the "property" you're testing for) too, right? Whereas fuzzing just looks for crashes?
  - masklinn 5y ago
    > Property-based testing requires you to define the condition of success or failure (the "property" you're testing for) too, right? That property can be "does not crash". And I'd say this is the structured / language-awareness part: with fuzzing you can't generally build an oracle. And if you can it's of course trivial: just have a wrapper script check the result against the oracle, and trigger whatever the fuzzer looks for indicating "failure" whether it's a return code or a segfault or…
bottled_poe 5y ago
Surely the coverage of fuzz testing is a superset of property testing?
- amw-zero 5y ago
  Right. That’s the point - they are both conceptually the same thing, testing via automatically generated inputs.
Kototama 5y ago
Fuzzing is more to ensure safety and property checking to ensure correctness IMHO. Both are related but not similar.

rtpg 5y ago

I think Prop testing is great, but stuff like AFL (with instrumentation to basically find your conditionals for you, and working backwards to modify data for it) is very different from property testing. My experience (real world, for actually existing code) is that property tests often require a lot of fiddling with the data generation, in order to actually stress your system in interesting ways. If you just throw "totally random" data into your system, you won't be testing very interesting properties. Amazing assurances and payoff from doing it, of course! Just like... "just generate random arbitraries" works a lot less when you're working with 100 field structs and only 10 or so "matter" in senses you care about. To my knowledge I have not seen a property testing toolkit that leverages code coverage in the way fuzzing does.

eru 5y ago
Python's hypothesis did coverage guided property-based testing for a bit. They turned it off (not sure if they turned it on again): the problem is that property based tests of the kind implemented with hypothesis are mainly supposed to be run as part of your CI/CD test suit. So they should be fast. Coverage guided fuzzing like AFL does usually takes longer than people are prepared to wait for in their build/test server.
- zhd 5y ago
  I've recently launched HypoFuzz - https://hypofuzz.com/ https://hypofuzz.com/ - which solves this by running it on a different server. You run the tests briefly in CI to check for regressions, and then leave them running permanently on a fuzz server to search for new bugs. Nelson Elhage has a good writeup of this approach at https://blog.nelhage.com/post/two-kinds-of-testing/ https://blog.nelhage.com/post/two-kinds-of-testing/
  - eru 5y ago
    Awesome!
gsg 5y ago
There are some examples, for example Crowbar is an OCaml tool that uses AFL to drive property based tests.

typical182 5y ago

People can have different definitions and still communicate usefully, and I think there is not 100% agreement on the exact boundaries between the two. That said, for me: they are distinct but related, and that distinction is useful. For example, Hypothesis[1] is a popular property testing framework. The authors have more recently created HypoFuzz[2], which includes this sentence in the introduction: “HypoFuzz runs your property-based test suite, using cutting-edge fuzzing techniques and coverage instrumentation to find even the rarest inputs which trigger an error.” Being able to talk about fuzzing and property testing as distinct things seems useful — saying something like “We added fuzzing techniques to our property testing framework” is more meaningful than “We added property testing techniques to our property testing framework” ;-) My personal hope is there will be more convergence, and work to add convenient first-class fuzzing support in a popular language like Go will hopefully help move the primary use case for fuzzing to be about correctness, with security moving to an important but secondary use case. [1] https://hypothesis.works https://hypothesis.works [2] https://hypofuzz.com https://hypofuzz.com

agumonkey 5y ago

the common point being, covering the input space right ?

Drup 5y ago

My understanding is that the difference between fuzz testing and property testing is how the input is crafted. Both can be viewed as a pair of things: a function to generate series of bits as input, and a way to turn these bits into the appropriate data under test. Property testing generates these bits using a specified distribution, and that's about it. Fuzz testing generates these bits by looking at how the program is executed, and uses a black box to try to explore all paths in the program. Most libraries for property testing comes with very convenient ways to craft the "input to data" part. Fuzz tools come with an almost magically effective way to craft interesting inputs. The two combines very well (and have been combined in several libraries).

jlouis 5y ago
Yep, but in the dual, they have the same goal: produce interesting inputs to the program which might exhibit trouble. This is why you can use one of the approaches to help the other side of the approach. The 3rd solution is concolic testing: use an SMT/SAT solver to flip branches. The path down to the branch imposes a formula. By inverting the formula, we can pick a certain branch path. Now you ask the SMT solver to check there there's no way to go down that branch. If it finds satisfiability, you have a counterexample and can explore that path.
- UncleMeat 5y ago
  Being thematically similar is not that interesting, if one works better than the others. Coverage guided fuzzing is eating symbolic execution and concolic testing for breakfast. It isn't even close. As much as I love these more principled approaches, the strategy of throwing bits at programs and applying careful coverage metrics is just way more effective, even for cases that seem to be hand picked for SMT-based approaches to win.
dnautics 5y ago
> Property testing generates these bits using a specified distribution, and that's about it I think most property testing frameworks also come with the concept of "shrinkage", which is a way to walk back a failed condition to try to find the "minimum requirement of failure". Though I am sure there are PT frameworks that haven't implemented this.

px43 5y ago

I've heard that a bit recently too. I think it's more that people selling property testing tools are trying to sell them as fuzzing tools to unsuspecting suckers.

pag 5y ago

DeepState [1] is a tool that lets you write Google Test-style unit tests, as well as property tests, in either C or C++, and plug in fuzzers and symbolic executors. That is, DeepState bridges this gap between fuzz testing and property testing. [1] https://github.com/trailofbits/deepstate https://github.com/trailofbits/deepstate

staticassertion 5y ago

They are fundamentally the same in a lot of important ways. Any fuzz test is almost certainly (if not entirely certainly) a property test. In Rust the driver for property and fuzz testing can be shared, which is nice[0]. https://docs.rs/arbitrary/1.0.1/arbitrary/trait.Arbitrary.html https://docs.rs/arbitrary/1.0.1/arbitrary/trait.Arbitrary.ht... By describing my data with arbitrary I've written programs that had traditional property testing and fuzz-testing as well, without any additional effort. It's only really post-AFL where fuzzing has suddenly become synonymous with instrumentation guided data generation, which is a totally fine distinction to make, but they're still fundamentally equivalent. The first fuzzers were virtually identical to prop test frameworks today. I'd be fine with dropping the "fuzzing" name entirely and instead just having us use the term property testing, with "data generation" being the thing we start differentiating ie: "random prop testing" or "instrumentation guided prop testing" or "type based prop testing" etc.

dllthomas 5y ago

I expect the observation has been made many times, but one particular example of note is https://danluu.com/testing/ https://danluu.com/testing/