3 ms·

What would you say are the main differences between a fuzzer and QuickCheck? The authors of quickcheck don't call it a fuzzer so I assume there is some differen

What would you say are the main differences between a fuzzer and QuickCheck? The authors of quickcheck don't call it a fuzzer so I assume there is some difference but both seem to randomize inputs?

exdsq 5y ago
You're comparing fuzzing and property based testing. Fuzzing is a way of generating a load of random inputs for a system under test. For example if you had a function add(a, b) -> int you'd pass in numbers, strings, bytes, whatever you want and it'll let you know if it breaks. Property based testing is similar but you check if properties hold across these values and the tool will then shrink this to the simplest case. For example you'd say that add(a, b) takes two integers and regardless of the numbers it should always have the associative property. So in fuzzing you put in junk an see what happens, in pbt you define properties and check they hold across sane inputs.
- Too 5y ago
  Isn't fuzzing more than just random though? I was under the impression fuzzing could introspect branches in the code and work its way backwards to generate inputs to reach that coverage?
  - exdsq 5y ago
    Yeah it can get that advanced! But the distinction is still generating junk until you get errors while pbt is about checking business logic (properties) hold under random generators. Saying this I’m sure the cutting edge of both overlap!
Seirdy 5y ago
I think that fuzz testing is best described in the context of two other types of tests that go beyond "traditional" unit/integration tests: property and mutation testing. The three complement each other so well in ways that are hard to describe without trying them yourself. Quickcheck is a property tester, which essentially can be thought of as a "bytecode fuzzer that also happens to check for correct behavior". It uses an algorithm to generate arbitrary inputs that satisfy a set of constraints, and then ensures that the outputs demonstrate a property of the function being fulfilled. Think about mathematical properties of functions or the lack thereof: commutativity, associativity, and even properties of linear relations you might remember from Linear Algebra. Values that violate the expected properties are saved; you can build unit tests that check these values later or cache them for future runs. Fuzzing is a form of black-box testing that repeatedly runs software differently and tries to crash it. Causes and behavior of crashes can later be inspected. Mutation testing alters every statement in your code one at a time to change its behavior. Each alteration is a "mutant". If an alteration does not cause one of your tests to fail, the mutant has not been "killed". Your goal is to have a good kill ratio. This is a far better metric for test effectiveness and dead code elimination than statement/branch coverage. Property testing is kind of like a bytecode fuzzer and mutation testing is very much like a fuzzer for your tests. But property testing is a white-box form of testing with good knowledge of your code; I'd consuder mutation testing to be "grey-box" testing of your test code. "True" fuzz testing is as black-box as you can get: just let an algorithm run your program for a long time (anywhere from hours to months) to find subtle bugs (crashes and other black-box failures) at scales humans struggle with. In conclusion: Yo dawg, I heard you like tests. So I fuzzed the mutation tests on your property tests to see if arbitrary input on arbitrary code produced arbitrary output without arbitrary exits.
noctune 5y ago
IMO, it's much the same and pretty much only a question of API. Property-based testing (like QuickCheck) generates randomness using a PRNG, creates a structure from that, and checks that some property holds for that structure. A fuzzer generates some arbitrary input and usually checks that the program returns successfully (i.e. doesn't crash). The reason I think they are the same is that we can rather easily turn one into the other and vice versa: - A PBT test can be turned into a fuzzer-test by sourcing the randomness from the input instead of a PRNG and failing the program if the property does not hold. - A fuzz-test can be turned into a PBT by sourcing the input from a PRNG. The real difference is that fuzzers tend to be coverage-guided, i.e. a fuzzer will use what part of your code is executed to guide its mutations and thus steer it towards more "interesting" inputs.

What would you say are the main differences between a fuzzer and QuickCheck? The authors of quickcheck don't call it a fuzzer so I assume there is some difference but both seem to randomize inputs?

exdsq 5y ago

You're comparing fuzzing and property based testing. Fuzzing is a way of generating a load of random inputs for a system under test. For example if you had a function add(a, b) -> int you'd pass in numbers, strings, bytes, whatever you want and it'll let you know if it breaks. Property based testing is similar but you check if properties hold across these values and the tool will then shrink this to the simplest case. For example you'd say that add(a, b) takes two integers and regardless of the numbers it should always have the associative property. So in fuzzing you put in junk an see what happens, in pbt you define properties and check they hold across sane inputs.

Too 5y ago
Isn't fuzzing more than just random though? I was under the impression fuzzing could introspect branches in the code and work its way backwards to generate inputs to reach that coverage?
- exdsq 5y ago
  Yeah it can get that advanced! But the distinction is still generating junk until you get errors while pbt is about checking business logic (properties) hold under random generators. Saying this I’m sure the cutting edge of both overlap!

Seirdy 5y ago

I think that fuzz testing is best described in the context of two other types of tests that go beyond "traditional" unit/integration tests: property and mutation testing. The three complement each other so well in ways that are hard to describe without trying them yourself. Quickcheck is a property tester, which essentially can be thought of as a "bytecode fuzzer that also happens to check for correct behavior". It uses an algorithm to generate arbitrary inputs that satisfy a set of constraints, and then ensures that the outputs demonstrate a property of the function being fulfilled. Think about mathematical properties of functions or the lack thereof: commutativity, associativity, and even properties of linear relations you might remember from Linear Algebra. Values that violate the expected properties are saved; you can build unit tests that check these values later or cache them for future runs. Fuzzing is a form of black-box testing that repeatedly runs software differently and tries to crash it. Causes and behavior of crashes can later be inspected. Mutation testing alters every statement in your code one at a time to change its behavior. Each alteration is a "mutant". If an alteration does not cause one of your tests to fail, the mutant has not been "killed". Your goal is to have a good kill ratio. This is a far better metric for test effectiveness and dead code elimination than statement/branch coverage. Property testing is kind of like a bytecode fuzzer and mutation testing is very much like a fuzzer for your tests. But property testing is a white-box form of testing with good knowledge of your code; I'd consuder mutation testing to be "grey-box" testing of your test code. "True" fuzz testing is as black-box as you can get: just let an algorithm run your program for a long time (anywhere from hours to months) to find subtle bugs (crashes and other black-box failures) at scales humans struggle with. In conclusion: Yo dawg, I heard you like tests. So I fuzzed the mutation tests on your property tests to see if arbitrary input on arbitrary code produced arbitrary output without arbitrary exits.

noctune 5y ago

IMO, it's much the same and pretty much only a question of API. Property-based testing (like QuickCheck) generates randomness using a PRNG, creates a structure from that, and checks that some property holds for that structure. A fuzzer generates some arbitrary input and usually checks that the program returns successfully (i.e. doesn't crash). The reason I think they are the same is that we can rather easily turn one into the other and vice versa: - A PBT test can be turned into a fuzzer-test by sourcing the randomness from the input instead of a PRNG and failing the program if the property does not hold. - A fuzz-test can be turned into a PBT by sourcing the input from a PRNG. The real difference is that fuzzers tend to be coverage-guided, i.e. a fuzzer will use what part of your code is executed to guide its mutations and thus steer it towards more "interesting" inputs.