3 ms·
I think that this is because QC can work with much richer types than fuzzers have access to. Think about the Haskell data types you know, and imagine automatica
by munin 10y ago
I think that this is because QC can work with much richer types than fuzzers have access to. Think about the Haskell data types you know, and imagine automatically generating them. Sounds pretty easy with some algorithm, right? Now think about the Java data types you know, and the invariants on constructing them. Yeah sure they have constructors but what preconditions do those constructors require? anything impure, like I/O? are there setters? is an object really "constructed" until some setters have been called? if so, how do you know that as the test generator?
It seems a lot nastier to me, which I think is why QC style automatic testing has made less inroads in impure languages.
It also makes fewer inroads to fuzzing because fuzzing starts a few steps back where you know even less about the application. You could do a grammar-based fuzz of the application, but then you have to worry about the map/terrain of the grammar/application. Are you missing testing some case in the application because your grammar is incomplete?
In QC, the map is the terrain because the "grammar" is just a type that your application uses and this isn't really the case in fuzzing. It could be, in fuzzing some languages (like Java) where you could interrogate the program you'r testing (even a binary program) for types that the program uses. Think about C programs though, where the application input and the "types" that the application uses are (or can be) pretty distant.
Anyway, I don't think it's disquieting. I think they're two very different software assurance techniques that apply in two very different domains and that explains why people don't say too much about them in the same breath. They're united by randomness, but I don't think that much more.
- cmrx64 10y agoSpeaking of, do you know of any tools for doing grammar-based fuzzing?
- munin 10y agopeachfuzz, quickfuzz, dharma, grammfuzz. They get done a bunch and usually people wind up re-inventing lisp to make a data specification DSL, then write some generators around it and call it a day. The generation strategy usually winds up being domain specific too (are you looking for OOB bugs? dangling refs / UAF? different patterns in different structured data lead to these errors without feedback) so honestly you should pick some parser combinator in a language that you know (like construct for python, or hammer for C) and then turn it into a fuzzer for your application yourself, using the faults you want to find as inspiration for the generators you write.
- rixrax 10y agoPeach Fuzzer[p] and Synopsys Defensics[d] should fit the bill. Both are commercial. I'm not aware of any OSS grammar-based fuzzers that would be good for anything (ok; there is Sulley and some others which seemingly haven't been maintained in eons). [p] www.peachfuzzer.com/ [d] https://www.synopsys.com/software-integrity/security-testing/fuzz-testing.html https://www.synopsys.com/software-integrity/security-testing...
- eru 10y agoYou don't need good static types for QuickCheck like checking. See Python's hypothesis (https://hypothesis.readthedocs.io/en/latest/ https://hypothesis.readthedocs.io/en/latest/) for the refutation. Letting the generation be guided by static types was just a good design decision for the original Haskell version of the library---it really helps with convenience.
- lomnakkus 10y agoWhat you say is, of course, true[1]. However, I find that the types actually do have a huge benefit, namely that you're "encouraged" to give more accurate types for your functions. For example: "frobnicate :: String -> String" has a huge state space, and if it actually really only works on a subset of String and you want to test it you have to manually provide "overrides" for the value generators... which is semi-painful if you have to do it for every test. So instead you add newtypes and change it to "frobnicate :: Blampf -> Schmeckle" and move all the "test-only" code into the Blampf and Schmeckle constructors[2] instead. [1] I was actually about to footnote here that it was invented in Erlang-land, but apparently not: https://en.wikipedia.org/wiki/QuickCheck https://en.wikipedia.org/wiki/QuickCheck . Not sure why I thought that, maybe just because Joe Armstrong has been an advocate of QC-style testing for a long time? Or maybe it was he who invented QC in Haskell? [2] Which, granted, in Haskell will mostly be run-time checked, but at least you'll get an error at the earliest possible opportunity.