5 ms·

Dr. Chaos – A structured fuzzing framework in Nim

64 points by planetis 4y ago

cjbprime 4y ago
Surprisingly, structured fuzzing (mutating inputs to a grammar's generator) has proven less effective than optimized traditional random fuzzing, so far. https://google.github.io/fuzzbench/ https://google.github.io/fuzzbench/ shows that grammar fuzzers (like AFLSmart) find less coverage per minute than standard fuzzers like aflplusplus, even for the cases where the grammar fuzzer has an extremely comprehensive grammar for that format.
- planetis 4y ago
  Interesting, how does FormatFuzzer or LPM rank? The one is a generator based format fuzzer and the other a mutator based structured fuzzer. And which benchmarks compare any of those two?
  - cjbprime 4y ago
    I'm not sure! Maybe someone can encourage the authors to submit to Fuzzbench. My intuition for the results so far is that (a) converting generator functions to bitstrings is very computationally expensive (since the comparison is coverage found per unit of time), and (b) perhaps most memory safety vulnerabilities may be occuring in technically invalid or grammatically inconsistent inputs that the grammar will never find.
    - planetis 4y ago
      I think we need fuzzers for more uses than file formats, like method fuzzing and in that case garbage data just doesn't work.
      - cjbprime 4y ago
        This is also a testable claim! File formats and RPC do not seem that dissimilar to me, especially when using a fuzzer like aflplusplus, which will automatically build a symbol dictionary by pulling strings out of the binary, and perform some light symbolic transformation (laf-intel and cmplog), see https://lafintel.wordpress.com/2016/08/15/circumventing-fuzzing-roadblocks-with-compiler-transformations/ https://lafintel.wordpress.com/2016/08/15/circumventing-fuzz...
      - planetis 4y ago
        It's just sound weird to me because depending of who you ask, they will tell you generator fuzzers work the best. However defining what's a good benchmark and what to measure other than code coverage is even more complicated. But I guess we are talking about google they might have figured it out.
        cjbprime 4y ago
        Yes, that's why I described the empirical results as surprising.
hahaitsfunny 4y ago

64 points by planetis 4y ago

Surprisingly, structured fuzzing (mutating inputs to a grammar's generator) has proven less effective than optimized traditional random fuzzing, so far. https://google.github.io/fuzzbench/ https://google.github.io/fuzzbench/ shows that grammar fuzzers (like AFLSmart) find less coverage per minute than standard fuzzers like aflplusplus, even for the cases where the grammar fuzzer has an extremely comprehensive grammar for that format.

planetis 4y ago
Interesting, how does FormatFuzzer or LPM rank? The one is a generator based format fuzzer and the other a mutator based structured fuzzer. And which benchmarks compare any of those two?
- cjbprime 4y ago
  I'm not sure! Maybe someone can encourage the authors to submit to Fuzzbench. My intuition for the results so far is that (a) converting generator functions to bitstrings is very computationally expensive (since the comparison is coverage found per unit of time), and (b) perhaps most memory safety vulnerabilities may be occuring in technically invalid or grammatically inconsistent inputs that the grammar will never find.
  - planetis 4y ago
    I think we need fuzzers for more uses than file formats, like method fuzzing and in that case garbage data just doesn't work.
    - cjbprime 4y ago
      This is also a testable claim! File formats and RPC do not seem that dissimilar to me, especially when using a fuzzer like aflplusplus, which will automatically build a symbol dictionary by pulling strings out of the binary, and perform some light symbolic transformation (laf-intel and cmplog), see https://lafintel.wordpress.com/2016/08/15/circumventing-fuzzing-roadblocks-with-compiler-transformations/ https://lafintel.wordpress.com/2016/08/15/circumventing-fuzz...
    - planetis 4y ago
      It's just sound weird to me because depending of who you ask, they will tell you generator fuzzers work the best. However defining what's a good benchmark and what to measure other than code coverage is even more complicated. But I guess we are talking about google they might have figured it out.
      - cjbprime 4y ago
        Yes, that's why I described the empirical results as surprising.

hahaitsfunny 4y ago