4 ms·

This paper is absolutely fascinating. Fuzzers have some inherent nondeterminism, meaning that some runs agains the same program will differ in how many bugs th

by tyoma 8y ago

This paper is absolutely fascinating. Fuzzers have some inherent nondeterminism, meaning that some runs agains the same program will differ in how many bugs they find. It turns out pretty much every fuzzer evaluation to date has not been statistically rigorous. We dont actually know if a lot of published results are, well, really an improvement. There is also a fascinating bit towards the end. It turns out doing the dumb thing and not tracking code coverage finds more crashes faster than intelligent coverage tracking, but only with good seed inputs (see page 7).

hannob 8y ago
> It turns out pretty much every fuzzer evaluation to date has not been statistically rigorous. Shall I tell you a not so well hidden secret? Almost nothing published in computer science, let alone in IT security, is statistically rigorous.
- Ar-Curunir 8y ago
  That's not true; plenty of papers in the systems field perform careful and rigorous experiments. This doesn't even include the theory side of stuff, where you only have proofs, and no statistics to get wrong.
akavel 8y ago
I didn't initally understand this from your summary, so trying to rephrase it a bit to how I understand it after glancing over the paper: The paper's thesis is that most (?) currently claimed advances in fuzzing algorithms were not proven correctly enough (a.k.a. "in a statically rigorous/significant way") to actually be advances. Meaning, they may well have been just random fluctuations. Then, the paper proposes how to correctly compare fuzzers, and what the community needs to be able to do that (e.g. an agreed upon common body of codebases to be fuzzed, a set of known fuzzable bugs in those codebases, minimum fuzzing time, etc.)
mwhicks1 8y ago
The last comment is not justified by the paper. The paper shows one case where the dumb thing finds tons of crashes, but several other cases in the same figure where it finds zero crashes. And the figure there is just one target program. More experiments on this issue might be interesting. One other thing to point out: Finding crashes and finding bugs are not the same thing, as section 7 carefully argues. It could well be that many of those crashes are duplicates. The blog post I linked above also summarizes this point.
Ono-Sendai 8y ago
Fuzzers don't have to be non-deterministic.

by tyoma 8y ago

hannob 8y ago

> It turns out pretty much every fuzzer evaluation to date has not been statistically rigorous. Shall I tell you a not so well hidden secret? Almost nothing published in computer science, let alone in IT security, is statistically rigorous.

Ar-Curunir 8y ago
That's not true; plenty of papers in the systems field perform careful and rigorous experiments. This doesn't even include the theory side of stuff, where you only have proofs, and no statistics to get wrong.

akavel 8y ago

I didn't initally understand this from your summary, so trying to rephrase it a bit to how I understand it after glancing over the paper: The paper's thesis is that most (?) currently claimed advances in fuzzing algorithms were not proven correctly enough (a.k.a. "in a statically rigorous/significant way") to actually be advances. Meaning, they may well have been just random fluctuations. Then, the paper proposes how to correctly compare fuzzers, and what the community needs to be able to do that (e.g. an agreed upon common body of codebases to be fuzzed, a set of known fuzzable bugs in those codebases, minimum fuzzing time, etc.)

mwhicks1 8y ago

The last comment is not justified by the paper. The paper shows one case where the dumb thing finds tons of crashes, but several other cases in the same figure where it finds zero crashes. And the figure there is just one target program. More experiments on this issue might be interesting. One other thing to point out: Finding crashes and finding bugs are not the same thing, as section 7 carefully argues. It could well be that many of those crashes are duplicates. The blog post I linked above also summarizes this point.

Ono-Sendai 8y ago

Fuzzers don't have to be non-deterministic.