I'm one of the authors of the paper. Two quick things I want to say: I've written a blog post that is a short but rigorous description of the paper's results:

I'm one of the authors of the paper. Two quick things I want to say: I've written a blog post that is a short but rigorous description of the paper's results: http://www.pl-enthusiast.net/2018/08/23/evaluating-empirical-evaluations-for-fuzz-testing/ http://www.pl-enthusiast.net/2018/08/23/evaluating-empirical... The paper does not make strong statements of the form, "fuzzers are/do X" or "all prior papers' claims are bogus." Rather, we say that the standard of evidence should be higher, and we demonstrate why failing to reach that standard could result in bogus claims. It's quite possible that a paper's idea really is an improvement. But it's also possible that additional evidence would cast doubt on that, or nuance it. For example, our experiments show that AFLFast probably does improve on AFL, though perhaps not as much as that paper made out (note we didn't do enough experiments ourselves to make that definitive statement).