3 ms·

The Common Thread: Fuzzing, Bug Triage, and Attacker Automation

38 points by jsnell 10y ago

jonhohle 10y ago
I've had very good experiences with AFL on a medium sized C++ code base. I ran it for 8 CPU months on the code base and was reasonably confident with the results. It generates samples for you, making the bugs easy to reproduce. Along with valgrind, most ended up being reasonably shallow - lots of out of bounds, uninitialized use, use after free, and incorrectly handles return codes from 3ᴿᴰ party libraries. I recall getting false positives around hangs, but replaying all the generated test files made that easy to confirm. Fortunately, the code base I was testing had an extensive set of input files for testing already.
- jonhohle 10y ago
  As a quick follow up, one of the things I found surprising were that some of these bugs could have resulted in DOS for some significantly important systems, and that the code was written by some of the smartest people I have ever worked with. If these people were unable to write secure C++, it really made me question whether anyone could reliably do it in a business environment.
  - microcolonel 10y ago
    I think good code and a business environment just don't go hand-in-hand. I work for a company where one of our core values is writing clean code. We have fired clients who compromise too much on quality. But despite this, corners are always cut when there's an arbitrary deadline; and in enterprise, there is always an arbitrary deadline.
tptacek 10y ago
The middle part of this post is concerned with the DARPA Cyber Grand Challenge, which was a contest in which CGC generated dozens of small broken C programs that did network I/O, and contestants had to use fully automated systems to find vulnerabilities in them. The through-line on this Dave Aitel post is that despite a decade of effort on high-level CS approaches to finding bugs, like lifting binaries to predicate expressions and running solvers on them, most bugs are still found by something not much more sophisticated than "cat /dev/urandom | program".
- munin 10y ago
  that kind of reduction is frustrating because not all bugs are equal. are all bugs found by fuzzing? does the value of the bugs found by means other than fuzzing outweigh the resources spent to find them?
  - tptacek 10y ago
    That's what Aitel is questioning.
    - munin 10y ago
      then he should run some numbers or make an argument that takes a position?
- dgacmu 10y ago
  That's not really true. Good fuzzers integrate automatic coverage tracking to help guide their input space exploration. It's not full symbolic execution, but it helps a ton. See, for example, libFuzzer.
  - tptacek 10y ago
    That's systems work, not computer science work. Fuzzers also don't literally just cat /dev/urandom. :)

38 points by jsnell 10y ago

I've had very good experiences with AFL on a medium sized C++ code base. I ran it for 8 CPU months on the code base and was reasonably confident with the results. It generates samples for you, making the bugs easy to reproduce. Along with valgrind, most ended up being reasonably shallow - lots of out of bounds, uninitialized use, use after free, and incorrectly handles return codes from 3ᴿᴰ party libraries. I recall getting false positives around hangs, but replaying all the generated test files made that easy to confirm. Fortunately, the code base I was testing had an extensive set of input files for testing already.

jonhohle 10y ago
As a quick follow up, one of the things I found surprising were that some of these bugs could have resulted in DOS for some significantly important systems, and that the code was written by some of the smartest people I have ever worked with. If these people were unable to write secure C++, it really made me question whether anyone could reliably do it in a business environment.
- microcolonel 10y ago
  I think good code and a business environment just don't go hand-in-hand. I work for a company where one of our core values is writing clean code. We have fired clients who compromise too much on quality. But despite this, corners are always cut when there's an arbitrary deadline; and in enterprise, there is always an arbitrary deadline.

tptacek 10y ago

The middle part of this post is concerned with the DARPA Cyber Grand Challenge, which was a contest in which CGC generated dozens of small broken C programs that did network I/O, and contestants had to use fully automated systems to find vulnerabilities in them. The through-line on this Dave Aitel post is that despite a decade of effort on high-level CS approaches to finding bugs, like lifting binaries to predicate expressions and running solvers on them, most bugs are still found by something not much more sophisticated than "cat /dev/urandom | program".

munin 10y ago
that kind of reduction is frustrating because not all bugs are equal. are all bugs found by fuzzing? does the value of the bugs found by means other than fuzzing outweigh the resources spent to find them?
- tptacek 10y ago
  That's what Aitel is questioning.
  - munin 10y ago
    then he should run some numbers or make an argument that takes a position?
dgacmu 10y ago
That's not really true. Good fuzzers integrate automatic coverage tracking to help guide their input space exploration. It's not full symbolic execution, but it helps a ton. See, for example, libFuzzer.
- tptacek 10y ago
  That's systems work, not computer science work. Fuzzers also don't literally just cat /dev/urandom. :)