4 ms·

First fuzzer I wrote (we didn't call them that, we were just trying to find bugs triggered by bad packets in a cable TV protocol suite) made random bad packets

First fuzzer I wrote (we didn't call them that, we were just trying to find bugs triggered by bad packets in a cable TV protocol suite) made random bad packets - by making good ones and then mutating them past some random point - it found 3 bugs (which was amazing at the time) but the real problem with all these systems is (in terms of my problem ): - the number of possible bad packets is literally billions of times larger than good ones - pretty soon the percentage of those that trigger bad behaviour gets close to 0

corndoge 6y ago
"all these systems" means black box fuzzing all modern fuzzing is coverage guided for this exact reason
jsnell 6y ago
Modern fuzzing tools are far more effective at probing the state space than just random mutations. It's hard to appreciate just how effective until you see it in practice. E.g. last time I fuzzed a network element with AFL, it took seconds from it to go from a starting corpus of a single ethernet IPv4 SYN packet to some double-encapsulated IP-in-NSH-IP-in-NSH-ethernet monstrosity that triggered a misparse. And seconds more for it to generate a IPv6 packet with a fragment extension header that triggered some other problem. A random walk would have no chance of finding that.
- mentat 6y ago
  Do you know of any good writeups for this particular kind of process? I too did fuzzing of network devices before it was called fuzzing and am interested in trying it again with modern tooling.
  - woodruffw 6y ago
    The AFL technical details document[1] is a decent reference for one particular subset of fuzzer feedback: code coverage metrics. [1]: https://lcamtuf.coredump.cx/afl//technical_details.txt https://lcamtuf.coredump.cx/afl//technical_details.txt
  - not2b 6y ago
    The electronic design automation world calls it "constrained random simulation" and has been using the technique for two decades for hardware verification, using the same kind of coverage-driven methodology that modern fuzzers use, though in some ways the problem is simpler with a synchronous hardware model where the state space is explicit.
- Taniwha 6y ago
  My problem at the time was BAD packets (ones that were still occasionally getting past a CRC check after noise) we had pretty good good packet tests
  - jsnell 6y ago
    Right, and a guided fuzzer would be effective at finding distinct classes of bad packets, where "bad" is defined as triggering your error function. Those complex packets that AFL was able to create from thin air were bad in the sense that they triggered bugs in our system, not bad in the sense of being malformed. The latter is interesting only insofar as those malformed packets trigger a bug. Btw, and maybe I'm misunderstanding what you wrote, if you were generating random packets without fixing up the checksum, you were already wasting basically all of your testing capacity. All that it ends up doing is checking that the negative case of checksum verification works.
    - gizmo686 6y ago
      For people who are not familiar with guided fuzzers, it would not surprise me if AFL actually managed to get good checksums on its own. I have seen it consistantly produce "magic" strings; presumably by walking the strcmp function calls, where each individual character comparison is another oppurtunity for execution to take a different path.

corndoge 6y ago

"all these systems" means black box fuzzing all modern fuzzing is coverage guided for this exact reason

jsnell 6y ago

Modern fuzzing tools are far more effective at probing the state space than just random mutations. It's hard to appreciate just how effective until you see it in practice. E.g. last time I fuzzed a network element with AFL, it took seconds from it to go from a starting corpus of a single ethernet IPv4 SYN packet to some double-encapsulated IP-in-NSH-IP-in-NSH-ethernet monstrosity that triggered a misparse. And seconds more for it to generate a IPv6 packet with a fragment extension header that triggered some other problem. A random walk would have no chance of finding that.

mentat 6y ago
Do you know of any good writeups for this particular kind of process? I too did fuzzing of network devices before it was called fuzzing and am interested in trying it again with modern tooling.
- woodruffw 6y ago
  The AFL technical details document[1] is a decent reference for one particular subset of fuzzer feedback: code coverage metrics. [1]: https://lcamtuf.coredump.cx/afl//technical_details.txt https://lcamtuf.coredump.cx/afl//technical_details.txt
- not2b 6y ago
  The electronic design automation world calls it "constrained random simulation" and has been using the technique for two decades for hardware verification, using the same kind of coverage-driven methodology that modern fuzzers use, though in some ways the problem is simpler with a synchronous hardware model where the state space is explicit.
Taniwha 6y ago
My problem at the time was BAD packets (ones that were still occasionally getting past a CRC check after noise) we had pretty good good packet tests
- jsnell 6y ago
  Right, and a guided fuzzer would be effective at finding distinct classes of bad packets, where "bad" is defined as triggering your error function. Those complex packets that AFL was able to create from thin air were bad in the sense that they triggered bugs in our system, not bad in the sense of being malformed. The latter is interesting only insofar as those malformed packets trigger a bug. Btw, and maybe I'm misunderstanding what you wrote, if you were generating random packets without fixing up the checksum, you were already wasting basically all of your testing capacity. All that it ends up doing is checking that the negative case of checksum verification works.
  - gizmo686 6y ago
    For people who are not familiar with guided fuzzers, it would not surprise me if AFL actually managed to get good checksums on its own. I have seen it consistantly produce "magic" strings; presumably by walking the strcmp function calls, where each individual character comparison is another oppurtunity for execution to take a different path.