4 ms·

I would strongly recommend fuzzing for testing any software. You are likely to find interesting things quickly, if the software hasn't been fuzzed before. I fu

I would strongly recommend fuzzing for testing any software. You are likely to find interesting things quickly, if the software hasn't been fuzzed before. I fuzzed early clang for segfaults, just for fun. I found dozens of bugs by just taking a large preprocessed C++ file, deleting a few lines, and then when finding a segfault, reducing by deleting lines and tokens until I reached a minimal example of the same error (make sure you are crashing in the same place, or you will probably find you keep rediscovering the same bug). Reported some great fun minimal crashes, like: :) :{} namespace I also fuzz in my day job. There we want to fuzz valid inputs -- we have a bunch of ways of generating two inputs which should produce the same output -- generate a pair of inputs, test outputs, repeat. It has found hundreds of bugs, and almost no bugs (famous last words) have been reported in releases.

stevekemp 8y ago
Seconded. I reported bugs in GNU Awk, found via fuzzing, and discovered that fingerprinting bogus SSH keys could result in null-pointer dereferences. Of course I also discovered bugs in my own software! (In my case I fuzzed a simple virtual machine and the display-mode of a console-based email-client.) It is disappointing how often it is trivial to find bugs via fuzzing, which I guess just underlines the point that we SHOULD be fuzzing more often. Even on programs/libraries that are "old".
MaxBarraclough 8y ago
You're doing fine work! > I found dozens of bugs by just taking a large preprocessed C++ file, deleting a few lines, and then when finding a segfault, reducing by deleting lines and tokens until I reached a minimal example of the same error To run with this idea: this process could be automated, and applied to any number of different compilers. Out of curiosity, did these bugs tend to be things like buffer-overflows? How many of the bugs you reported do you think would have been avoided had the compiler been written in a 'safe' language rather than C++?
- CJefferson 8y ago
  Actually, many of the bugs were asserts triggering (segfault isn't quite right). In the case of C++, there are just so many weird states the compiler can find itself in, particularly in the case of clang which tries hard to produce good errors, so keeps parsing even after an error has been found.
  - MaxBarraclough 8y ago
    > many of the bugs were asserts triggering I'd say that's the best kind of bug. Reflects well on the codebase. C++ parsing is an infamously untidy problem.
jwilk 8y ago
Minimizing testcases can (and should!) be automated. For C(-like) code, there's venerable Delta (http://delta.tigris.org/ http://delta.tigris.org/) and more powerful C-Reduce (https://embed.cs.utah.edu/creduce/ https://embed.cs.utah.edu/creduce/). For binary formats, there's afl-tmin, part of AFL (http://lcamtuf.coredump.cx/afl/ http://lcamtuf.coredump.cx/afl/).
- CJefferson 8y ago
  Don't worry, I did automate! However, I did it myself, for the experience. Some horrible bash and awk scripts did sufficiently for my purposes.
kakwa_ 8y ago
Fuzzing is really helpful. Even trying really hard to check for buffer overflow, it's nearly impossible to catch it all. Kudos for the AFL creator http://lcamtuf.coredump.cx/afl/ http://lcamtuf.coredump.cx/afl/ I would love if it was easier to do "external checks" (for example network fuzzing). Something like a dissociation of the "submit payload" and the "tortured piece of software" would be really really nice.

stevekemp 8y ago

Seconded. I reported bugs in GNU Awk, found via fuzzing, and discovered that fingerprinting bogus SSH keys could result in null-pointer dereferences. Of course I also discovered bugs in my own software! (In my case I fuzzed a simple virtual machine and the display-mode of a console-based email-client.) It is disappointing how often it is trivial to find bugs via fuzzing, which I guess just underlines the point that we SHOULD be fuzzing more often. Even on programs/libraries that are "old".

MaxBarraclough 8y ago

You're doing fine work! > I found dozens of bugs by just taking a large preprocessed C++ file, deleting a few lines, and then when finding a segfault, reducing by deleting lines and tokens until I reached a minimal example of the same error To run with this idea: this process could be automated, and applied to any number of different compilers. Out of curiosity, did these bugs tend to be things like buffer-overflows? How many of the bugs you reported do you think would have been avoided had the compiler been written in a 'safe' language rather than C++?

CJefferson 8y ago
Actually, many of the bugs were asserts triggering (segfault isn't quite right). In the case of C++, there are just so many weird states the compiler can find itself in, particularly in the case of clang which tries hard to produce good errors, so keeps parsing even after an error has been found.
- MaxBarraclough 8y ago
  > many of the bugs were asserts triggering I'd say that's the best kind of bug. Reflects well on the codebase. C++ parsing is an infamously untidy problem.

jwilk 8y ago

Minimizing testcases can (and should!) be automated. For C(-like) code, there's venerable Delta (http://delta.tigris.org/ http://delta.tigris.org/) and more powerful C-Reduce (https://embed.cs.utah.edu/creduce/ https://embed.cs.utah.edu/creduce/). For binary formats, there's afl-tmin, part of AFL (http://lcamtuf.coredump.cx/afl/ http://lcamtuf.coredump.cx/afl/).

CJefferson 8y ago
Don't worry, I did automate! However, I did it myself, for the experience. Some horrible bash and awk scripts did sufficiently for my purposes.

kakwa_ 8y ago

Fuzzing is really helpful. Even trying really hard to check for buffer overflow, it's nearly impossible to catch it all. Kudos for the AFL creator http://lcamtuf.coredump.cx/afl/ http://lcamtuf.coredump.cx/afl/ I would love if it was easier to do "external checks" (for example network fuzzing). Something like a dissociation of the "submit payload" and the "tortured piece of software" would be really really nice.