4 ms·

Fuzz testing is probably one of the most underused tactics, at least in my observations. It's easy to write a decently thorough fuzzer that will absolutely crus

Fuzz testing is probably one of the most underused tactics, at least in my observations. It's easy to write a decently thorough fuzzer that will absolutely crush a wide class of bugs that you have.

thebruce87m 8y ago
Do you have an example of this? A tutorial or specific tool to get a taste of what’s involved?
- pickdenis 8y ago
  Well, not anything I can easily share, but here's a simple-enough example. I was writing code to handle base-pair sequences (ACTG) in a very strange encoding, which naturally lead to tons of edge cases. I carefully wrote my code for appending two sequences together, but was not totally sure I got it right, but decided after writing a few unit tests that it was probably correct. Sure enough, a couple of months later, I received bug reports that, after debugging for a while, I pinned down to sequences not being appended properly. I stared at the code for some time and couldn't see any mishandled edge cases. It was then I realized that I could automatically generate an arbitrary amount of unit tests, because it's simple enough to call strcat and encode that to get the correct result. So, that's what I did: generate two strings of ACTG, encode them, concat, and compare against the result against that of the encoding of the two strings concatenated. Sure enough, a mishandled edge case popped up after trying a couple hundred strings. There was probably no way I could have found that bug without this approach. It's a really simple concept, but you need a couple of things: first, a very wide input space. Next, you need an alternate way of verifying the result. If you just want your program to not crash, you get this for free. In my case above, I could apply the two functions (concatenate/encode) in the other order to easily calculate the expected result. If you have these two things, a fuzzer is the logical thing to write to make sure your code is rock solid.
guidovranken 8y ago
Fuzzers are extremely underused. Even in large, mature projects, a fuzzer being part of the source tree is much more an exception than the rule. Once you've done it a few times, it takes only 5 minutes to write a basic fuzzer for a function that takes a binary blob or a string as an input, after which you can run it indefinitely. But chances are it will find a bug worthy of a fix within minutes. Various "safe" languages are just as vulnerable to denial-of-service vulnerabilities as C/C++. Since I started fuzzing Go projects I've come across numerous parsers of untrusted data that flat-out crash if a slice is accessed out-of-bounds. Deep recursions (A() calls B() calls A() etc) can crash both Go and Rust. And in probably every general programming language you can implement logic that leads to excessive resource allocation or computation, and unintended infinite loops. These kinds of bugs can obviously be problematic for network-facing components. A royal use of runtime assertions in debug builds for arbitrary invariants/conditions combined with fuzzing usually uncovers many more bugs.
- pjmlp 8y ago
  The big difference is that C and C++ usually don't crash, rather silently corrupt memory with all possible outcomes. If it is a server process, it can even corrupt data for days until someone actually notices it, if ever. Or they just crash in a totally unrelated memory segment a couple of hours later. Crashing when the out-of-bounds takes place is a much better outcome.

Fuzz testing is probably one of the most underused tactics, at least in my observations. It's easy to write a decently thorough fuzzer that will absolutely crush a wide class of bugs that you have.

thebruce87m 8y ago

Do you have an example of this? A tutorial or specific tool to get a taste of what’s involved?

pickdenis 8y ago
Well, not anything I can easily share, but here's a simple-enough example. I was writing code to handle base-pair sequences (ACTG) in a very strange encoding, which naturally lead to tons of edge cases. I carefully wrote my code for appending two sequences together, but was not totally sure I got it right, but decided after writing a few unit tests that it was probably correct. Sure enough, a couple of months later, I received bug reports that, after debugging for a while, I pinned down to sequences not being appended properly. I stared at the code for some time and couldn't see any mishandled edge cases. It was then I realized that I could automatically generate an arbitrary amount of unit tests, because it's simple enough to call strcat and encode that to get the correct result. So, that's what I did: generate two strings of ACTG, encode them, concat, and compare against the result against that of the encoding of the two strings concatenated. Sure enough, a mishandled edge case popped up after trying a couple hundred strings. There was probably no way I could have found that bug without this approach. It's a really simple concept, but you need a couple of things: first, a very wide input space. Next, you need an alternate way of verifying the result. If you just want your program to not crash, you get this for free. In my case above, I could apply the two functions (concatenate/encode) in the other order to easily calculate the expected result. If you have these two things, a fuzzer is the logical thing to write to make sure your code is rock solid.

guidovranken 8y ago

Fuzzers are extremely underused. Even in large, mature projects, a fuzzer being part of the source tree is much more an exception than the rule. Once you've done it a few times, it takes only 5 minutes to write a basic fuzzer for a function that takes a binary blob or a string as an input, after which you can run it indefinitely. But chances are it will find a bug worthy of a fix within minutes. Various "safe" languages are just as vulnerable to denial-of-service vulnerabilities as C/C++. Since I started fuzzing Go projects I've come across numerous parsers of untrusted data that flat-out crash if a slice is accessed out-of-bounds. Deep recursions (A() calls B() calls A() etc) can crash both Go and Rust. And in probably every general programming language you can implement logic that leads to excessive resource allocation or computation, and unintended infinite loops. These kinds of bugs can obviously be problematic for network-facing components. A royal use of runtime assertions in debug builds for arbitrary invariants/conditions combined with fuzzing usually uncovers many more bugs.

pjmlp 8y ago
The big difference is that C and C++ usually don't crash, rather silently corrupt memory with all possible outcomes. If it is a server process, it can even corrupt data for days until someone actually notices it, if ever. Or they just crash in a totally unrelated memory segment a couple of hours later. Crashing when the out-of-bounds takes place is a much better outcome.