15 ms·

Afl.rs: Fuzzing Rust code with american-fuzzy-lop

155 points by yberreby 10y ago

cpach 10y ago
Nice! It’s already proven to be useful: https://github.com/frewsxcv/afl.rs#trophy-case https://github.com/frewsxcv/afl.rs#trophy-case
- legulere 10y ago
  And all the bugs just caused panics, while they could have been exploitable in C.
  - fulafel 10y ago
    The ASN.1 one is a different kind of crash though still sounds controlled (SIGILL).
    - eddyb 10y ago
      That's LLVM's abort intrinsic - no a lot of those left around, I believe OOM, panicking while panicking and stack overflow use it or used to use it.
  - comex 10y ago
    Of note, this report in rustc (from the list) was apparently true heap corruption due to a bug in unsafe code: https://github.com/rust-lang/rust/issues/24276 https://github.com/rust-lang/rust/issues/24276
- zamalek 10y ago
  Someone on the brotli team is absolutely smitten with the tool.
  - StavrosK 10y ago
    What's not to be smitten about? It's like a magical bug finder.
  - masklinn 10y ago
    It seems to be more that the afl-rs dev used brotli-rs as testing grounds.
    - frewsxcv 10y ago
      If anything, it felt like I was practicing some sort of fuzz-driven-development for brotli-rs knocking out a bunch of corner cases. Coincidentally though, I am fuzzing a different Brotli library testing some unpushed afl.rs changes: https://github.com/dropbox/rust-brotli-no-stdlib https://github.com/dropbox/rust-brotli-no-stdlib
loktarogar 10y ago
I expected this to be something about Australian football, but this is pretty good too
- frewsxcv 10y ago
  Yes, the shared acronym makes Googling for anything AFL related rather annoying. The results are usually a mixture of Australian football, American rabbits, and fuzz testing.
cm3 10y ago
Any idea as to why the following requirement? This will limit Afl.rs users quite a bit. afl.rs needs to compile against a version of LLVM that matches rustc's. The easy solution (if you can wait on a slow build) is to build rustc from source and put it in your PATH. Then afl.rs's build script will find llvm-config automatically. Otherwise, the environment variable LLVM_CONFIG should hold the path to llvm-config when you build afl.rs. I was under the impression that Afl can test any application that takes stdin. I'm underinformed for sure, so what's the idea behind explicitly adding code to support Afl fuzzing?
- fennecfoxen 10y ago
  This might be working with libfuzzer, one of the tools afl lists on its homepage as derivatives/offshoots: http://llvm.org/docs/LibFuzzer.html http://llvm.org/docs/LibFuzzer.html afl is fundamentally instrumentation-based and likes a good look at what sorts of different branches a program is taking, it's a little bit more hands-on than just feeding things to STDIN.
  - sanxiyn 10y ago
    No, afl.rs is not using libFuzzer. afl.rs is using llvm_mode code in afl.
- zuzun 10y ago
  I think it injects code at branches to observe the branch coverage. After all you have to compile your C programs with the afl-gcc binary. Maybe you could ptrace every single instruction executed by the program and then stop at every jump and use the relative values of the IP to identify branches... I don't know if that would work, but it would be much slower.
- masklinn 10y ago
  Data input goes through stdin, but AFL needs to instrument the binary to observe branch coverage and changes in branch coverage across varying inputs. From the alf-fuzz site: > American fuzzy lop is a security-oriented fuzzer that employs a novel type of compile-time instrumentation and genetic algorithms to automatically discover clean, interesting test cases that trigger new internal states in the targeted binary. That's why you need to compile C programs with afl-gcc and C++ programs with afl-g++: http://lcamtuf.coredump.cx/afl/QuickStartGuide.txt http://lcamtuf.coredump.cx/afl/QuickStartGuide.txt
ekidd 10y ago
Randomly looking at the "Trophy Case", it looks like most of these errors are run-time "panics". We can break Rust errors down into three main categories: 1. Compile-time errors. This includes most memory-related errors, which are mostly caught by the borrow checker. These are very serious errors, often with ugly security consequences. Rust's big selling point is that it can catch many different kinds of errors at compile time—but not all. 2. Run-time panics. This includes "index out of bound" errors, integer overflow errors (in debug builds only), and assertions inserted by the programmer. This is Rust's second line of defense, so to speak. 3. Expected run-time errors. These are mostly reported using return values of type Error, which is the normal way to handle errors in Rust. Most of the errors caught by AFL seem to be errors in group (2) that ought to be in group (3). In most cases, these errors couldn't be moved into group (1), because they're not the kind of thing that's easily caught at compile-time. So this is a really cool tool for Rust developers, especially ones working on libraries that parse untrusted input. I was especially impressed by the fact that AFL could discover overflow errors, which Rust normally only protects against in Debug mode.
- eddyb 10y ago
  > These are mostly reported using return values of type Error However, unlike more dynamic languages, the return types are usually Result<SuccessValue, SomeErrorType>, e.g. writing to a file or a socket will get you Result<usize, io::Error>, values of which can be Ok(usize) (the number of bytes written) or Err(io::Error) (an I/O error which you can further inspect). Although if you want to, you can use Result<T, Box<Error>> where Error is a trait implemented by types representing some error information. You then lose the ability to inspect errors other than for reporting them wholesale, but the APIs are slightly more uniform, type-wise. You can read about all of this and more in the dedicated chapter of the official Rust Book: https://doc.rust-lang.org/book/error-handling.html https://doc.rust-lang.org/book/error-handling.html.
  - ekidd 10y ago
    > However, unlike more dynamic languages, the return types are usually Result<SuccessValue, SomeErrorType> Yes, this is a more complete and precise explanation. :-) I rather like Rust's error handling in practice—it adds only a small amount of syntactic noise, and even the annoying bits (such as converting and wrapping error types) can mostly be isolated in a single file per project.
pjmlp 10y ago
> Nightly build of Rust Oh well....
- frewsxcv 10y ago
  Regarding using stable builds of Rust with afl.rs: If I could, I would. afl.rs uses a Rust compiler plugin to register the LLVM pass needed to instrument Rust programs so that AFL can run on them: https://github.com/frewsxcv/afl.rs/blob/master/afl-plugin/lib.rs#L7-L16 https://github.com/frewsxcv/afl.rs/blob/master/afl-plugin/li... Compiler plugins in Rust are currently unstable, and as far as I know, are not on a path to stability in the near-term. https://doc.rust-lang.org/book/compiler-plugins.html https://doc.rust-lang.org/book/compiler-plugins.html
  - steveklabnik 10y ago
    Depends on how you define "near term." It's being actively worked on, but is still gonna take a while.
  - Manishearth 10y ago
    It doesn't need to use llvm_mode. You can do what gcc does and have an afl-gcc binary which mucks with the assembly instead of actually inserting hooks smartly. This isn't as good, but works without nightly :)
    - frewsxcv 10y ago
      Huh, wasn't aware that's how afl-gcc worked. I'll have to look into it sometime, thanks for making me aware of it.
- Manishearth 10y ago
  This is unfair. AFL support for Rust is in fact better than that for gcc because Rust at least provides a plugin interface that doesn't require you to have its sources lying around -- the GCC support instruments the assembly instead of compiler IR, which is far less effective. Rust nightly at least has proper afl support, gcc doesn't. It works for clang because llvm plugins in clang are stable, but we could get there easily -- in fact stabilizing something akin to `-Xclang` is probably something that could be done in Rust today without issue. A stable version of afl-rs could exist that does the same thing gcc does, as I mentioned below. As far as compiler tools are concerned, I'm mostly okay with stuff being nightly-only. These dependencies do not pollute the dependency graph and are something you only deal with locally.
deleted 10y ago
[deleted]

155 points by yberreby 10y ago

cpach 10y ago

Nice! It’s already proven to be useful: https://github.com/frewsxcv/afl.rs#trophy-case https://github.com/frewsxcv/afl.rs#trophy-case

legulere 10y ago
And all the bugs just caused panics, while they could have been exploitable in C.
- fulafel 10y ago
  The ASN.1 one is a different kind of crash though still sounds controlled (SIGILL).
  - eddyb 10y ago
    That's LLVM's abort intrinsic - no a lot of those left around, I believe OOM, panicking while panicking and stack overflow use it or used to use it.
- comex 10y ago
  Of note, this report in rustc (from the list) was apparently true heap corruption due to a bug in unsafe code: https://github.com/rust-lang/rust/issues/24276 https://github.com/rust-lang/rust/issues/24276
zamalek 10y ago
Someone on the brotli team is absolutely smitten with the tool.
- StavrosK 10y ago
  What's not to be smitten about? It's like a magical bug finder.
- masklinn 10y ago
  It seems to be more that the afl-rs dev used brotli-rs as testing grounds.
  - frewsxcv 10y ago
    If anything, it felt like I was practicing some sort of fuzz-driven-development for brotli-rs knocking out a bunch of corner cases. Coincidentally though, I am fuzzing a different Brotli library testing some unpushed afl.rs changes: https://github.com/dropbox/rust-brotli-no-stdlib https://github.com/dropbox/rust-brotli-no-stdlib

loktarogar 10y ago

I expected this to be something about Australian football, but this is pretty good too

frewsxcv 10y ago
Yes, the shared acronym makes Googling for anything AFL related rather annoying. The results are usually a mixture of Australian football, American rabbits, and fuzz testing.

cm3 10y ago

Any idea as to why the following requirement? This will limit Afl.rs users quite a bit. afl.rs needs to compile against a version of LLVM that matches rustc's. The easy solution (if you can wait on a slow build) is to build rustc from source and put it in your PATH. Then afl.rs's build script will find llvm-config automatically. Otherwise, the environment variable LLVM_CONFIG should hold the path to llvm-config when you build afl.rs. I was under the impression that Afl can test any application that takes stdin. I'm underinformed for sure, so what's the idea behind explicitly adding code to support Afl fuzzing?

fennecfoxen 10y ago
This might be working with libfuzzer, one of the tools afl lists on its homepage as derivatives/offshoots: http://llvm.org/docs/LibFuzzer.html http://llvm.org/docs/LibFuzzer.html afl is fundamentally instrumentation-based and likes a good look at what sorts of different branches a program is taking, it's a little bit more hands-on than just feeding things to STDIN.
- sanxiyn 10y ago
  No, afl.rs is not using libFuzzer. afl.rs is using llvm_mode code in afl.
zuzun 10y ago
I think it injects code at branches to observe the branch coverage. After all you have to compile your C programs with the afl-gcc binary. Maybe you could ptrace every single instruction executed by the program and then stop at every jump and use the relative values of the IP to identify branches... I don't know if that would work, but it would be much slower.
masklinn 10y ago
Data input goes through stdin, but AFL needs to instrument the binary to observe branch coverage and changes in branch coverage across varying inputs. From the alf-fuzz site: > American fuzzy lop is a security-oriented fuzzer that employs a novel type of compile-time instrumentation and genetic algorithms to automatically discover clean, interesting test cases that trigger new internal states in the targeted binary. That's why you need to compile C programs with afl-gcc and C++ programs with afl-g++: http://lcamtuf.coredump.cx/afl/QuickStartGuide.txt http://lcamtuf.coredump.cx/afl/QuickStartGuide.txt

ekidd 10y ago

Randomly looking at the "Trophy Case", it looks like most of these errors are run-time "panics". We can break Rust errors down into three main categories: 1. Compile-time errors. This includes most memory-related errors, which are mostly caught by the borrow checker. These are very serious errors, often with ugly security consequences. Rust's big selling point is that it can catch many different kinds of errors at compile time—but not all. 2. Run-time panics. This includes "index out of bound" errors, integer overflow errors (in debug builds only), and assertions inserted by the programmer. This is Rust's second line of defense, so to speak. 3. Expected run-time errors. These are mostly reported using return values of type Error, which is the normal way to handle errors in Rust. Most of the errors caught by AFL seem to be errors in group (2) that ought to be in group (3). In most cases, these errors couldn't be moved into group (1), because they're not the kind of thing that's easily caught at compile-time. So this is a really cool tool for Rust developers, especially ones working on libraries that parse untrusted input. I was especially impressed by the fact that AFL could discover overflow errors, which Rust normally only protects against in Debug mode.

eddyb 10y ago
> These are mostly reported using return values of type Error However, unlike more dynamic languages, the return types are usually Result<SuccessValue, SomeErrorType>, e.g. writing to a file or a socket will get you Result<usize, io::Error>, values of which can be Ok(usize) (the number of bytes written) or Err(io::Error) (an I/O error which you can further inspect). Although if you want to, you can use Result<T, Box<Error>> where Error is a trait implemented by types representing some error information. You then lose the ability to inspect errors other than for reporting them wholesale, but the APIs are slightly more uniform, type-wise. You can read about all of this and more in the dedicated chapter of the official Rust Book: https://doc.rust-lang.org/book/error-handling.html https://doc.rust-lang.org/book/error-handling.html.
- ekidd 10y ago
  > However, unlike more dynamic languages, the return types are usually Result<SuccessValue, SomeErrorType> Yes, this is a more complete and precise explanation. :-) I rather like Rust's error handling in practice—it adds only a small amount of syntactic noise, and even the annoying bits (such as converting and wrapping error types) can mostly be isolated in a single file per project.

pjmlp 10y ago

> Nightly build of Rust Oh well....

frewsxcv 10y ago
Regarding using stable builds of Rust with afl.rs: If I could, I would. afl.rs uses a Rust compiler plugin to register the LLVM pass needed to instrument Rust programs so that AFL can run on them: https://github.com/frewsxcv/afl.rs/blob/master/afl-plugin/lib.rs#L7-L16 https://github.com/frewsxcv/afl.rs/blob/master/afl-plugin/li... Compiler plugins in Rust are currently unstable, and as far as I know, are not on a path to stability in the near-term. https://doc.rust-lang.org/book/compiler-plugins.html https://doc.rust-lang.org/book/compiler-plugins.html
- steveklabnik 10y ago
  Depends on how you define "near term." It's being actively worked on, but is still gonna take a while.
- Manishearth 10y ago
  It doesn't need to use llvm_mode. You can do what gcc does and have an afl-gcc binary which mucks with the assembly instead of actually inserting hooks smartly. This isn't as good, but works without nightly :)
  - frewsxcv 10y ago
    Huh, wasn't aware that's how afl-gcc worked. I'll have to look into it sometime, thanks for making me aware of it.
Manishearth 10y ago
This is unfair. AFL support for Rust is in fact better than that for gcc because Rust at least provides a plugin interface that doesn't require you to have its sources lying around -- the GCC support instruments the assembly instead of compiler IR, which is far less effective. Rust nightly at least has proper afl support, gcc doesn't. It works for clang because llvm plugins in clang are stable, but we could get there easily -- in fact stabilizing something akin to `-Xclang` is probably something that could be done in Rust today without issue. A stable version of afl-rs could exist that does the same thing gcc does, as I mentioned below. As far as compiler tools are concerned, I'm mostly okay with stuff being nightly-only. These dependencies do not pollute the dependency graph and are something you only deal with locally.

deleted 10y ago

[deleted]