5 ms·

Fuzzing the .NET JIT Compiler

73 points by matthewwarren 8y ago

mehrdada 8y ago
An obvious next step to consider might be applying EMI techniques: https://mehrdad.afshari.me/publications/compiler-validation-via-equivalence-modulo-inputs.pdf https://mehrdad.afshari.me/publications/compiler-validation-...
- matthewwarren 8y ago
  Thanks for the link, I'd not heard of EMI techniques before, they look interesting
  - matt_d 8y ago
    If you're interested, here's a collection of resources on compilers correctness (testing & fuzzing, validation, and verification): https://github.com/MattPD/cpplinks/blob/master/compilers.correctness.md https://github.com/MattPD/cpplinks/blob/master/compilers.cor...
hobls 8y ago
> Or in other words, a fuzzer is a program that tries to create source code that finds bugs in a compiler. This is a very narrow definition of a fuzzer. There are a lot of types of fuzzer that do not generate source code, and are not intended to test compilers.
- seanmcdirmid 8y ago
  When I was an undergrad in 1997, I created a fuzzer (well, we didn't know to call it that back then) to insert one byte changes into Java classfiles, and then used that to test our bytecode verifier along with Sun and Microsoft's. Suffice it to say, we found lots of bugs, the most famous one being what my grad advisor called the vacuum bug that could read a web client's environment variables (among other things): https://archive.nytimes.com/www.nytimes.com/library/cyber/under/072397under.html https://archive.nytimes.com/www.nytimes.com/library/cyber/un...
  - kodablah 8y ago
    Tangentially related, speaking of Java, I wrote a path guided one recently for code running on the JVM: https://github.com/cretz/javan-warty-pig https://github.com/cretz/javan-warty-pig. It's found bugs for me on internal things, but I haven't run it on anything popular to build a trophy case.
  - hobls 8y ago
    Very cool! I think fuzzers are SO interesting. One of my favorite examples is lcamtuf’s post on generating valid JPGs from nothing [1]. Just so awesome. 1: https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-thin-air.html https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-th...
  - matthewwarren 8y ago
    Nice! Not many people can say that their university project made it into the New York Times!
- sehugg 8y ago
  To test a MS-DOS database program I was writing, I replaced the WaitForKey routine with GetRandom. It was fun to watch! And I think it was a fuzzer.
  - matthewwarren 8y ago
    Nice, that's a pretty cool approach! (random user input)
- matthewwarren 8y ago
  > This is a very narrow definition of a fuzzer. There are a lot of types of fuzzer that do not generate source code, and are not intended to test compilers Yeah, that's true, I completely missed out other uses for 'fuzzers', thanks for clarifying that
kodablah 8y ago
I wonder, instead of an AST-generating fuzzer, if they could have used AFL on an instrumented binary that may find more possibilities since it just works on paths and bytes without being constrained by what this fuzzer knows are certain constructs.
- ygra 8y ago
  Can you easily run AFL on .NET (or jitted) code? My understanding was that AFL works by instrumenting paths taken and kinda needed a gcc-produced binary with normal Linux debug symbol stuff.
  - 0xad 8y ago
    What AFL needs is _instrumentation_ and of course the easiest way is to get that at compilation step, however you are not constrained by anything to get that part via other means. Check https://github.com/ivanfratric/winafl https://github.com/ivanfratric/winafl that uses DynamoRIO. So, to answer your question -- it wouldn't be easy but it can be done.
  - matthewwarren 8y ago
    The .NET Profiling API gives you a way to modify IL code before it's JITted, so that could be one way to do this. See http://mattwarren.org/2018/08/21/Monitoring-and-Observability-in-the-.NET-Runtime/#profiling http://mattwarren.org/2018/08/21/Monitoring-and-Observabilit... for a bit more info ('SetILFunctionBody() API')
  - kodablah 8y ago
    I think there are Windows and VM-based approaches, but it probably isn't to difficult to mimic what AFL does on the CLR.
0xad 8y ago
Great article. Kudos. On a side node, my old project https://github.com/dyjakan/interpreter-bugs https://github.com/dyjakan/interpreter-bugs along with short presentation I did on WarCon 2017 https://github.com/dyjakan/conference-talks/blob/master/2017-WarCon-Interpreters-Under-Pressure.pdf https://github.com/dyjakan/conference-talks/blob/master/2017... (references might be interesting for others).
rkagerer 8y ago
Nice use of Beyond Compare for the side-by-side IL snapshots.
- matthewwarren 8y ago
  I'm glad you liked that! (I spent more time that I should've trying to show that in a meaningful way and I was pretty glad when I remembered about 'Beyond Compare')

73 points by matthewwarren 8y ago

An obvious next step to consider might be applying EMI techniques: https://mehrdad.afshari.me/publications/compiler-validation-via-equivalence-modulo-inputs.pdf https://mehrdad.afshari.me/publications/compiler-validation-...

matthewwarren 8y ago
Thanks for the link, I'd not heard of EMI techniques before, they look interesting
- matt_d 8y ago
  If you're interested, here's a collection of resources on compilers correctness (testing & fuzzing, validation, and verification): https://github.com/MattPD/cpplinks/blob/master/compilers.correctness.md https://github.com/MattPD/cpplinks/blob/master/compilers.cor...

hobls 8y ago

> Or in other words, a fuzzer is a program that tries to create source code that finds bugs in a compiler. This is a very narrow definition of a fuzzer. There are a lot of types of fuzzer that do not generate source code, and are not intended to test compilers.

seanmcdirmid 8y ago
When I was an undergrad in 1997, I created a fuzzer (well, we didn't know to call it that back then) to insert one byte changes into Java classfiles, and then used that to test our bytecode verifier along with Sun and Microsoft's. Suffice it to say, we found lots of bugs, the most famous one being what my grad advisor called the vacuum bug that could read a web client's environment variables (among other things): https://archive.nytimes.com/www.nytimes.com/library/cyber/under/072397under.html https://archive.nytimes.com/www.nytimes.com/library/cyber/un...
- kodablah 8y ago
  Tangentially related, speaking of Java, I wrote a path guided one recently for code running on the JVM: https://github.com/cretz/javan-warty-pig https://github.com/cretz/javan-warty-pig. It's found bugs for me on internal things, but I haven't run it on anything popular to build a trophy case.
- hobls 8y ago
  Very cool! I think fuzzers are SO interesting. One of my favorite examples is lcamtuf’s post on generating valid JPGs from nothing [1]. Just so awesome. 1: https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-thin-air.html https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-th...
- matthewwarren 8y ago
  Nice! Not many people can say that their university project made it into the New York Times!
sehugg 8y ago
To test a MS-DOS database program I was writing, I replaced the WaitForKey routine with GetRandom. It was fun to watch! And I think it was a fuzzer.
- matthewwarren 8y ago
  Nice, that's a pretty cool approach! (random user input)
matthewwarren 8y ago
> This is a very narrow definition of a fuzzer. There are a lot of types of fuzzer that do not generate source code, and are not intended to test compilers Yeah, that's true, I completely missed out other uses for 'fuzzers', thanks for clarifying that

kodablah 8y ago

I wonder, instead of an AST-generating fuzzer, if they could have used AFL on an instrumented binary that may find more possibilities since it just works on paths and bytes without being constrained by what this fuzzer knows are certain constructs.

ygra 8y ago
Can you easily run AFL on .NET (or jitted) code? My understanding was that AFL works by instrumenting paths taken and kinda needed a gcc-produced binary with normal Linux debug symbol stuff.
- 0xad 8y ago
  What AFL needs is _instrumentation_ and of course the easiest way is to get that at compilation step, however you are not constrained by anything to get that part via other means. Check https://github.com/ivanfratric/winafl https://github.com/ivanfratric/winafl that uses DynamoRIO. So, to answer your question -- it wouldn't be easy but it can be done.
- matthewwarren 8y ago
  The .NET Profiling API gives you a way to modify IL code before it's JITted, so that could be one way to do this. See http://mattwarren.org/2018/08/21/Monitoring-and-Observability-in-the-.NET-Runtime/#profiling http://mattwarren.org/2018/08/21/Monitoring-and-Observabilit... for a bit more info ('SetILFunctionBody() API')
- kodablah 8y ago
  I think there are Windows and VM-based approaches, but it probably isn't to difficult to mimic what AFL does on the CLR.

0xad 8y ago

Great article. Kudos. On a side node, my old project https://github.com/dyjakan/interpreter-bugs https://github.com/dyjakan/interpreter-bugs along with short presentation I did on WarCon 2017 https://github.com/dyjakan/conference-talks/blob/master/2017-WarCon-Interpreters-Under-Pressure.pdf https://github.com/dyjakan/conference-talks/blob/master/2017... (references might be interesting for others).

rkagerer 8y ago

Nice use of Beyond Compare for the side-by-side IL snapshots.

matthewwarren 8y ago
I'm glad you liked that! (I spent more time that I should've trying to show that in a meaningful way and I was pretty glad when I remembered about 'Beyond Compare')