4 ms·

Fuzzing Java in OSS-Fuzz

115 points by theafh 6y ago

The_rationalist 6y ago
Another great tool is Pitest for mutation testing on the JVM (make coverage more meaningful) https://github.com/hcoles/pitest https://github.com/hcoles/pitest
- spockz 6y ago
  If you want an easy way to have better mutation coverage, check out property based testing. Eg junit-quickcheck for Java. https://github.com/pholser/junit-quickcheck https://github.com/pholser/junit-quickcheck
  - The_rationalist 6y ago
    Thanks!
  - exdsq 6y ago
    How does property based testing relate to mutation testing, unless you just mean you'll have a larger test vector?
  - chriswarbo 6y ago
    I've found Scalacheck quite pleasant (run through Scalatest)
fhenneke 6y ago
One of the authors of Jazzer here. Feel free to ask any questions regarding Jazzer (https://github.com/CodeIntelligenceTesting/jazzer https://github.com/CodeIntelligenceTesting/jazzer) or how to integrate Java/JVM projects into OSS-Fuzz.
- victor106 6y ago
  This is awesome!!! going to explore and start using it. Thanks for working on this.
- exdsq 6y ago
  I've been interested in applying fuzzing to some projects I work on but every time I go to do this I'm not really sure where to start. Do you have any recommended sources to learn about how to use them in practice? :)
  - fhenneke 6y ago
    That depends on the language you want to fuzz. A good general introduction and hands-on "course" for C/C++ is https://github.com/Dor1s/libfuzzer-workshop https://github.com/Dor1s/libfuzzer-workshop. If you prefer Java and just want to get a feeling for how concrete fuzz targets can look like, take a look at the Jazzer examples at https://github.com/CodeIntelligenceTesting/jazzer/tree/main/examples/src/main/java/com/example https://github.com/CodeIntelligenceTesting/jazzer/tree/main/....
lucb1e 6y ago
> fuzzing code written in memory safe languages, such as JVM-based languages, is useful for finding bugs where code behaves incorrectly or crashes. Incorrect behavior can be just as dangerous as memory corruption. For example, Jazzer was used to find CVE-2021-23899 in json-sanitizer which could be exploited for cross-site scripting (XSS). Bugs causing crashes or incorrect exceptions can sometimes be used for denial of service. So the prime examples are denial of service and cross-site scripting. Definitely dangerous and good to apply fuzzing, but I do want to emphasize that this is a different class than all the memory corruption vulnerabilities we're seeing in C(-like) languages. Executing arbitrary code on systems in many cases is definitely worse than executing code in a browser with the privileges of the currently logged-in user. As someone working in security, it seems to me like moving away from memory-unsafe languages is long overdue by 2020---scratch that, 2021.
- fhenneke 6y ago
  While I agree with most of what you are saying (let's hope that 2021 will finally be "the year of memory-safe languages"), remote code execution issues also affect Java. https://securitylab.github.com/research/securing-the-fight-against-covid19-through-oss https://securitylab.github.com/research/securing-the-fight-a... is just one of the latest prominent examples.
  - lucb1e 6y ago
    Yep, for sure. One of the things we see most frequently leading to code execution is still deserialization... but it's much more rare now, so a step forwards from my point of view.
spullara 6y ago
I was hoping this was about fuzzing OpenJDK, but alas.
- fhenneke 6y ago
  With minor changes to the default exclude list for coverage instrumentation, you can use Jazzer to fuzz the Java standard library (at least the parts that are implemented in Java).
  - spullara 6y ago
    Was mostly thinking about things like the garbage collector, JIT, and other native code.
saagarjha 6y ago
Recent discussion of Jazzer: https://news.ycombinator.com/item?id=26090139 https://news.ycombinator.com/item?id=26090139

115 points by theafh 6y ago

Another great tool is Pitest for mutation testing on the JVM (make coverage more meaningful) https://github.com/hcoles/pitest https://github.com/hcoles/pitest

spockz 6y ago
If you want an easy way to have better mutation coverage, check out property based testing. Eg junit-quickcheck for Java. https://github.com/pholser/junit-quickcheck https://github.com/pholser/junit-quickcheck
- The_rationalist 6y ago
  Thanks!
- exdsq 6y ago
  How does property based testing relate to mutation testing, unless you just mean you'll have a larger test vector?
- chriswarbo 6y ago
  I've found Scalacheck quite pleasant (run through Scalatest)

fhenneke 6y ago

One of the authors of Jazzer here. Feel free to ask any questions regarding Jazzer (https://github.com/CodeIntelligenceTesting/jazzer https://github.com/CodeIntelligenceTesting/jazzer) or how to integrate Java/JVM projects into OSS-Fuzz.

victor106 6y ago
This is awesome!!! going to explore and start using it. Thanks for working on this.
exdsq 6y ago
I've been interested in applying fuzzing to some projects I work on but every time I go to do this I'm not really sure where to start. Do you have any recommended sources to learn about how to use them in practice? :)
- fhenneke 6y ago
  That depends on the language you want to fuzz. A good general introduction and hands-on "course" for C/C++ is https://github.com/Dor1s/libfuzzer-workshop https://github.com/Dor1s/libfuzzer-workshop. If you prefer Java and just want to get a feeling for how concrete fuzz targets can look like, take a look at the Jazzer examples at https://github.com/CodeIntelligenceTesting/jazzer/tree/main/examples/src/main/java/com/example https://github.com/CodeIntelligenceTesting/jazzer/tree/main/....

lucb1e 6y ago

> fuzzing code written in memory safe languages, such as JVM-based languages, is useful for finding bugs where code behaves incorrectly or crashes. Incorrect behavior can be just as dangerous as memory corruption. For example, Jazzer was used to find CVE-2021-23899 in json-sanitizer which could be exploited for cross-site scripting (XSS). Bugs causing crashes or incorrect exceptions can sometimes be used for denial of service. So the prime examples are denial of service and cross-site scripting. Definitely dangerous and good to apply fuzzing, but I do want to emphasize that this is a different class than all the memory corruption vulnerabilities we're seeing in C(-like) languages. Executing arbitrary code on systems in many cases is definitely worse than executing code in a browser with the privileges of the currently logged-in user. As someone working in security, it seems to me like moving away from memory-unsafe languages is long overdue by 2020---scratch that, 2021.

fhenneke 6y ago
While I agree with most of what you are saying (let's hope that 2021 will finally be "the year of memory-safe languages"), remote code execution issues also affect Java. https://securitylab.github.com/research/securing-the-fight-against-covid19-through-oss https://securitylab.github.com/research/securing-the-fight-a... is just one of the latest prominent examples.
- lucb1e 6y ago
  Yep, for sure. One of the things we see most frequently leading to code execution is still deserialization... but it's much more rare now, so a step forwards from my point of view.

spullara 6y ago

I was hoping this was about fuzzing OpenJDK, but alas.

fhenneke 6y ago
With minor changes to the default exclude list for coverage instrumentation, you can use Jazzer to fuzz the Java standard library (at least the parts that are implemented in Java).
- spullara 6y ago
  Was mostly thinking about things like the garbage collector, JIT, and other native code.

saagarjha 6y ago

Recent discussion of Jazzer: https://news.ycombinator.com/item?id=26090139 https://news.ycombinator.com/item?id=26090139