8 ms·

Announcing OSS-Fuzz: Continuous Fuzzing for Open Source Software

212 points by tanin 10y ago

newman314 10y ago
I'd like to see openssh added to the list of fuzzed projects.
- wyldfire 10y ago
  Adding a new project doesn't seem too tough. Guide: https://github.com/google/oss-fuzz/blob/master/docs/new_project_guide.md https://github.com/google/oss-fuzz/blob/master/docs/new_proj...
  - newman314 10y ago
    I'm not against doing the work to create the pull (unless someone else wants to) but looking at https://www.openssh.com/report.html https://www.openssh.com/report.html, it's not clear to me if openssh[at]openssh.com or openssh-unix-dev[at]mindrot.org should be used for something like this.
    - greglindahl 10y ago
      Try firing it at yourself to start with, then ask the openssh community once you've got some useful data? Edit: Thanks for the downvote! Every time I consider sending in a pile of automated bug-reports to a project where I'm not already part of the community, I look at them myself first. If that's bad, I'm happy to be bad.
orf 10y ago
Awesome, it's found a few bugs with Sqlite3 already: https://bugs.chromium.org/p/oss-fuzz/issues/list?can=1&q=type%3DBug-Security%2CBug+-component%3AInfra+status%3AFixed%2CVerified+sqlite3&sort=-id&colspec=ID+Type+Component+Status+Library+Reported+Owner+Summary&cells=ids https://bugs.chromium.org/p/oss-fuzz/issues/list?can=1&q=typ...
- deleted 10y ago
  [deleted]
mtgx 10y ago
> In order for a project to be accepted to OSS-Fuzz, it needs to have a large user base and/or be critical to Global IT infrastructure Let's see - Firefox and/or the Tor browser? I imagine Google wouldn't be too happy about doing free security research for Firefox, but it seems to fit the bill quite well for the goals and mission of the Core Infrastructure Initiative organization.
- orf 10y ago
  Why wouldn't they be happy? Google's a diverse company, it's not like every department/team has an unwritten goal to push Chrome at the expense of everything else.
  - ronjouch 10y ago
    > "it's not like every department/team has an unwritten goal to push Chrome at the expense of everything else." It often feels like it, yes. Could it be that you don't see it because you are already using Chrome? I use Edge on my Windows gaming machine (for minimal maintenance, I install drivers, Steam, and that's pretty much it) and configured Edge to clean everything including cookies on exit, like a permanent stateless Private Mode available for the occasional search. End result: many Google apps (Search, Maps, YouTube, and probably many others) constantly display huge "Get Chrome" banners, that end up not only painful for the user experience, but contain plain lies: "A more modern, faster browser" says one, whereas Edge/Firefox/Chrome are now objectively in the same ballpark. This constant soliciting across vast swaths of the web by one dominant actor controlling the end-to-end experience is abusive.
    - orf 10y ago
      I use Firefox and I agree with you, I don't like the "Use Chrome" banners that I see when using IE (i dont get them with ff). However, baring that, is there anything else they've done? IIRC they still pay Mozilla for making Google the default homepage
      - ronjouch 10y ago
nchammas 10y ago
On the topic of fuzz testing, Python has an excellent library for property-based testing called Hypothesis [0] [1]. I don't think it does guided testing like AFL or libFuzzer (which OSS-Fuzz uses), but it's very powerful nonetheless. [0] http://hypothesis.works/ http://hypothesis.works/ [1] https://github.com/HypothesisWorks/hypothesis-python https://github.com/HypothesisWorks/hypothesis-python
- wyldfire 10y ago
  I wrote a wrapper around CPython for fuzz testing with libFuzzer [1]. It's dog slow because it's designed for anyone to be able to write relatively simple test cases in Python (so every iteration of the fuzzer starts the interpreter, executes the test, tears it down...). But a neat concept IMHO. [1] https://bitbucket.org/ebadf/fuzzpy https://bitbucket.org/ebadf/fuzzpy
- galapago 10y ago
  We developed a fuzzer [0] using property-based testing in Haskell using QuickCheck (as expected!) [0] http://QuickFuzz.org/ http://QuickFuzz.org/
- nradov 10y ago
  I developed a fuzzer that generates random values based on ABNF rules, such as often appear in IETF RFCs. So it can be used for testing RFC implementations. It's written in Java but can be called from other tools. https://github.com/nradov/abnffuzzer https://github.com/nradov/abnffuzzer
- Twirrim 10y ago
  There is a good wrapper for afl, https://pypi.python.org/pypi/python-afl https://pypi.python.org/pypi/python-afl that allows you to relatively easily use afl with your python code. Just be sure to take advantage of looping or you'll be limited by the python runtime start-up overhead.
- numlocked 10y ago
  Seconded. I've used it for testing various functions in recommendation systems, and it's great at finding boundary conditions you didn't think of, or issues with floating point math. And, as it was the first time I used any testing tools like this, it was a fun emotional experience to have an antagonistic test suite.
theparanoid 10y ago
When will it be defunded, code search was fantastic until it was discontinued.
- hodgesrm 10y ago
  I don't understand the downvotes. This is unfortunately an issue with Google projects. code.google.com was another useful service that bit the dust.
  - DannyBee 10y ago
    The downvotes are because 1. it's irrelevant. This is an open source project, google code was a project hosting environment. 2. Google code "bit the dust" after being around 10+ years, and being replaced by something else .... (and had tools/etc and hand holding for the deprecation). So, 10 years from now, if someone has made "the github of open source fuzzing", and it's not this project, yeah, i guess it's fair that it might bite the dust. But honestly, much like Google code, if 10 years from now, someone else has replaced oss fuzz with something that has hundreds of thousands of open source projects, i'd call that success, not failure. 3. It's even less of a hard dependency.
    - tedunangst 10y ago
      github is a terrible replacement for code search.
  - dispose13432 10y ago
    Because unlike google code, this isn't going to be a hard dependency. When google code went down, existing sites (which used it as a CDN) broke and communities broke up. Here, you can do this on your own computer if you want. It's like complaining that SF took down Compile Farm (which is actually worse, they had proprietary Unixen on their system).
    - dgacmu 10y ago
      Definitely this. The hard work, from my experience, in hooking up to a fuzzer is writing the wrappers to hook in to the fuzzer, and creating a set of examples from which to begin fuzzing intelligently. Automated fuzzing is the icing on top of that hard work. It magnifies the utility of doing the work to fuzz by making it run continuously and by throwing more computational resources at it to find more bugs. But, unlike, say, coding to some closed API where you're up a creek if the service goes away, most of the hard work is portable, and you can still run it on your own cluster. AFL and libFuzzer are open source and quite awesome on their own.
theparanoid 10y ago
When will it be discontinued; code search was fantastic until it was shutdown.
d33 10y ago
Finally some serious community fuzzing effort! I always thought of fuzzing-project.org as of a joke, based on how little information there was. Hopefully this one will be better...
seanwilson 10y ago
> Recent security stories confirm that errors like buffer overflow and use-after-free can have serious, widespread consequences when they occur in critical open source software. This project is awesome and incredibly valuable but what alternatives are there to making the libraries it checks more secure besides rewriting them in another language? When languages exist where buffer overflows and use-after-free are essentially impossible it's a bit depressing that we have to rely on fuzzing unless fuzzing can find these kinds of bugs with high reliability?
- tanin 10y ago
  > what alternatives are there to making the libraries it checks more secure besides rewriting them in another language? I'm not sure what you mean by rewriting a library. For example, if a library is written in Ruby, fuzzing might not be suitable. you shouldn't rewrite it in C, so that you can fuzz it. (I thinkI misunderstand your point here. Please correct me.) If we want to discover vulnerabilities in Ruby, we should fuzz the Ruby VM directly. We shouldn't fuzz a Ruby library.

212 points by tanin 10y ago

newman314 10y ago

I'd like to see openssh added to the list of fuzzed projects.

wyldfire 10y ago
Adding a new project doesn't seem too tough. Guide: https://github.com/google/oss-fuzz/blob/master/docs/new_project_guide.md https://github.com/google/oss-fuzz/blob/master/docs/new_proj...
- newman314 10y ago
  I'm not against doing the work to create the pull (unless someone else wants to) but looking at https://www.openssh.com/report.html https://www.openssh.com/report.html, it's not clear to me if openssh[at]openssh.com or openssh-unix-dev[at]mindrot.org should be used for something like this.
  - greglindahl 10y ago
    Try firing it at yourself to start with, then ask the openssh community once you've got some useful data? Edit: Thanks for the downvote! Every time I consider sending in a pile of automated bug-reports to a project where I'm not already part of the community, I look at them myself first. If that's bad, I'm happy to be bad.

orf 10y ago

Awesome, it's found a few bugs with Sqlite3 already: https://bugs.chromium.org/p/oss-fuzz/issues/list?can=1&q=type%3DBug-Security%2CBug+-component%3AInfra+status%3AFixed%2CVerified+sqlite3&sort=-id&colspec=ID+Type+Component+Status+Library+Reported+Owner+Summary&cells=ids https://bugs.chromium.org/p/oss-fuzz/issues/list?can=1&q=typ...

deleted 10y ago
[deleted]

mtgx 10y ago

> In order for a project to be accepted to OSS-Fuzz, it needs to have a large user base and/or be critical to Global IT infrastructure Let's see - Firefox and/or the Tor browser? I imagine Google wouldn't be too happy about doing free security research for Firefox, but it seems to fit the bill quite well for the goals and mission of the Core Infrastructure Initiative organization.

orf 10y ago
Why wouldn't they be happy? Google's a diverse company, it's not like every department/team has an unwritten goal to push Chrome at the expense of everything else.
- ronjouch 10y ago
  > "it's not like every department/team has an unwritten goal to push Chrome at the expense of everything else." It often feels like it, yes. Could it be that you don't see it because you are already using Chrome? I use Edge on my Windows gaming machine (for minimal maintenance, I install drivers, Steam, and that's pretty much it) and configured Edge to clean everything including cookies on exit, like a permanent stateless Private Mode available for the occasional search. End result: many Google apps (Search, Maps, YouTube, and probably many others) constantly display huge "Get Chrome" banners, that end up not only painful for the user experience, but contain plain lies: "A more modern, faster browser" says one, whereas Edge/Firefox/Chrome are now objectively in the same ballpark. This constant soliciting across vast swaths of the web by one dominant actor controlling the end-to-end experience is abusive.
  - orf 10y ago
    I use Firefox and I agree with you, I don't like the "Use Chrome" banners that I see when using IE (i dont get them with ff). However, baring that, is there anything else they've done? IIRC they still pay Mozilla for making Google the default homepage
    - ronjouch 10y ago

nchammas 10y ago

On the topic of fuzz testing, Python has an excellent library for property-based testing called Hypothesis [0] [1]. I don't think it does guided testing like AFL or libFuzzer (which OSS-Fuzz uses), but it's very powerful nonetheless. [0] http://hypothesis.works/ http://hypothesis.works/ [1] https://github.com/HypothesisWorks/hypothesis-python https://github.com/HypothesisWorks/hypothesis-python

wyldfire 10y ago
I wrote a wrapper around CPython for fuzz testing with libFuzzer [1]. It's dog slow because it's designed for anyone to be able to write relatively simple test cases in Python (so every iteration of the fuzzer starts the interpreter, executes the test, tears it down...). But a neat concept IMHO. [1] https://bitbucket.org/ebadf/fuzzpy https://bitbucket.org/ebadf/fuzzpy
galapago 10y ago
We developed a fuzzer [0] using property-based testing in Haskell using QuickCheck (as expected!) [0] http://QuickFuzz.org/ http://QuickFuzz.org/
nradov 10y ago
I developed a fuzzer that generates random values based on ABNF rules, such as often appear in IETF RFCs. So it can be used for testing RFC implementations. It's written in Java but can be called from other tools. https://github.com/nradov/abnffuzzer https://github.com/nradov/abnffuzzer
Twirrim 10y ago
There is a good wrapper for afl, https://pypi.python.org/pypi/python-afl https://pypi.python.org/pypi/python-afl that allows you to relatively easily use afl with your python code. Just be sure to take advantage of looping or you'll be limited by the python runtime start-up overhead.
numlocked 10y ago
Seconded. I've used it for testing various functions in recommendation systems, and it's great at finding boundary conditions you didn't think of, or issues with floating point math. And, as it was the first time I used any testing tools like this, it was a fun emotional experience to have an antagonistic test suite.

theparanoid 10y ago

When will it be defunded, code search was fantastic until it was discontinued.

hodgesrm 10y ago
I don't understand the downvotes. This is unfortunately an issue with Google projects. code.google.com was another useful service that bit the dust.
- DannyBee 10y ago
  The downvotes are because 1. it's irrelevant. This is an open source project, google code was a project hosting environment. 2. Google code "bit the dust" after being around 10+ years, and being replaced by something else .... (and had tools/etc and hand holding for the deprecation). So, 10 years from now, if someone has made "the github of open source fuzzing", and it's not this project, yeah, i guess it's fair that it might bite the dust. But honestly, much like Google code, if 10 years from now, someone else has replaced oss fuzz with something that has hundreds of thousands of open source projects, i'd call that success, not failure. 3. It's even less of a hard dependency.
  - tedunangst 10y ago
    github is a terrible replacement for code search.
- dispose13432 10y ago
  Because unlike google code, this isn't going to be a hard dependency. When google code went down, existing sites (which used it as a CDN) broke and communities broke up. Here, you can do this on your own computer if you want. It's like complaining that SF took down Compile Farm (which is actually worse, they had proprietary Unixen on their system).
  - dgacmu 10y ago
    Definitely this. The hard work, from my experience, in hooking up to a fuzzer is writing the wrappers to hook in to the fuzzer, and creating a set of examples from which to begin fuzzing intelligently. Automated fuzzing is the icing on top of that hard work. It magnifies the utility of doing the work to fuzz by making it run continuously and by throwing more computational resources at it to find more bugs. But, unlike, say, coding to some closed API where you're up a creek if the service goes away, most of the hard work is portable, and you can still run it on your own cluster. AFL and libFuzzer are open source and quite awesome on their own.

theparanoid 10y ago

When will it be discontinued; code search was fantastic until it was shutdown.

d33 10y ago

Finally some serious community fuzzing effort! I always thought of fuzzing-project.org as of a joke, based on how little information there was. Hopefully this one will be better...

seanwilson 10y ago

> Recent security stories confirm that errors like buffer overflow and use-after-free can have serious, widespread consequences when they occur in critical open source software. This project is awesome and incredibly valuable but what alternatives are there to making the libraries it checks more secure besides rewriting them in another language? When languages exist where buffer overflows and use-after-free are essentially impossible it's a bit depressing that we have to rely on fuzzing unless fuzzing can find these kinds of bugs with high reliability?

tanin 10y ago
> what alternatives are there to making the libraries it checks more secure besides rewriting them in another language? I'm not sure what you mean by rewriting a library. For example, if a library is written in Ruby, fuzzing might not be suitable. you shouldn't rewrite it in C, so that you can fuzz it. (I thinkI misunderstand your point here. Please correct me.) If we want to discover vulnerabilities in Ruby, we should fuzz the Ruby VM directly. We shouldn't fuzz a Ruby library.