4 ms·

OSS-Fuzz: Five months later, and rewarding projects

132 points by tanin 9y ago

the_why_of_y 9y ago
Interesting that they managed to find 8 security bugs in SQLite, which is renowned for having a test suite with ~100% code coverage - yet more evidence for the Dijkstra dictum that testing cannot prove the absence of bugs. https://sqlite.org/testing.html https://sqlite.org/testing.html
- obstinate 9y ago
  100% test coverage is an ambiguous term. Is it line coverage, branch coverage, and do the branches include arithmetic nonlinearity branches and exception flags? Are all possibly syscall error states tested? One hundred percent line coverage is admirable, but it's just a start! (Edit: I see that they had far more than just 100% line coverage, but as you can see even this is not enough to find all cases.)
  - deleted 9y ago
    [deleted]
  - masklinn 9y ago
    > 100% test coverage is an ambiguous term. Is it line coverage, branch coverage The linked document answers that both repeatedly and at length: > The SQLite core, including the unix VFS, has 100% branch test coverage under TH3 in its default configuration as measured by gcov. [follow half a dozen paragraph explaining what they mean precisely by branch coverage]
    - obstinate 9y ago
      And my edit denotes that I saw that and acknowledged it! :)
      - Dor1s 9y ago
        Even with 100% coverage, there might be the following code (just a one of numerous examples): char buffer[15]; ..... memcpy(buffer, source, strlen(source)); Depending on the `source`, the bug can be triggered or not, but the coverage is the same for different `source` values.
- kibwen 9y ago
  Rather than their code coverage, I'd say SQLite is known for having literally millions of test cases in their test suite. The number of lines of code in their test suite is some absurd multiple of the number of lines of code in SQLite itself.
- SEJeff 9y ago
  That is quite fascinating. I do somewhat wonder if some of that can be lessened by using more "property based testing"[1]. Property based testing can be thought of writing tests that describe the functionality you are trying to cover. It then interpolates (not too unlike fuzzing actually) to come up with every way that might be non-compliant and then checks to ensure your hypothesis holds up. Amusingly enough, the most active python property based testing library is named hypothesis[2], which describes quite well how you write unit tests using it. [1] http://blog.jessitron.com/2013/04/property-based-testing-what-is-it.html http://blog.jessitron.com/2013/04/property-based-testing-wha... [2] https://hypothesis.readthedocs.io/en/latest/ https://hypothesis.readthedocs.io/en/latest/
  - nickpsecurity 9y ago
    Here's one of those for C: https://github.com/silentbicycle/theft https://github.com/silentbicycle/theft Slides on automated tool below. They use "model-based testing" which is another phrase to get interesting search results. https://www.slideshare.net/dganesan11/ganesan-automated-testcasegenerationtutorial2012 https://www.slideshare.net/dganesan11/ganesan-automated-test... Also combinational testing which showed promise at catching in high 90's of defect percentage with test case minimization as a bonus. See summary and slides. http://csrc.nist.gov/groups/SNS/acts/index.html http://csrc.nist.gov/groups/SNS/acts/index.html
- f- 9y ago
  I had pretty good results fuzzing SQLite back in the day: https://lcamtuf.blogspot.com/2015/04/finding-bugs-in-sqlite-easy-way.html https://lcamtuf.blogspot.com/2015/04/finding-bugs-in-sqlite-... I think they eventually incorporated AFL into their continuous testing and squashed several dozen bugs. OSS Fuzz scales it up, but yup - the bottom line is that you might think you have 100% test coverage, but you really still need to fuzz =)
- gsnedders 9y ago
  It's probably worth pointing out that some categories of bugs are almost invariably caused by the absence of a branch, hence 100% branch coverage doesn't show up such failings. (e.g., null pointer dereferences are typically a failure to branch on a null pointer, malloc failures being unhandled are almost always a failure to branch on the return value of malloc, etc.)

132 points by tanin 9y ago

Interesting that they managed to find 8 security bugs in SQLite, which is renowned for having a test suite with ~100% code coverage - yet more evidence for the Dijkstra dictum that testing cannot prove the absence of bugs. https://sqlite.org/testing.html https://sqlite.org/testing.html

obstinate 9y ago
100% test coverage is an ambiguous term. Is it line coverage, branch coverage, and do the branches include arithmetic nonlinearity branches and exception flags? Are all possibly syscall error states tested? One hundred percent line coverage is admirable, but it's just a start! (Edit: I see that they had far more than just 100% line coverage, but as you can see even this is not enough to find all cases.)
- deleted 9y ago
  [deleted]
- masklinn 9y ago
  > 100% test coverage is an ambiguous term. Is it line coverage, branch coverage The linked document answers that both repeatedly and at length: > The SQLite core, including the unix VFS, has 100% branch test coverage under TH3 in its default configuration as measured by gcov. [follow half a dozen paragraph explaining what they mean precisely by branch coverage]
  - obstinate 9y ago
    And my edit denotes that I saw that and acknowledged it! :)
    - Dor1s 9y ago
      Even with 100% coverage, there might be the following code (just a one of numerous examples): char buffer[15]; ..... memcpy(buffer, source, strlen(source)); Depending on the `source`, the bug can be triggered or not, but the coverage is the same for different `source` values.
kibwen 9y ago
Rather than their code coverage, I'd say SQLite is known for having literally millions of test cases in their test suite. The number of lines of code in their test suite is some absurd multiple of the number of lines of code in SQLite itself.
SEJeff 9y ago
That is quite fascinating. I do somewhat wonder if some of that can be lessened by using more "property based testing"[1]. Property based testing can be thought of writing tests that describe the functionality you are trying to cover. It then interpolates (not too unlike fuzzing actually) to come up with every way that might be non-compliant and then checks to ensure your hypothesis holds up. Amusingly enough, the most active python property based testing library is named hypothesis[2], which describes quite well how you write unit tests using it. [1] http://blog.jessitron.com/2013/04/property-based-testing-what-is-it.html http://blog.jessitron.com/2013/04/property-based-testing-wha... [2] https://hypothesis.readthedocs.io/en/latest/ https://hypothesis.readthedocs.io/en/latest/
- nickpsecurity 9y ago
  Here's one of those for C: https://github.com/silentbicycle/theft https://github.com/silentbicycle/theft Slides on automated tool below. They use "model-based testing" which is another phrase to get interesting search results. https://www.slideshare.net/dganesan11/ganesan-automated-testcasegenerationtutorial2012 https://www.slideshare.net/dganesan11/ganesan-automated-test... Also combinational testing which showed promise at catching in high 90's of defect percentage with test case minimization as a bonus. See summary and slides. http://csrc.nist.gov/groups/SNS/acts/index.html http://csrc.nist.gov/groups/SNS/acts/index.html
f- 9y ago
I had pretty good results fuzzing SQLite back in the day: https://lcamtuf.blogspot.com/2015/04/finding-bugs-in-sqlite-easy-way.html https://lcamtuf.blogspot.com/2015/04/finding-bugs-in-sqlite-... I think they eventually incorporated AFL into their continuous testing and squashed several dozen bugs. OSS Fuzz scales it up, but yup - the bottom line is that you might think you have 100% test coverage, but you really still need to fuzz =)
gsnedders 9y ago
It's probably worth pointing out that some categories of bugs are almost invariably caused by the absence of a branch, hence 100% branch coverage doesn't show up such failings. (e.g., null pointer dereferences are typically a failure to branch on a null pointer, malloc failures being unhandled are almost always a failure to branch on the return value of malloc, etc.)