6 ms·

The section on static analysis is somewhat damning: Static analysis has not proven to be especially helpful in finding bugs in SQLite. Static analysis has foun

by panic 10y ago

The section on static analysis is somewhat damning: Static analysis has not proven to be especially helpful in finding bugs in SQLite. Static analysis has found a few bugs in SQLite, but those are the exceptions. More bugs have been introduced into SQLite while trying to get it to compile without warnings than have been found by static analysis.

masklinn 10y ago
That's only "damning" if like SQLite your project has several orders of magnitude more test code than tested code, which I'm reasonably certain is not the case.
- fit2rule 10y ago
  Static analysis is rarely helpful after the fact. Its usefulness is directly related to its entry into the project timeline. Just like "handle all warnings, always" is a generally good mantra, having static analysis is supposed to keep you ahead of the curve. If you don't do it early, and often, you're not doing it properly. Also, having these tools, but no coding rules by code contributors, enforced with real code-reviews and reviewer(s), will also deliver 'not worth it' results. Its a process, not a chisel.
  - mikeash 10y ago
    That doesn't match what I've seen. When Apple added support for the clang static analyzer to Xcode, a ton of Objective-C programmers started running it on their code bases, and among those I know, it found a ton of bugs that had gone dormant for years.
    - fit2rule 10y ago
      Sure, of course it'll find bugs after the fact. But the value to the project in doing static analysis is in not adding bugs in the first place, because you caught them with static analysis as a standard part of your development process. Who knows what those unknown bugs cost those developers over the years ..
      - mikeash 10y ago
        Are you implying that there is no value in finding bugs after the fact? Sure, it's better to catch them up front. But catching them later is still extremely valuable.
        fit2rule 10y ago
        Catching them before they're deployed anywhere is the point. Deploying quality software from 1.0 is a different order of value than 'experience improved for the 1.1 user because: bugs fixed'. Both circumstances have value: I meant only to state that static analysis should be up-front, to get the absolute most value out of it, because it will massively improve your software quality to have it there.
- deleted 10y ago
  [deleted]
TickleSteve 10y ago
SQLite is already a mature codebase. You would hope (expect?) that most issues that a Static Analyser would pick up would be purely stylistic ones in this case. This speaks more to the maturity and robustness of the SQLite codebase than it does of static analysis as a type of tool. Static Analysis is very useful in the development phase of a project where you can treat them effectively as compiler warnings if your build system is setup that way. ...and that isn't to say that its not worth still using SA on mature codebases, just because a codebase is mature does not indicate its quality.
- CodexArcanum 10y ago
  I've found static analysis (and related concepts e.g. type systems) to be extremely useful for safely refactoring code. Much more so than tests, which tend to just break and require rewriting. But as you say, once the code base is mature that kind of large scale code shuffling is less likely. (Not to speak poorly of testing, I'm a big fan of that too. Why not use every tool at one's disposal to ensure quality?)
  - TickleSteve 10y ago
    true, though there is a difference.... SA will capture issues related to the implementation but in itself it has no knowledge of application-specific behaviour. Tests on the other hand do have application-specific knowledge and this is where their value lies.
nickpsecurity 10y ago
TickleSteve made one good point. I'll add to that one two things: the analyzer; the point of them. The analyzer itself sucks compared to a number of others. Most commercial work tries to use several given each invariably misses things. The other point is that many use static analysis to reduce the testing burden by catching problems early. As in, they might not have needed all those tests on undefined behavior and such if they coded in a way that passed on static analysis.
- throwawaysocks 10y ago
  Exactly! The point of static analysis is avoiding the need for "millions and millions" of tests.
junke 10y ago
Related talk about Tis-interpreter and SQLite: http://gdr-gpl.cnrs.fr/sites/default/files/documentsGPL/JourneesNationales/GPL2016/sqlite-in-tis-interpreter.pdf http://gdr-gpl.cnrs.fr/sites/default/files/documentsGPL/Jour... Note also: > We want verification to be about proving absence of bugs, not about finding bugs
pcwalton 10y ago
Well, if your static analysis consists of compiler warnings and a few clang passes while your dynamic analysis is avionics/military grade, of course the dynamic analysis is going to be more effective. Would you expect anything else? A more interesting comparison would be between the most powerful/restrictive static analyses vs. the most powerful dynamic analyses.
adekok 10y ago
Once a code base gets mature (i.e. tested mature), static analysis becomes less useful. In my experience, though, it's worth still running it, just to catch issues with new code. Every release of FreeRADIUS (http://freeradius.org http://freeradius.org) gets run through three different static analysis tools. They each find slightly different things. At this point, the bulk of issues they find are false positives. Addressing those doesn't create bugs, and I'm not sure why it would.
- psaccounts 10y ago
  Could you share which 3 static analysis tools you use? We found Coverity to be prohibitively expensive; but unfortunately there isn't any real alternative!
- adrianratnapala 10y ago
  Addressing those doesn't create bugs, and I'm not sure why it would. Anything that changes behaviour could create bugs. Here is an example signed bar = something_possibly_negative(); if(some_case) { bar = calc_returning_unsigned_where_MSB_is_an_error_flag(); if(bar < 0) return ERROR; } // now bar is valid, whether negative or positive. This is a correct - if dodgy - way to test the MSB. But now if you get a warning about the signed integer not having the full range of the unsigned integer you might "fix" it by changing the first line to long long bar = something_possibly_negative(); Now the signed bar covers the full range of the unsigned calculation - including the garbage in the error case. For more in this vein see: http://ithare.com/best-practices-vs-witch-hunts/ http://ithare.com/best-practices-vs-witch-hunts/
thesz 10y ago
http://lcamtuf.blogspot.ru/2015/04/finding-bugs-in-sqlite-easy-way.html http://lcamtuf.blogspot.ru/2015/04/finding-bugs-in-sqlite-ea... Fuzzing found real actionable bugs in SQLite. As I consider fuzzing a variant of static checking (it executes a subset of symbolic evaluation space), I consider the result above as a resounding success of static methods. Also please look at the date when the phrase was added: https://web.archive.org/web/20090205181900/http://sqlite.org/testing.html https://web.archive.org/web/20090205181900/http://sqlite.org... - it's 2009, the very first version of the page! And they didn't change it a bit since. Basically you base your damnation on an outdated information.
- emeraldd 10y ago
  How does fuzzing fit into static analysis? By my understanding, Static Analysis is defined as examining the code without executing it and fuzzing would inherently require executing the code ...
  - thesz 10y ago
    Executing code with single input value is a subset of (symbolic) execution of the same code with all values. You cannot "examine the code" without executing it, partially or in full, with some input or intermediate values or (symbolically) with all of them. I am sure I sound contrary to "common definitions" - so be it. I also think that dynamic typing is a subset of static typing, more or less for the same reason.
    - emeraldd 10y ago
      > You cannot "examine the code" without executing it, partially or in full, with some input or intermediate values or (symbolically) with all of them. It is entirely possible to break a stream of text down into symbols and examine the relationship of those symbols without evaluating (i.e. executing) them. It is not really necessary or possible to know their meaning/definition at all, only their "shape" or the "part of speech" that they represent. In the simplest form, static analyzers are a set of heuristic pattern matchers that a stream of text is sent through. It might get fancy and build an AST first, but that still does not constitute execution or evaluation in the technical sense.

by panic 10y ago

masklinn 10y ago

That's only "damning" if like SQLite your project has several orders of magnitude more test code than tested code, which I'm reasonably certain is not the case.

fit2rule 10y ago
Static analysis is rarely helpful after the fact. Its usefulness is directly related to its entry into the project timeline. Just like "handle all warnings, always" is a generally good mantra, having static analysis is supposed to keep you ahead of the curve. If you don't do it early, and often, you're not doing it properly. Also, having these tools, but no coding rules by code contributors, enforced with real code-reviews and reviewer(s), will also deliver 'not worth it' results. Its a process, not a chisel.
- mikeash 10y ago
  That doesn't match what I've seen. When Apple added support for the clang static analyzer to Xcode, a ton of Objective-C programmers started running it on their code bases, and among those I know, it found a ton of bugs that had gone dormant for years.
  - fit2rule 10y ago
    Sure, of course it'll find bugs after the fact. But the value to the project in doing static analysis is in not adding bugs in the first place, because you caught them with static analysis as a standard part of your development process. Who knows what those unknown bugs cost those developers over the years ..
    - mikeash 10y ago
      Are you implying that there is no value in finding bugs after the fact? Sure, it's better to catch them up front. But catching them later is still extremely valuable.
      - fit2rule 10y ago
        Catching them before they're deployed anywhere is the point. Deploying quality software from 1.0 is a different order of value than 'experience improved for the 1.1 user because: bugs fixed'. Both circumstances have value: I meant only to state that static analysis should be up-front, to get the absolute most value out of it, because it will massively improve your software quality to have it there.
deleted 10y ago
[deleted]

TickleSteve 10y ago

SQLite is already a mature codebase. You would hope (expect?) that most issues that a Static Analyser would pick up would be purely stylistic ones in this case. This speaks more to the maturity and robustness of the SQLite codebase than it does of static analysis as a type of tool. Static Analysis is very useful in the development phase of a project where you can treat them effectively as compiler warnings if your build system is setup that way. ...and that isn't to say that its not worth still using SA on mature codebases, just because a codebase is mature does not indicate its quality.

CodexArcanum 10y ago
I've found static analysis (and related concepts e.g. type systems) to be extremely useful for safely refactoring code. Much more so than tests, which tend to just break and require rewriting. But as you say, once the code base is mature that kind of large scale code shuffling is less likely. (Not to speak poorly of testing, I'm a big fan of that too. Why not use every tool at one's disposal to ensure quality?)
- TickleSteve 10y ago
  true, though there is a difference.... SA will capture issues related to the implementation but in itself it has no knowledge of application-specific behaviour. Tests on the other hand do have application-specific knowledge and this is where their value lies.

nickpsecurity 10y ago

TickleSteve made one good point. I'll add to that one two things: the analyzer; the point of them. The analyzer itself sucks compared to a number of others. Most commercial work tries to use several given each invariably misses things. The other point is that many use static analysis to reduce the testing burden by catching problems early. As in, they might not have needed all those tests on undefined behavior and such if they coded in a way that passed on static analysis.

throwawaysocks 10y ago
Exactly! The point of static analysis is avoiding the need for "millions and millions" of tests.

junke 10y ago

Related talk about Tis-interpreter and SQLite: http://gdr-gpl.cnrs.fr/sites/default/files/documentsGPL/JourneesNationales/GPL2016/sqlite-in-tis-interpreter.pdf http://gdr-gpl.cnrs.fr/sites/default/files/documentsGPL/Jour... Note also: > We want verification to be about proving absence of bugs, not about finding bugs

pcwalton 10y ago

Well, if your static analysis consists of compiler warnings and a few clang passes while your dynamic analysis is avionics/military grade, of course the dynamic analysis is going to be more effective. Would you expect anything else? A more interesting comparison would be between the most powerful/restrictive static analyses vs. the most powerful dynamic analyses.

adekok 10y ago

Once a code base gets mature (i.e. tested mature), static analysis becomes less useful. In my experience, though, it's worth still running it, just to catch issues with new code. Every release of FreeRADIUS (http://freeradius.org http://freeradius.org) gets run through three different static analysis tools. They each find slightly different things. At this point, the bulk of issues they find are false positives. Addressing those doesn't create bugs, and I'm not sure why it would.

psaccounts 10y ago
Could you share which 3 static analysis tools you use? We found Coverity to be prohibitively expensive; but unfortunately there isn't any real alternative!
adrianratnapala 10y ago
Addressing those doesn't create bugs, and I'm not sure why it would. Anything that changes behaviour could create bugs. Here is an example signed bar = something_possibly_negative(); if(some_case) { bar = calc_returning_unsigned_where_MSB_is_an_error_flag(); if(bar < 0) return ERROR; } // now bar is valid, whether negative or positive. This is a correct - if dodgy - way to test the MSB. But now if you get a warning about the signed integer not having the full range of the unsigned integer you might "fix" it by changing the first line to long long bar = something_possibly_negative(); Now the signed bar covers the full range of the unsigned calculation - including the garbage in the error case. For more in this vein see: http://ithare.com/best-practices-vs-witch-hunts/ http://ithare.com/best-practices-vs-witch-hunts/

thesz 10y ago

http://lcamtuf.blogspot.ru/2015/04/finding-bugs-in-sqlite-easy-way.html http://lcamtuf.blogspot.ru/2015/04/finding-bugs-in-sqlite-ea... Fuzzing found real actionable bugs in SQLite. As I consider fuzzing a variant of static checking (it executes a subset of symbolic evaluation space), I consider the result above as a resounding success of static methods. Also please look at the date when the phrase was added: https://web.archive.org/web/20090205181900/http://sqlite.org/testing.html https://web.archive.org/web/20090205181900/http://sqlite.org... - it's 2009, the very first version of the page! And they didn't change it a bit since. Basically you base your damnation on an outdated information.

emeraldd 10y ago
How does fuzzing fit into static analysis? By my understanding, Static Analysis is defined as examining the code without executing it and fuzzing would inherently require executing the code ...
- thesz 10y ago
  Executing code with single input value is a subset of (symbolic) execution of the same code with all values. You cannot "examine the code" without executing it, partially or in full, with some input or intermediate values or (symbolically) with all of them. I am sure I sound contrary to "common definitions" - so be it. I also think that dynamic typing is a subset of static typing, more or less for the same reason.
  - emeraldd 10y ago
    > You cannot "examine the code" without executing it, partially or in full, with some input or intermediate values or (symbolically) with all of them. It is entirely possible to break a stream of text down into symbols and examine the relationship of those symbols without evaluating (i.e. executing) them. It is not really necessary or possible to know their meaning/definition at all, only their "shape" or the "part of speech" that they represent. In the simplest form, static analyzers are a set of heuristic pattern matchers that a stream of text is sent through. It might get fancy and build an AST first, but that still does not constitute execution or evaluation in the technical sense.