6 ms·

So I've dealt with dozens of Fortune-100 companies implementing and using static code analysis tools. They can and will help but in general I feel that these to

So I've dealt with dozens of Fortune-100 companies implementing and using static code analysis tools. They can and will help but in general I feel that these tools are not much more than the code-equivalent of the syntax- and grammar- checker in your word processing software. I've been doing manual code reviews for a living now (mostly security related) for roughly 3 years now and while I get assisted from time to time by code analysis tools I still find heaps of bugs not caught by any of the tools mentioned by Carmack. The biggest issue for a development shop is to properly integrate these tools and to not overwhelm developers with too much false positives. I've had cases where a developer got a 1500 page PDF spit out by one of these static analysis tools. After spending two weeks going through everything the developer ended up with 50 pages of actual bugs; the rest were describing false positives. Then I got on-site and I still logged dozens and dozens of security-related bugs that the static analysis tools failed to find. Edit: also consider that one even needs a SAT solver to even do proper C-style preprocessor dependency checking. A lot of these code analysis tools are being run on debug builds only and then there when the release build is being made these tools are not being run meaning they fail to catch a lot of issues. It's insanely hard to write proper code analysis tools and static source code analysis tools which do not integrate with the compilation process I wouldn't trust at all. Nowadays with clang there are very nice possibilities for someone to write your own simple checks and integrate them into the build process. But even clang doesn't expose everything about the preprocessor that you might want to have from a static code analysis perspective.

btilly 14y ago
Beware the false false positive! A couple of years ago I ran across an article from Coverity about the challenges that they had getting people to use their tool. One of the most interesting was that the tool would find real bugs, but the developer wouldn't understand that the bug actually was real, and would say, "This tool is crap." Then they would not get a sale. They had this problem particularly strongly with race conditions. There are a number of checks that they took out because, even though they were finding real bugs, developers were convincing themselves that the bugs were not real even though they really were. It really does not help that the developers who are asked to evaluate are likely to be the same people who made the mistakes in the first place, so all kinds of defensive behavior are to be expected.
- santaragolabs 14y ago
  Ha, you're completely right. I've been brought in on several occasions to help my clients decide on which static code analysis tool to use after an evaluation period and I've encountered these situations too. But it helps if it's an outside consultant evaluating the product and not the developers themselves. Through the years I've learned how to talk with developers about their bugs and how to help them get better in preventing it. It helps that I've been a developer for years too so I know where they're coming from and how they sometimes tend to get all defensive.
- eru 14y ago
  You are probably taking about this article (http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later/fulltext http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lin...). A very good read, indeed.
  - btilly 14y ago
    I was, and it was. Thanks for finding it again.
ScottBurson 14y ago
I would love to see examples of classes of bugs not caught by the tools. Do you recall any offhand? (Full disclosure: I work for HP Fortify.)
- santaragolabs 14y ago
  I'd prefer not to go too much in specifics due to NDA's and all that; but one simple example that I've encountered: I've seen simple cases where a function returned a signed integer value modulo x. Which of course means that this function can only return a value within 0..x-1. This function was called in another function which then did some allocations based on it and the static analyzer complained about integer overflows which were obviously not possible in this case. Dependency chains in preprocessors is a big pain in general too. #ifdef A #define B #endif #ifdef C #ifdef B blabla #endif #endif This will yield a dependency chain for B of (A,C) -> B resulting in blabla being compiled. With complex code bases this already becomes quickly unmanageable. As far as I know no code analyzer is able to handle this. And I haven't even started talking about supporting #if constructs (ex: #if X > 2). You would need to implement SAT solvers for that too. Some more info on this at the clang mailing list here: http://lists.cs.uiuc.edu/pipermail/cfe-dev/2012-July/022720.html http://lists.cs.uiuc.edu/pipermail/cfe-dev/2012-July/022720..... These are just other reasons why seperating the preprocessor from the rest of the language was such a bad idea; you lose all the static code analysis stuff you can do on C/C++ code because of it and preprocessor directives control a lot of things in modern and big code bases.
  - Figs 14y ago
    > I've seen simple cases where a function returned a signed integer value modulo x. Which of course means that this function can only return a value within 0..x-1. Actually, signed modulo can return negative numbers... See, for example, this StackOverflow question: http://stackoverflow.com/questions/7594508/modulo-operator-with-negative-values http://stackoverflow.com/questions/7594508/modulo-operator-w...
    - santaragolabs 14y ago

btilly 14y ago

Beware the false false positive! A couple of years ago I ran across an article from Coverity about the challenges that they had getting people to use their tool. One of the most interesting was that the tool would find real bugs, but the developer wouldn't understand that the bug actually was real, and would say, "This tool is crap." Then they would not get a sale. They had this problem particularly strongly with race conditions. There are a number of checks that they took out because, even though they were finding real bugs, developers were convincing themselves that the bugs were not real even though they really were. It really does not help that the developers who are asked to evaluate are likely to be the same people who made the mistakes in the first place, so all kinds of defensive behavior are to be expected.

santaragolabs 14y ago
Ha, you're completely right. I've been brought in on several occasions to help my clients decide on which static code analysis tool to use after an evaluation period and I've encountered these situations too. But it helps if it's an outside consultant evaluating the product and not the developers themselves. Through the years I've learned how to talk with developers about their bugs and how to help them get better in preventing it. It helps that I've been a developer for years too so I know where they're coming from and how they sometimes tend to get all defensive.
eru 14y ago
You are probably taking about this article (http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later/fulltext http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lin...). A very good read, indeed.
- btilly 14y ago
  I was, and it was. Thanks for finding it again.

ScottBurson 14y ago

I would love to see examples of classes of bugs not caught by the tools. Do you recall any offhand? (Full disclosure: I work for HP Fortify.)

santaragolabs 14y ago
I'd prefer not to go too much in specifics due to NDA's and all that; but one simple example that I've encountered: I've seen simple cases where a function returned a signed integer value modulo x. Which of course means that this function can only return a value within 0..x-1. This function was called in another function which then did some allocations based on it and the static analyzer complained about integer overflows which were obviously not possible in this case. Dependency chains in preprocessors is a big pain in general too. #ifdef A #define B #endif #ifdef C #ifdef B blabla #endif #endif This will yield a dependency chain for B of (A,C) -> B resulting in blabla being compiled. With complex code bases this already becomes quickly unmanageable. As far as I know no code analyzer is able to handle this. And I haven't even started talking about supporting #if constructs (ex: #if X > 2). You would need to implement SAT solvers for that too. Some more info on this at the clang mailing list here: http://lists.cs.uiuc.edu/pipermail/cfe-dev/2012-July/022720.html http://lists.cs.uiuc.edu/pipermail/cfe-dev/2012-July/022720..... These are just other reasons why seperating the preprocessor from the rest of the language was such a bad idea; you lose all the static code analysis stuff you can do on C/C++ code because of it and preprocessor directives control a lot of things in modern and big code bases.
- Figs 14y ago
  > I've seen simple cases where a function returned a signed integer value modulo x. Which of course means that this function can only return a value within 0..x-1. Actually, signed modulo can return negative numbers... See, for example, this StackOverflow question: http://stackoverflow.com/questions/7594508/modulo-operator-with-negative-values http://stackoverflow.com/questions/7594508/modulo-operator-w...
  - santaragolabs 14y ago