3 ms·

Also, the main issue with static analysis tools tends to be not false negatives, but false positives. That is, they churn out tons and tons of alerts that aren'

by m0zg 7y ago

Also, the main issue with static analysis tools tends to be not false negatives, but false positives. That is, they churn out tons and tons of alerts that aren't actually bugs. Some such systems alert so much that they aren't worth using.

Matthias247 7y ago
Yes, that's the main culprit with traditional static analysis. No one wants to review the results, because the amount of signal to noise is far too low. And also since it's an optional thing and not enforced by the compiler. I think this is where languages with stronger inbuilt analysis (e.g. Rust) win: The results are better, and since the analysis is always running as part of a compiler pass there are no huge jumps in indicated bugs at once (like what would happen if one would run Coverity on a legacy C++ codebase).
- taeric 7y ago
  This is less true of more advanced static analysis tools. I mean, ultimately we agree. Most people don't trust static analysis tools because they have had bad experiences with them. I just suspect most people should try them again. The state of the art is quite good in that space.
- pm215 7y ago
  Yes, I think the 'done by the compiler' part is really important for takeup. You can see a good example of a variant of this in the article: > diff time [ie in the standard code-review workflow] deployment saw a 70% fix rate, where a more traditional "offline" or "batch" deployment (where bug lists are presented to engineers, outside their workflow) saw a 0% fix rate That's the difference between "static analysis presented as part of the workflow a developer goes through anyway" and "static analysis presented after the fact". If you're in a position to enforce a code-review workflow that tools can hook into then "at code review time" works, but "at compile time" is better still since it shortens the feedback loop and ensures that everybody sees the issues while they're thinking about the code, even for smaller situations with more ad-hoc or nonexistent code review setups.
- apaprocki 7y ago
  From experience on large codebases, get to -Wall -Wextra “clean” in both the latest versions of GCC and Clang and then tools like Coverity will produce much more useful results. The signal it provides to me at that point is exactly what it is meant to provide: mostly improper error handling analysis and N-level deep branches that result in a poor result due to an error or bad decision in another file that a human would not associate with the current call chain or think to look at. To be fair, the tools work much better when you know you have complicated pieces that you spend a little time writing correct models for (e.g. custom assertion/error handling, runtime supplied vtables, custom allocators, etc.).
muglug 7y ago
It sounds (from the article) like they have some sort of heuristic for determining potential severity, and they're ok with more false-positives in areas where the potential damage from a false-negative is very high.
- m0zg 7y ago
  I might be biased, but I've never seen these systems work well in practice. Some 15-17 years ago Microsoft depoloyed a system called PreFix which would find genuine, hard to find bugs, but then bury them under a mountain of false positives, so few teams ran it, and even fewer looked at the results. I like what LLVM did in this area. Its SCA is not very comprehensive (so it can't be relied upon for deep analysis), but when it does find something it's usually a legit issue. But the balance of deep analysis and low false positives remains elusive. I'd be really stunned if FB really achieved a breakthrough in this area. I do want to be wrong about this.
  - j88439h84 7y ago
    In python, Pylint and mypy find real bugs all the time, plenty of false positives but still very usable.
    - clarus 7y ago
      Pylint and mypy are about syntax and type-checking. While I agree these kind of tools work well, if think that by static analysis people usually imply something which goes farther than that. For example, the languages Facebook is citing (C++, Java, ...) already include type-checking in the compiler.
      - j88439h84 7y ago
        Pylint catches lots of semantic bugs, far from just syntax.
  - d7gibo 7y ago
    The mountain of false positives isn't an issue if you run the static analysis tools from the start of the project's development.

by m0zg 7y ago

Matthias247 7y ago

Yes, that's the main culprit with traditional static analysis. No one wants to review the results, because the amount of signal to noise is far too low. And also since it's an optional thing and not enforced by the compiler. I think this is where languages with stronger inbuilt analysis (e.g. Rust) win: The results are better, and since the analysis is always running as part of a compiler pass there are no huge jumps in indicated bugs at once (like what would happen if one would run Coverity on a legacy C++ codebase).

taeric 7y ago
This is less true of more advanced static analysis tools. I mean, ultimately we agree. Most people don't trust static analysis tools because they have had bad experiences with them. I just suspect most people should try them again. The state of the art is quite good in that space.
pm215 7y ago
Yes, I think the 'done by the compiler' part is really important for takeup. You can see a good example of a variant of this in the article: > diff time [ie in the standard code-review workflow] deployment saw a 70% fix rate, where a more traditional "offline" or "batch" deployment (where bug lists are presented to engineers, outside their workflow) saw a 0% fix rate That's the difference between "static analysis presented as part of the workflow a developer goes through anyway" and "static analysis presented after the fact". If you're in a position to enforce a code-review workflow that tools can hook into then "at code review time" works, but "at compile time" is better still since it shortens the feedback loop and ensures that everybody sees the issues while they're thinking about the code, even for smaller situations with more ad-hoc or nonexistent code review setups.
apaprocki 7y ago
From experience on large codebases, get to -Wall -Wextra “clean” in both the latest versions of GCC and Clang and then tools like Coverity will produce much more useful results. The signal it provides to me at that point is exactly what it is meant to provide: mostly improper error handling analysis and N-level deep branches that result in a poor result due to an error or bad decision in another file that a human would not associate with the current call chain or think to look at. To be fair, the tools work much better when you know you have complicated pieces that you spend a little time writing correct models for (e.g. custom assertion/error handling, runtime supplied vtables, custom allocators, etc.).

muglug 7y ago

It sounds (from the article) like they have some sort of heuristic for determining potential severity, and they're ok with more false-positives in areas where the potential damage from a false-negative is very high.

m0zg 7y ago
I might be biased, but I've never seen these systems work well in practice. Some 15-17 years ago Microsoft depoloyed a system called PreFix which would find genuine, hard to find bugs, but then bury them under a mountain of false positives, so few teams ran it, and even fewer looked at the results. I like what LLVM did in this area. Its SCA is not very comprehensive (so it can't be relied upon for deep analysis), but when it does find something it's usually a legit issue. But the balance of deep analysis and low false positives remains elusive. I'd be really stunned if FB really achieved a breakthrough in this area. I do want to be wrong about this.
- j88439h84 7y ago
  In python, Pylint and mypy find real bugs all the time, plenty of false positives but still very usable.
  - clarus 7y ago
    Pylint and mypy are about syntax and type-checking. While I agree these kind of tools work well, if think that by static analysis people usually imply something which goes farther than that. For example, the languages Facebook is citing (C++, Java, ...) already include type-checking in the compiler.
    - j88439h84 7y ago
      Pylint catches lots of semantic bugs, far from just syntax.
- d7gibo 7y ago
  The mountain of false positives isn't an issue if you run the static analysis tools from the start of the project's development.