4 ms·

It sounds (from the article) like they have some sort of heuristic for determining potential severity, and they're ok with more false-positives in areas where t

by muglug 7y ago

It sounds (from the article) like they have some sort of heuristic for determining potential severity, and they're ok with more false-positives in areas where the potential damage from a false-negative is very high.

m0zg 7y ago
I might be biased, but I've never seen these systems work well in practice. Some 15-17 years ago Microsoft depoloyed a system called PreFix which would find genuine, hard to find bugs, but then bury them under a mountain of false positives, so few teams ran it, and even fewer looked at the results. I like what LLVM did in this area. Its SCA is not very comprehensive (so it can't be relied upon for deep analysis), but when it does find something it's usually a legit issue. But the balance of deep analysis and low false positives remains elusive. I'd be really stunned if FB really achieved a breakthrough in this area. I do want to be wrong about this.
- j88439h84 7y ago
  In python, Pylint and mypy find real bugs all the time, plenty of false positives but still very usable.
  - clarus 7y ago
    Pylint and mypy are about syntax and type-checking. While I agree these kind of tools work well, if think that by static analysis people usually imply something which goes farther than that. For example, the languages Facebook is citing (C++, Java, ...) already include type-checking in the compiler.
    - j88439h84 7y ago
      Pylint catches lots of semantic bugs, far from just syntax.
- d7gibo 7y ago
  The mountain of false positives isn't an issue if you run the static analysis tools from the start of the project's development.
  - seanmcdirmid 7y ago
    It is as your code base becomes littered with annotations to suppress each historic false positive.
- nickpsecurity 7y ago
  You kind of explained it yourself saying 15-17 years ago. Lots of things can improve in that time-frame. PreFix wasn't even that focused on reducing false positives IIRC. Some today are focused on keeping false positives down. A few benchmarked are here in this also-biased article: https://runtimeverification.com/match/1.0-SNAPSHOT/docs/benchmark/ https://runtimeverification.com/match/1.0-SNAPSHOT/docs/benc... I bring them up because they made the open-source K Framework and a C semantics. Another commenter says PVS-Studio is pretty good. Since Synopsis owns Coverity now, I'd recommend RV-Match (little to no false positives) followed by PVS-Studio.
  - m0zg 7y ago
    Don't know about Coverity, but a customer did a Fortify scan on our code last year (as a part of acceptance), and it didn't find any issues. Which I find really hard to believe that there aren't any issues, seeing that the codebase is 200KLOC of pretty gnarly C++ with probably another 500KLOC in source-level C++ deps (we compile from source where we can). Either the team was comprised entirely of gods of C++ programming, or it doesn't really "fortify" all that much. I'll let you decide which one is more probable.

by muglug 7y ago

m0zg 7y ago

I might be biased, but I've never seen these systems work well in practice. Some 15-17 years ago Microsoft depoloyed a system called PreFix which would find genuine, hard to find bugs, but then bury them under a mountain of false positives, so few teams ran it, and even fewer looked at the results. I like what LLVM did in this area. Its SCA is not very comprehensive (so it can't be relied upon for deep analysis), but when it does find something it's usually a legit issue. But the balance of deep analysis and low false positives remains elusive. I'd be really stunned if FB really achieved a breakthrough in this area. I do want to be wrong about this.

j88439h84 7y ago
In python, Pylint and mypy find real bugs all the time, plenty of false positives but still very usable.
- clarus 7y ago
  Pylint and mypy are about syntax and type-checking. While I agree these kind of tools work well, if think that by static analysis people usually imply something which goes farther than that. For example, the languages Facebook is citing (C++, Java, ...) already include type-checking in the compiler.
  - j88439h84 7y ago
    Pylint catches lots of semantic bugs, far from just syntax.
d7gibo 7y ago
The mountain of false positives isn't an issue if you run the static analysis tools from the start of the project's development.
- seanmcdirmid 7y ago
  It is as your code base becomes littered with annotations to suppress each historic false positive.
nickpsecurity 7y ago
You kind of explained it yourself saying 15-17 years ago. Lots of things can improve in that time-frame. PreFix wasn't even that focused on reducing false positives IIRC. Some today are focused on keeping false positives down. A few benchmarked are here in this also-biased article: https://runtimeverification.com/match/1.0-SNAPSHOT/docs/benchmark/ https://runtimeverification.com/match/1.0-SNAPSHOT/docs/benc... I bring them up because they made the open-source K Framework and a C semantics. Another commenter says PVS-Studio is pretty good. Since Synopsis owns Coverity now, I'd recommend RV-Match (little to no false positives) followed by PVS-Studio.
- m0zg 7y ago
  Don't know about Coverity, but a customer did a Fortify scan on our code last year (as a part of acceptance), and it didn't find any issues. Which I find really hard to believe that there aren't any issues, seeing that the codebase is 200KLOC of pretty gnarly C++ with probably another 500KLOC in source-level C++ deps (we compile from source where we can). Either the team was comprised entirely of gods of C++ programming, or it doesn't really "fortify" all that much. I'll let you decide which one is more probable.