4 ms·

:) Cool that you recognize that. This is correct. Coverity does a bunch of specific analyses designed to eliminate False positives. This includes analysis to d

by jetru 8y ago

:) Cool that you recognize that. This is correct. Coverity does a bunch of specific analyses designed to eliminate False positives. This includes analysis to determine which data states are not possible for a given code path, and doesn't report those specific issues. This also works using data across function calls. For C/C++/C#, Coverity has by far the lowest false positive rates, which make it probably the best in class for those languages. Disclosure: I used to work at Coverity.

sanxiyn 8y ago
Fascinating! The specific example you mentioned sounds publishable, any idea? I mean, I am just curious, I no longer work on this. I now have quite different perspective wrt false positives; that false positive rate is not important. It's all about perspectives. Name the tool bug search engine instead of bug finding tool. You rarely look beyond the first page of search engine result. Develop a ranking algorithm such that all alarms in the first page is relevant. You can use any probabilistic voodoo to rank. Etc. I don't know whether this can work. But I think it's worth a try.
- tacostakohashi 8y ago
  I think the takeaway here is that Coverity have just made a (clever) business decision to eliminate false positives, so that (nearly) 100% of their defects are "correct". It's quite possible that there are real defects that they don't show because their confidence is less than 100%.
  - sanxiyn 8y ago
    No, not really. "false positives are the enemy" (or at least, in my alternative model, relevancy) becomes glaringly obvious for anyone who tried this at scale. The question is how to reduce false positives (or improve relevancy). The part that "real defects not shown" exist is called unsoundness and pretty much everybody now agrees you should be unsound. (Or being sound is a different market.) Of course, this is not obvious to people who haven't tried, so it is reported over and over and over. The most recently, from Google. Lessons from Building Static Analysis Tools at Google (2018). https://cacm.acm.org/magazines/2018/4/226371-lessons-from-building-static-analysis-tools-at-google/fulltext https://cacm.acm.org/magazines/2018/4/226371-lessons-from-bu... I mean, I could have told them. Select quotes: > Unlike compile-time checks, analysis results shown during code review are allowed to include up to 10% effective false positives. IMPORTANT NOTE: 10% effective false positives means much less than 10% false positives! In above quote, Google defines true bugs marked as not-a-bug by a developer as effectively false positives. If what you say is true, but users misunderstand, it doesn't count.

by jetru 8y ago

sanxiyn 8y ago

Fascinating! The specific example you mentioned sounds publishable, any idea? I mean, I am just curious, I no longer work on this. I now have quite different perspective wrt false positives; that false positive rate is not important. It's all about perspectives. Name the tool bug search engine instead of bug finding tool. You rarely look beyond the first page of search engine result. Develop a ranking algorithm such that all alarms in the first page is relevant. You can use any probabilistic voodoo to rank. Etc. I don't know whether this can work. But I think it's worth a try.

tacostakohashi 8y ago
I think the takeaway here is that Coverity have just made a (clever) business decision to eliminate false positives, so that (nearly) 100% of their defects are "correct". It's quite possible that there are real defects that they don't show because their confidence is less than 100%.
- sanxiyn 8y ago
  No, not really. "false positives are the enemy" (or at least, in my alternative model, relevancy) becomes glaringly obvious for anyone who tried this at scale. The question is how to reduce false positives (or improve relevancy). The part that "real defects not shown" exist is called unsoundness and pretty much everybody now agrees you should be unsound. (Or being sound is a different market.) Of course, this is not obvious to people who haven't tried, so it is reported over and over and over. The most recently, from Google. Lessons from Building Static Analysis Tools at Google (2018). https://cacm.acm.org/magazines/2018/4/226371-lessons-from-building-static-analysis-tools-at-google/fulltext https://cacm.acm.org/magazines/2018/4/226371-lessons-from-bu... I mean, I could have told them. Select quotes: > Unlike compile-time checks, analysis results shown during code review are allowed to include up to 10% effective false positives. IMPORTANT NOTE: 10% effective false positives means much less than 10% false positives! In above quote, Google defines true bugs marked as not-a-bug by a developer as effectively false positives. If what you say is true, but users misunderstand, it doesn't count.