3 ms·

Did Coverity not pick up on any of the bugs that were later found? The folks behind PVStudio regularly spam the /r/cpp subreddit with actually very intersting

Did Coverity not pick up on any of the bugs that were later found? The folks behind PVStudio regularly spam the /r/cpp subreddit with actually very intersting articles about issues their static analyser finds in several open source projects. Does anyone here use it? (I do not work for them or use their product)

loeg 6y ago
Coverity has a bunch of false positives and true positives, and FreeBSD wasn't coverity-clean before this patch landed. No one in the FreeBSD dev community has time to watch every Coverity report that comes in.
- justin66 6y ago
  How does that work, is there a periodic review of static analysis results?
  - bluGill 6y ago
    If the number of false positives in your project is > 10 static analysis is a waste of time. It doesn't matter if your project is 10 lines, or 100 million, 10 is the most false positives static analysis is allowed to have before it is shut off as useless. It is very hard to be a static analysis tool developer. All the easy cases have been done, most have been folded into the compiler. What is left is the hard cases where you have to figure out how to not trigger on the false positives. Get this wrong and your customers will dump you - doesn't matter if you get it wrong by not detection many problems, or you get it wrong by too many false positives. It doesn't help that once the customer has fixed all the problems they forget about them, but the false positives come up all the time. You can't get out by making it possible to mark false positives. Once you allow that everyone will just get in the habit of marking all messages as a false positive. I tracked one production bug to a line that was immediately preceded by a false positive suppression - the problem was real but the someone decided instead of thinking they would suppress the issue like every other one found.
    - justin66 6y ago
      I've got some very real value out of static analysis tools, so I don't know quite what to say about all that. For sure it is a tool that takes time and effort to use correctly. Having used it, it can sometimes take a little imagination to consider how much time might have been wasted later in the game had the tool not been used. Not sure that any of that has to do with FreeBSD's use of Coverity, which I'm curious about. (rereading the OP I guess I'm not sure the FreeBSD project itself is a regular user of Coverity or if there's something else to it)

loeg 6y ago

Coverity has a bunch of false positives and true positives, and FreeBSD wasn't coverity-clean before this patch landed. No one in the FreeBSD dev community has time to watch every Coverity report that comes in.

justin66 6y ago
How does that work, is there a periodic review of static analysis results?
- bluGill 6y ago
  If the number of false positives in your project is > 10 static analysis is a waste of time. It doesn't matter if your project is 10 lines, or 100 million, 10 is the most false positives static analysis is allowed to have before it is shut off as useless. It is very hard to be a static analysis tool developer. All the easy cases have been done, most have been folded into the compiler. What is left is the hard cases where you have to figure out how to not trigger on the false positives. Get this wrong and your customers will dump you - doesn't matter if you get it wrong by not detection many problems, or you get it wrong by too many false positives. It doesn't help that once the customer has fixed all the problems they forget about them, but the false positives come up all the time. You can't get out by making it possible to mark false positives. Once you allow that everyone will just get in the habit of marking all messages as a false positive. I tracked one production bug to a line that was immediately preceded by a false positive suppression - the problem was real but the someone decided instead of thinking they would suppress the issue like every other one found.
  - justin66 6y ago
    I've got some very real value out of static analysis tools, so I don't know quite what to say about all that. For sure it is a tool that takes time and effort to use correctly. Having used it, it can sometimes take a little imagination to consider how much time might have been wasted later in the game had the tool not been used. Not sure that any of that has to do with FreeBSD's use of Coverity, which I'm curious about. (rereading the OP I guess I'm not sure the FreeBSD project itself is a regular user of Coverity or if there's something else to it)