4 ms·
Right, and so I've developed the view that the most interesting quality of a static analysis tool is not "how many bugs does it find" but "how many of the thing
by pm215 10y ago
Right, and so I've developed the view that the most interesting quality of a static analysis tool is not "how many bugs does it find" but "how many of the things it points out are false positives" and (if you want to be really tough) "how many of the things it points out are likely to cause problems for a real user or be actually exploitable by an attacker". Most software projects already have a huge list of bugs they haven't fixed yet, so a tool that identifies 10 easy to fix real bugs is great, but a tool that mixes those 10 real bugs in with a hundred false positives is much less useful, simply because of the time it takes somebody to sift through the list...
- eremeevp 10y agoI would say, that in this case, with the 50\50 false positive ratio, it is quite manageable, i.e. for every 2 analyzer messages, there is 1 interesting issue in the source code.
- nickpsecurity 10y agoYes. I initially promoted all kinds of static analysis. These days, I'm firmly convinced the friction aspect means the tool needs to be plug-n-play with almost no false positives. Ideally no false positives at all. That maximizes uptake and knocks out many bugs even if more remain. Another thing I think advocates need to do with lists of such tools is always mention that each one might see errors another won't. This needs to be a refrain that instantly is in people's minds when they wonder about using such tools. Reason is that many make the mistakes of thinking one tool not finding anything means (a) tools don't find stuff in general or (b) there's nothing to find. That implication is even on SQLlite testing page where they applied Clang's analyzer then implied a weaker tool finding nothing in a good codebase meant static analysis itself had nothing to offer (?!).