5 ms·

Big discussion: Do Static Source Code Analysis Tools Really Work?

twiceaday 15y ago
If the tools were completely free and easy to integrate would there even be a discussion?
- wink 15y ago
  Yes, because you can be swamped in false positives so much that it's easier on time, money and nerves to just not use the tool. And this is a general problem in static source code analysis, not just the "bad tools". I had this awesome example in one book about metrics where the static analysis on some 100k LOC project produced output with the same line length as "War and Peace". Good luck sifting through those. But I'm absolutely not saying you shouldn't use them, but as some commenters in the original post pointed out, you need to check if they produce sensible output for your project or if they can catch some really nasty bugs, if it's worth the additional work.
tzs 15y ago
Carmack wrote about static code analysis a few months ago: http://altdevblogaday.com/2011/12/24/static-code-analysis/ http://altdevblogaday.com/2011/12/24/static-code-analysis/ It works.
seclorum 15y ago
I've used Coverity in the past to radically increase the quality of a large SIL-4 code-base. (Safety-critical code for a major industrial transportation systems provider.) Without question, it helped my project gain a large step in terms of quality and functionality and was definitely worth it in the context of the regular costs of the project. I won't say the codebase became 100% bug-free, but I will say that it helped us rapidly increase the quality of our code in ways that could only have cost us a lot of time with testing, testing, testing .. and more testing. It found bugs that were extremely esoteric, but nevertheless, extremely important to eradicate, especially in a SIL-4 codebase. My only desire is that these powerful tools become more widely available outside of the safety-critical scene. It would be wonderful to have this sort of power for some of my own smaller, less critical projects - it definitely makes one a better, more careful, programmer.
funkah 15y ago
They do. Really.
moldbug 15y ago
Great comment from the Slashdot thread (direct link: http://developers.slashdot.org/comments.pl?sid=557384&cid=23464230 http://developers.slashdot.org/comments.pl?sid=557384&ci...): My experience has been that while in the hands of people who know what they're doing, they're a nice tool to have, well, beware managers using their output as metrics. And beware even more a consultant with such a tool that he doesn't even understand. The thing is, these tools produce A) a lot of "false positives", code which is really OK and everyone understand why it's ok, but the tool will still complain, and B) usually includes some metrics of dubious quality at best, to be taken only as a signal for a human to look at it and understand why it's ok or not ok. E.g., ne such tool, which I had the misfortune of sitting through a salesman hype session of, seemed to be really little more than a glorified grep. It really just looked at the source text, not at what's happening. So for example if you got a database connection and a statement in a "try" block, it wanted to see the close statements in the "finally" block. Well, applied to an actual project, there was a method which just closed the connection and the statements supplied as an array. Just because, you know, it's freaking stupid to copy-and-paste cute little "if (connection != null) { try { connection.close(); } catch (SQLException e) { // ignore }}" blocks a thousand times over in each "finally" block, when you can write it once and just call the method in your finally block. This tool had a trouble understanding that it _is_ all right. Unless it saw the "connection.close()" right there, in the finally block, it didn't count. Other examples include more mundane stuff like the tools recommending that you synchronize or un-synchronize a getter, even when everyone understands why it's OK for it to be as it is. E.g., a _stateless_ class as a singleton is just an (arguably premature and unneded) speed optimization, because some people think they're saving so much by a singleton instead of the couple of cycles it takes to do a new on a class with no members and no state. It doesn't really freaking matter if there's exactly one of it, or someone gets a copy of it. But invariably the tools will make an "OMG, unsynchronized singleton" fuss, because they don't look deep enough to see if there's actually some state that must be unique. Etc. Now taken as something that each developper understands, runs on his own when he needs it, and uses his judgment of each point, it's a damn good thing anyway. Enter the clueless PHB with a metric and chart fetish, stage left. This guy doesn't understand what those things are, but might make it his personal duty to chart some progress by showing how much fewer warnings he's got from the team this week than last week. So useless man-hours are spent on useless morphing perfectly good code, into something that games the tool. For each 1 real bug found, there'll be 100 harmless warnings that he makes it his personal mission to get out of the code. Enter the snake-oil vendor's salesman, stage right. This guy only cares about selling some extra copies to justify his salary. He'll hype to the boss exactly the possibility to generate such charts (out of mostly false positives) and manage by such charts. If the boss wasn't already in a mind to do that management anti-pattern, the salesman will try to teach him to. 'Cause that's usually the only advantage that his expensive tool has over those open source tools that you mention. I'm not kidding. I actually tried to corner one into; Me: "ok, but you said not everything it flags there is a bug, right?" Him: "Yes, you need to actually look at them and see if they're bugs or not." Me: "Then what sense does it make to generate charts based on wholesale counting entities which may, or may not be bugs?" Him: "Well, you can use the charts to see, say, a trend that you have less of them over time, so the project is getting better." Me: "But they may or may not be actual bugs. How do you know if this week's mix has more or less actual bugs than last weeks, regardless of what the total there is?" Him: "Well, yes, you need to actually look at them in turn to see which are actual bugs." Me: "But that's not what the tool counts. It counts a total which includes an unknown, and likely majority, number of false positives." Him: "Well, yes." Me: "So what use is that kind of a chart then?" Him: "Well, you can get a line or bar graph that shows how much progress is made in removing them." Lather, rinse, repeat, give up. Or enter the consultant with a tool. Now while I'll be the first to say that a lot of projects do end up needing a good consultant to get the code to perform right, I'll also warn about the thousands of parasites posing as one. There are more than enough guys who don't actually have any clue, and all they do is run the code through such a tool, and expect miracles to happen. They don't see the big fat SQL query that causes the delay, or the needless loop it's in, but they'll advise blind compliance to the 1 microsecond saving tips of their tool. And of course, again, try to justify his fee by reducing such a total of harmless warnings. And I guess the moral of the story is that you also need to train the people to understand those tools, and understand what they can ignore and what not. It might mean a bit more expense than just downloading the tools. Especially training some managers what _not_ to do with them.

If the tools were completely free and easy to integrate would there even be a discussion?

wink 15y ago
Yes, because you can be swamped in false positives so much that it's easier on time, money and nerves to just not use the tool. And this is a general problem in static source code analysis, not just the "bad tools". I had this awesome example in one book about metrics where the static analysis on some 100k LOC project produced output with the same line length as "War and Peace". Good luck sifting through those. But I'm absolutely not saying you shouldn't use them, but as some commenters in the original post pointed out, you need to check if they produce sensible output for your project or if they can catch some really nasty bugs, if it's worth the additional work.

tzs 15y ago

Carmack wrote about static code analysis a few months ago: http://altdevblogaday.com/2011/12/24/static-code-analysis/ http://altdevblogaday.com/2011/12/24/static-code-analysis/ It works.

seclorum 15y ago

I've used Coverity in the past to radically increase the quality of a large SIL-4 code-base. (Safety-critical code for a major industrial transportation systems provider.) Without question, it helped my project gain a large step in terms of quality and functionality and was definitely worth it in the context of the regular costs of the project. I won't say the codebase became 100% bug-free, but I will say that it helped us rapidly increase the quality of our code in ways that could only have cost us a lot of time with testing, testing, testing .. and more testing. It found bugs that were extremely esoteric, but nevertheless, extremely important to eradicate, especially in a SIL-4 codebase. My only desire is that these powerful tools become more widely available outside of the safety-critical scene. It would be wonderful to have this sort of power for some of my own smaller, less critical projects - it definitely makes one a better, more careful, programmer.

funkah 15y ago

They do. Really.

moldbug 15y ago

Great comment from the Slashdot thread (direct link: http://developers.slashdot.org/comments.pl?sid=557384&cid=23464230 http://developers.slashdot.org/comments.pl?sid=557384&ci...): My experience has been that while in the hands of people who know what they're doing, they're a nice tool to have, well, beware managers using their output as metrics. And beware even more a consultant with such a tool that he doesn't even understand. The thing is, these tools produce A) a lot of "false positives", code which is really OK and everyone understand why it's ok, but the tool will still complain, and B) usually includes some metrics of dubious quality at best, to be taken only as a signal for a human to look at it and understand why it's ok or not ok. E.g., ne such tool, which I had the misfortune of sitting through a salesman hype session of, seemed to be really little more than a glorified grep. It really just looked at the source text, not at what's happening. So for example if you got a database connection and a statement in a "try" block, it wanted to see the close statements in the "finally" block. Well, applied to an actual project, there was a method which just closed the connection and the statements supplied as an array. Just because, you know, it's freaking stupid to copy-and-paste cute little "if (connection != null) { try { connection.close(); } catch (SQLException e) { // ignore }}" blocks a thousand times over in each "finally" block, when you can write it once and just call the method in your finally block. This tool had a trouble understanding that it _is_ all right. Unless it saw the "connection.close()" right there, in the finally block, it didn't count. Other examples include more mundane stuff like the tools recommending that you synchronize or un-synchronize a getter, even when everyone understands why it's OK for it to be as it is. E.g., a _stateless_ class as a singleton is just an (arguably premature and unneded) speed optimization, because some people think they're saving so much by a singleton instead of the couple of cycles it takes to do a new on a class with no members and no state. It doesn't really freaking matter if there's exactly one of it, or someone gets a copy of it. But invariably the tools will make an "OMG, unsynchronized singleton" fuss, because they don't look deep enough to see if there's actually some state that must be unique. Etc. Now taken as something that each developper understands, runs on his own when he needs it, and uses his judgment of each point, it's a damn good thing anyway. Enter the clueless PHB with a metric and chart fetish, stage left. This guy doesn't understand what those things are, but might make it his personal duty to chart some progress by showing how much fewer warnings he's got from the team this week than last week. So useless man-hours are spent on useless morphing perfectly good code, into something that games the tool. For each 1 real bug found, there'll be 100 harmless warnings that he makes it his personal mission to get out of the code. Enter the snake-oil vendor's salesman, stage right. This guy only cares about selling some extra copies to justify his salary. He'll hype to the boss exactly the possibility to generate such charts (out of mostly false positives) and manage by such charts. If the boss wasn't already in a mind to do that management anti-pattern, the salesman will try to teach him to. 'Cause that's usually the only advantage that his expensive tool has over those open source tools that you mention. I'm not kidding. I actually tried to corner one into; Me: "ok, but you said not everything it flags there is a bug, right?" Him: "Yes, you need to actually look at them and see if they're bugs or not." Me: "Then what sense does it make to generate charts based on wholesale counting entities which may, or may not be bugs?" Him: "Well, you can use the charts to see, say, a trend that you have less of them over time, so the project is getting better." Me: "But they may or may not be actual bugs. How do you know if this week's mix has more or less actual bugs than last weeks, regardless of what the total there is?" Him: "Well, yes, you need to actually look at them in turn to see which are actual bugs." Me: "But that's not what the tool counts. It counts a total which includes an unknown, and likely majority, number of false positives." Him: "Well, yes." Me: "So what use is that kind of a chart then?" Him: "Well, you can get a line or bar graph that shows how much progress is made in removing them." Lather, rinse, repeat, give up. Or enter the consultant with a tool. Now while I'll be the first to say that a lot of projects do end up needing a good consultant to get the code to perform right, I'll also warn about the thousands of parasites posing as one. There are more than enough guys who don't actually have any clue, and all they do is run the code through such a tool, and expect miracles to happen. They don't see the big fat SQL query that causes the delay, or the needless loop it's in, but they'll advise blind compliance to the 1 microsecond saving tips of their tool. And of course, again, try to justify his fee by reducing such a total of harmless warnings. And I guess the moral of the story is that you also need to train the people to understand those tools, and understand what they can ignore and what not. It might mean a bit more expense than just downloading the tools. Especially training some managers what _not_ to do with them.