3 ms·
It's important to keep in mind that there are vast differences in static analysis tools. Some of the commercial ones that John mentions in the post are quite po
by ThePhysicist 8y ago
It's important to keep in mind that there are vast differences in static analysis tools. Some of the commercial ones that John mentions in the post are quite powerful but also very expensive and out of reach for most startups / smaller companies. Most of these started off as spin-offs from research groups, and there's hundreds of man-years of research behind some of them, which (IMHO) makes the technology hard to replicate in the open-source world.
Then there's open-source static analysis tools, which unfortunately are often much less powerful. Especially for dynamically typed scripting languages like Python it's very difficult writing good static analyzers because there's no type information available (projects like mypy should improve that though). In my opinion, when working with such languages gradual typing like done by MyPy / type annotations in Python or transpilation like Typescript does is a very good first investment when developing in these languages, and will also make static analysis easier later.
There are some studies (I can't find the link now) that compared various measures for improving code quality in terms of cost and effectiveness, and the takeaway was that if you're starting out with QA manual code reviews are usually the most cost-effective way to improve software quality. Of course you will need to invest some time in getting a high-quality review process set up, and ensure that there are developers in your team that have the experience to do meaningful reviews.
- saagarjha 8y agoFor the examples mentioned in the post, it seems like the Clang static analyzer might be enough? I’ve never seen anyone using a commercial analyzer; are these significantly better?
- ThePhysicist 8y agoI have limited experience with them but I think tools like Coverity or Veracode are significantly better, but maybe they are just better at marketing. Coverity actually offers free scans for some open-source projects, here's the report for the Linux kernel: https://scan.coverity.com/projects/linux https://scan.coverity.com/projects/linux They have a large list of other projects as well (it seems you can't drill down into individual errors and only see summary statistics though, which sucks): https://scan.coverity.com/projects https://scan.coverity.com/projects I didn't find anything comparable for Veracode, but comparing the Coverity results with a static analyzer like Clang should give you an idea of the effectiveness. It's not a magic bullet (IMHO) but it definitely can find bugs that other tools don't see.
- atq2119 8y agoCoverity is pretty good, but shows quite a lot of false positives in my experience. This may obviously depend on the project, of course. What I really wish for is an easily understandable way to teach these advanced static analysis tools about why something is a false positive, so that the tool can avoid showing the same false positive again in the future when a minor detail of the surrounding code changes.
- UncleMeat 8y ago> What I really wish for is an easily understandable way to teach these advanced static analysis tools about why something is a false positive, so that the tool can avoid showing the same false positive again in the future when a minor detail of the surrounding code changes. People have been doing research on this for decades. The problem is that the way that humans understand code and the way that static analysis understands code are wildly different and the root cause of a false positive is either fundamentally not possible to avoid or so far removed from the final analysis that you can't really address it. Static analysis results are often brittle in fundamental but non-obvious ways. For example, a human might think that moving code between two functions shouldn't change much. But when you are using K Context Sensitivity, suddenly just wrapping some code in a helper function will completely fuck your points-to results.
- pjmlp 8y agoMy experience in most projects I have been involved is that code reviews usually come after unit tests, which come after documentation.
- rasjani 8y agoWhile back, i did a tool that uses clang sa, tidy and cppcheck to provide more information to our existing Coverity scans. For coverity scans, mostly the meaningful stuff was enabled and results compared to clang & cppcheck where not that different. Next time id just to with those and not bother with coverity unless licensing is already there and some aspects require coverity.. Anyway, more is better. Coverity is nice once you get it setup and you have recentlish version but combine it with open source offerings too!
- marktangotango 8y agoInteresting that back when McPeak was working on Elsa/elkhound clang wasn’t available. clang is definitely a game changer for c/c++ static analysis.
- mmirate 8y agoPlenty of idiomatic Python still contains enough magic to stymie mypy et al.