4 ms·

In-Depth: Static Code Analysis (2011)

122 points by tzhenghao 8y ago

ThePhysicist 8y ago
It's important to keep in mind that there are vast differences in static analysis tools. Some of the commercial ones that John mentions in the post are quite powerful but also very expensive and out of reach for most startups / smaller companies. Most of these started off as spin-offs from research groups, and there's hundreds of man-years of research behind some of them, which (IMHO) makes the technology hard to replicate in the open-source world. Then there's open-source static analysis tools, which unfortunately are often much less powerful. Especially for dynamically typed scripting languages like Python it's very difficult writing good static analyzers because there's no type information available (projects like mypy should improve that though). In my opinion, when working with such languages gradual typing like done by MyPy / type annotations in Python or transpilation like Typescript does is a very good first investment when developing in these languages, and will also make static analysis easier later. There are some studies (I can't find the link now) that compared various measures for improving code quality in terms of cost and effectiveness, and the takeaway was that if you're starting out with QA manual code reviews are usually the most cost-effective way to improve software quality. Of course you will need to invest some time in getting a high-quality review process set up, and ensure that there are developers in your team that have the experience to do meaningful reviews.
- saagarjha 8y ago
  For the examples mentioned in the post, it seems like the Clang static analyzer might be enough? I’ve never seen anyone using a commercial analyzer; are these significantly better?
  - ThePhysicist 8y ago
    I have limited experience with them but I think tools like Coverity or Veracode are significantly better, but maybe they are just better at marketing. Coverity actually offers free scans for some open-source projects, here's the report for the Linux kernel: https://scan.coverity.com/projects/linux https://scan.coverity.com/projects/linux They have a large list of other projects as well (it seems you can't drill down into individual errors and only see summary statistics though, which sucks): https://scan.coverity.com/projects https://scan.coverity.com/projects I didn't find anything comparable for Veracode, but comparing the Coverity results with a static analyzer like Clang should give you an idea of the effectiveness. It's not a magic bullet (IMHO) but it definitely can find bugs that other tools don't see.
    - atq2119 8y ago
      Coverity is pretty good, but shows quite a lot of false positives in my experience. This may obviously depend on the project, of course. What I really wish for is an easily understandable way to teach these advanced static analysis tools about why something is a false positive, so that the tool can avoid showing the same false positive again in the future when a minor detail of the surrounding code changes.
      - UncleMeat 8y ago
        > What I really wish for is an easily understandable way to teach these advanced static analysis tools about why something is a false positive, so that the tool can avoid showing the same false positive again in the future when a minor detail of the surrounding code changes. People have been doing research on this for decades. The problem is that the way that humans understand code and the way that static analysis understands code are wildly different and the root cause of a false positive is either fundamentally not possible to avoid or so far removed from the final analysis that you can't really address it. Static analysis results are often brittle in fundamental but non-obvious ways. For example, a human might think that moving code between two functions shouldn't change much. But when you are using K Context Sensitivity, suddenly just wrapping some code in a helper function will completely fuck your points-to results.
- pjmlp 8y ago
  My experience in most projects I have been involved is that code reviews usually come after unit tests, which come after documentation.
- rasjani 8y ago
  While back, i did a tool that uses clang sa, tidy and cppcheck to provide more information to our existing Coverity scans. For coverity scans, mostly the meaningful stuff was enabled and results compared to clang & cppcheck where not that different. Next time id just to with those and not bother with coverity unless licensing is already there and some aspects require coverity.. Anyway, more is better. Coverity is nice once you get it setup and you have recentlish version but combine it with open source offerings too!
  - marktangotango 8y ago
    Interesting that back when McPeak was working on Elsa/elkhound clang wasn’t available. clang is definitely a game changer for c/c++ static analysis.
- mmirate 8y ago
  Plenty of idiomatic Python still contains enough magic to stymie mypy et al.
pieterr 8y ago
Paraphrasing Dijkstra: “Using static code analysis shows the presence, not the absence of bugs.” See: (1) “Finding Heartbleed with CodeSonar”, (2) “Why Do Software Assurance Tools Have Problems Finding Bugs Like Heartbleed?” [1] http://blogs.grammatech.com/finding-heartbleed-with-codesonar http://blogs.grammatech.com/finding-heartbleed-with-codesona... [2] https://www.swampinabox.org/doc/SWAMP-WP003-Heartbleed.pdf https://www.swampinabox.org/doc/SWAMP-WP003-Heartbleed.pdf
- nickpsecurity 8y ago
  I think that's a misapplication of the quote. Testing just checks an algorithm on specific inputs. The proofs check it for all inputs. Some static analyzers are designed to show absence of bugs of specific type for all inputs. They're more like proof. Astree Analyzer and TrustinSoft Analyzer are examples that claim to prove absence of errors with sound analysis.
nestorD 8y ago
Here is a good serie of articles on static analysis tools : http://btorpey.github.io/blog/categories/static-analysis/ http://btorpey.github.io/blog/categories/static-analysis/
shodan666 8y ago
There are some awesome free static analysers for Python: PYLINT: https://www.pylint.org/ https://www.pylint.org/ [one the best, all in one] RADON: https://radon.readthedocs.io/en/latest/ https://radon.readthedocs.io/en/latest/ - raw metrics: SLOC, comment lines, blank lines, &c. - Cyclomatic Complexity (i.e. McCabe’s Complexity) - Halstead metrics (all of them) - Maintainability Index Sonarlint for Pycharm/IDEA: https://www.sonarlint.org/intellij/ https://www.sonarlint.org/intellij/ I use all of them at once, and results are amazing! No more dirty code and finally we are able to measure the quality of our code over time in some quantifiable units (instead of individual judgment). Its not the best, but at least good foundation for writing good software. At each commit we got some cool report like: RADON: Maintability index (MI): generate_api.py - A (46.50) RADON: Average complexity: A (3.0) PYLINT: Your code has been rated at 9.31/10 (previous run: 9.31/10, +0.00) I call it "linter driven development.
nickpsecurity 8y ago
Two others worth checking out that focus on soundness with low to no false positives are RV-Match and TrustinSoft Analyzer: https://runtimeverification.com/match/ https://runtimeverification.com/match/ https://trust-in-soft.com/products/ https://trust-in-soft.com/products/ The RTV people are the one's behind K framework and an executable semantics for C with a GCC-like interface: http://www.kframework.org/index.php/Main_Page http://www.kframework.org/index.php/Main_Page https://github.com/kframework/c-semantics https://github.com/kframework/c-semantics That RV-Match builds on an open framework, does well on benchmarks (yeah skepticism here), and has little to no noise (most important) is what I like most about it. I say most important since digging through false positives makes developers not want to use a tool. I haven't tried either product myself, though. Here's a list of open tools that did well in small-scale experiments with and without false positives: https://klee.github.io https://klee.github.io http://www.cprover.org/cbmc/ http://www.cprover.org/cbmc/ https://cseweb.ucsd.edu/~rjhala/blast.html https://cseweb.ucsd.edu/~rjhala/blast.html http://www.splint.org http://www.splint.org http://spinroot.com/uno/ http://spinroot.com/uno/ http://cppcheck.sourceforge.net/#features http://cppcheck.sourceforge.net/#features https://en.wikipedia.org/wiki/Sparse https://en.wikipedia.org/wiki/Sparse https://cpachecker.sosy-lab.org https://cpachecker.sosy-lab.org

122 points by tzhenghao 8y ago

It's important to keep in mind that there are vast differences in static analysis tools. Some of the commercial ones that John mentions in the post are quite powerful but also very expensive and out of reach for most startups / smaller companies. Most of these started off as spin-offs from research groups, and there's hundreds of man-years of research behind some of them, which (IMHO) makes the technology hard to replicate in the open-source world. Then there's open-source static analysis tools, which unfortunately are often much less powerful. Especially for dynamically typed scripting languages like Python it's very difficult writing good static analyzers because there's no type information available (projects like mypy should improve that though). In my opinion, when working with such languages gradual typing like done by MyPy / type annotations in Python or transpilation like Typescript does is a very good first investment when developing in these languages, and will also make static analysis easier later. There are some studies (I can't find the link now) that compared various measures for improving code quality in terms of cost and effectiveness, and the takeaway was that if you're starting out with QA manual code reviews are usually the most cost-effective way to improve software quality. Of course you will need to invest some time in getting a high-quality review process set up, and ensure that there are developers in your team that have the experience to do meaningful reviews.

saagarjha 8y ago
For the examples mentioned in the post, it seems like the Clang static analyzer might be enough? I’ve never seen anyone using a commercial analyzer; are these significantly better?
- ThePhysicist 8y ago
  I have limited experience with them but I think tools like Coverity or Veracode are significantly better, but maybe they are just better at marketing. Coverity actually offers free scans for some open-source projects, here's the report for the Linux kernel: https://scan.coverity.com/projects/linux https://scan.coverity.com/projects/linux They have a large list of other projects as well (it seems you can't drill down into individual errors and only see summary statistics though, which sucks): https://scan.coverity.com/projects https://scan.coverity.com/projects I didn't find anything comparable for Veracode, but comparing the Coverity results with a static analyzer like Clang should give you an idea of the effectiveness. It's not a magic bullet (IMHO) but it definitely can find bugs that other tools don't see.
  - atq2119 8y ago
    Coverity is pretty good, but shows quite a lot of false positives in my experience. This may obviously depend on the project, of course. What I really wish for is an easily understandable way to teach these advanced static analysis tools about why something is a false positive, so that the tool can avoid showing the same false positive again in the future when a minor detail of the surrounding code changes.
    - UncleMeat 8y ago
      > What I really wish for is an easily understandable way to teach these advanced static analysis tools about why something is a false positive, so that the tool can avoid showing the same false positive again in the future when a minor detail of the surrounding code changes. People have been doing research on this for decades. The problem is that the way that humans understand code and the way that static analysis understands code are wildly different and the root cause of a false positive is either fundamentally not possible to avoid or so far removed from the final analysis that you can't really address it. Static analysis results are often brittle in fundamental but non-obvious ways. For example, a human might think that moving code between two functions shouldn't change much. But when you are using K Context Sensitivity, suddenly just wrapping some code in a helper function will completely fuck your points-to results.
pjmlp 8y ago
My experience in most projects I have been involved is that code reviews usually come after unit tests, which come after documentation.
rasjani 8y ago
While back, i did a tool that uses clang sa, tidy and cppcheck to provide more information to our existing Coverity scans. For coverity scans, mostly the meaningful stuff was enabled and results compared to clang & cppcheck where not that different. Next time id just to with those and not bother with coverity unless licensing is already there and some aspects require coverity.. Anyway, more is better. Coverity is nice once you get it setup and you have recentlish version but combine it with open source offerings too!
- marktangotango 8y ago
  Interesting that back when McPeak was working on Elsa/elkhound clang wasn’t available. clang is definitely a game changer for c/c++ static analysis.
mmirate 8y ago
Plenty of idiomatic Python still contains enough magic to stymie mypy et al.

pieterr 8y ago

Paraphrasing Dijkstra: “Using static code analysis shows the presence, not the absence of bugs.” See: (1) “Finding Heartbleed with CodeSonar”, (2) “Why Do Software Assurance Tools Have Problems Finding Bugs Like Heartbleed?” [1] http://blogs.grammatech.com/finding-heartbleed-with-codesonar http://blogs.grammatech.com/finding-heartbleed-with-codesona... [2] https://www.swampinabox.org/doc/SWAMP-WP003-Heartbleed.pdf https://www.swampinabox.org/doc/SWAMP-WP003-Heartbleed.pdf

nickpsecurity 8y ago
I think that's a misapplication of the quote. Testing just checks an algorithm on specific inputs. The proofs check it for all inputs. Some static analyzers are designed to show absence of bugs of specific type for all inputs. They're more like proof. Astree Analyzer and TrustinSoft Analyzer are examples that claim to prove absence of errors with sound analysis.

nestorD 8y ago

Here is a good serie of articles on static analysis tools : http://btorpey.github.io/blog/categories/static-analysis/ http://btorpey.github.io/blog/categories/static-analysis/

shodan666 8y ago

There are some awesome free static analysers for Python: PYLINT: https://www.pylint.org/ https://www.pylint.org/ [one the best, all in one] RADON: https://radon.readthedocs.io/en/latest/ https://radon.readthedocs.io/en/latest/ - raw metrics: SLOC, comment lines, blank lines, &c. - Cyclomatic Complexity (i.e. McCabe’s Complexity) - Halstead metrics (all of them) - Maintainability Index Sonarlint for Pycharm/IDEA: https://www.sonarlint.org/intellij/ https://www.sonarlint.org/intellij/ I use all of them at once, and results are amazing! No more dirty code and finally we are able to measure the quality of our code over time in some quantifiable units (instead of individual judgment). Its not the best, but at least good foundation for writing good software. At each commit we got some cool report like: RADON: Maintability index (MI): generate_api.py - A (46.50) RADON: Average complexity: A (3.0) PYLINT: Your code has been rated at 9.31/10 (previous run: 9.31/10, +0.00) I call it "linter driven development.

nickpsecurity 8y ago

Two others worth checking out that focus on soundness with low to no false positives are RV-Match and TrustinSoft Analyzer: https://runtimeverification.com/match/ https://runtimeverification.com/match/ https://trust-in-soft.com/products/ https://trust-in-soft.com/products/ The RTV people are the one's behind K framework and an executable semantics for C with a GCC-like interface: http://www.kframework.org/index.php/Main_Page http://www.kframework.org/index.php/Main_Page https://github.com/kframework/c-semantics https://github.com/kframework/c-semantics That RV-Match builds on an open framework, does well on benchmarks (yeah skepticism here), and has little to no noise (most important) is what I like most about it. I say most important since digging through false positives makes developers not want to use a tool. I haven't tried either product myself, though. Here's a list of open tools that did well in small-scale experiments with and without false positives: https://klee.github.io https://klee.github.io http://www.cprover.org/cbmc/ http://www.cprover.org/cbmc/ https://cseweb.ucsd.edu/~rjhala/blast.html https://cseweb.ucsd.edu/~rjhala/blast.html http://www.splint.org http://www.splint.org http://spinroot.com/uno/ http://spinroot.com/uno/ http://cppcheck.sourceforge.net/#features http://cppcheck.sourceforge.net/#features https://en.wikipedia.org/wiki/Sparse https://en.wikipedia.org/wiki/Sparse https://cpachecker.sosy-lab.org https://cpachecker.sosy-lab.org