4 ms·

That is not what dynamic analysis means but I appreciate everyone redefining terms to suit their marketing. https://en.wikipedia.org/wiki/Dynamic_program_analys

That is not what dynamic analysis means but I appreciate everyone redefining terms to suit their marketing. https://en.wikipedia.org/wiki/Dynamic_program_analysis https://en.wikipedia.org/wiki/Dynamic_program_analysis

sriram_sun 10y ago
Good point. The wikipedia definition is my understanding as well. Analyzing during build time is what scan-build does. They call it the Clang Static Analyzer.
- moyix 10y ago
  Coverity has been doing "dynamic analysis" in the way this article uses the term for a long time; they ended up creating a system call tracer that lets them monitor an arbitrary build process to find code to analyze: http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later/fulltext http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lin... (starting from "Laws of Bug Finding")
Animats 10y ago
Does this look at code at all? Or is it just some build tool to make sure you're only using Approved Versions of Approved Components?
joshuata 10y ago
Agreed. Their definition of static analysis is laughably bizarre as well; there are many security and safety properties that can be proved by static analysis but not by dynamic analysis. Additionally, many of the problems they attribute to static analysis (non-specific versioning and recursive dependencies) are caused by the package managers themselves, and can be solved using static analysis if the build tool is well defined and well behaved. Their tool has a few major weaknesses: 1. Builds are not reproducible - Reproducible builds require pinned package versions, which they specifically avoid. This could result in security holes if a dependent package version was bumped after the parent project was tested. 2. Subject of analysis - Their main target of analysis is the build file. This severely limits the extent of their analysis and requires them to build new tools for each build tool used. 3. Underestimation - Since a code path must be exercised in order to analyze it, you are guaranteed that the set of vulnerabilities detected is a subset of (or equal to) the true set of vulnerabilities. This is the opposite of what I believe should be the default: Always prefer false positives over false negatives. Both static analysis and binary analysis allow the programmer to over-estimate their analysis, guaranteeing that all vulnerabilities will be found.
- nickpsecurity 10y ago
  I was going to say same about build systems being able to do static analysis. Some can even be correct-by-construction. Further, if compiler like CompCert or CakeML is used, then static analysis on source will apply to binary too. Finally, static tools can analyze a lot more than just one file or whatever. This article is BS.
- jwn 10y ago
  I'm not sure what you mean by #1, the builds are as reproducible as the build system being used. For #2, SourceClear doesn't build the software under test, it's hooked into the build via various methods. For Maven and Gradle, those are plugins. For NPM and Bundler, the existing build files contain the complete dependency graphs as determined by the build system. The analysis is quite accurate, I daresay more so than any other tool. Yes, it requires implementation work for each build stack, but that's the price you pay for accuracy. In response to your #3, SourceClear doesn't report only vulnerabilities verified by call paths, it reports on all components will known vulnerabilities and denotes if a call path was found. Disclaimer, I'm a Co-Founder and have spent a great deal of time writing scanning code.
jcranmer 10y ago
None of the terms they define share the definitions that are understood by everyone in the field. Static analysis refers to analysis techniques that understand code without recourse to execution, usually during compilation. (Static binary analysis does exist, but the difficulties of disentangling code from data in native code makes it usually impracticable). The Java FindBugs tools or Clang's static analyzer are good examples of static analysis techniques. Dynamic analysis refers to analysis techniques that rely on gathering information using actual executions of a program. Even pedestrian techniques like profiling and code coverage are considered dynamic analysis techniques. Binary analysis is a completely orthogonal concept. It refers to the analysis of programs without recourse to the source code, and is usually understood to refer almost exclusively to native machine code as opposed to bytecode languages like JVM or .NET IL. Most binary analysis tools are dynamic (such as Pin or Valgrind) but there do exist static binary analysis tools such as GrammaTech's CodeSurfer. This doesn't really capture the whole picture of techniques, though. Symbolic execution and its step-child concolic execution aren't best thought of as static or dynamic analysis.
skybrian 10y ago
If you consider the build system to be a language in itself, I think this is appropriately called dynamic analysis. For example, a profiler for a Makefile should record the time taken for each build step as the Makefile is running. It's not about profiling the program being built.

sriram_sun 10y ago

Good point. The wikipedia definition is my understanding as well. Analyzing during build time is what scan-build does. They call it the Clang Static Analyzer.

moyix 10y ago
Coverity has been doing "dynamic analysis" in the way this article uses the term for a long time; they ended up creating a system call tracer that lets them monitor an arbitrary build process to find code to analyze: http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later/fulltext http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lin... (starting from "Laws of Bug Finding")

Animats 10y ago

Does this look at code at all? Or is it just some build tool to make sure you're only using Approved Versions of Approved Components?

joshuata 10y ago

Agreed. Their definition of static analysis is laughably bizarre as well; there are many security and safety properties that can be proved by static analysis but not by dynamic analysis. Additionally, many of the problems they attribute to static analysis (non-specific versioning and recursive dependencies) are caused by the package managers themselves, and can be solved using static analysis if the build tool is well defined and well behaved. Their tool has a few major weaknesses: 1. Builds are not reproducible - Reproducible builds require pinned package versions, which they specifically avoid. This could result in security holes if a dependent package version was bumped after the parent project was tested. 2. Subject of analysis - Their main target of analysis is the build file. This severely limits the extent of their analysis and requires them to build new tools for each build tool used. 3. Underestimation - Since a code path must be exercised in order to analyze it, you are guaranteed that the set of vulnerabilities detected is a subset of (or equal to) the true set of vulnerabilities. This is the opposite of what I believe should be the default: Always prefer false positives over false negatives. Both static analysis and binary analysis allow the programmer to over-estimate their analysis, guaranteeing that all vulnerabilities will be found.

nickpsecurity 10y ago
I was going to say same about build systems being able to do static analysis. Some can even be correct-by-construction. Further, if compiler like CompCert or CakeML is used, then static analysis on source will apply to binary too. Finally, static tools can analyze a lot more than just one file or whatever. This article is BS.
jwn 10y ago
I'm not sure what you mean by #1, the builds are as reproducible as the build system being used. For #2, SourceClear doesn't build the software under test, it's hooked into the build via various methods. For Maven and Gradle, those are plugins. For NPM and Bundler, the existing build files contain the complete dependency graphs as determined by the build system. The analysis is quite accurate, I daresay more so than any other tool. Yes, it requires implementation work for each build stack, but that's the price you pay for accuracy. In response to your #3, SourceClear doesn't report only vulnerabilities verified by call paths, it reports on all components will known vulnerabilities and denotes if a call path was found. Disclaimer, I'm a Co-Founder and have spent a great deal of time writing scanning code.

jcranmer 10y ago

None of the terms they define share the definitions that are understood by everyone in the field. Static analysis refers to analysis techniques that understand code without recourse to execution, usually during compilation. (Static binary analysis does exist, but the difficulties of disentangling code from data in native code makes it usually impracticable). The Java FindBugs tools or Clang's static analyzer are good examples of static analysis techniques. Dynamic analysis refers to analysis techniques that rely on gathering information using actual executions of a program. Even pedestrian techniques like profiling and code coverage are considered dynamic analysis techniques. Binary analysis is a completely orthogonal concept. It refers to the analysis of programs without recourse to the source code, and is usually understood to refer almost exclusively to native machine code as opposed to bytecode languages like JVM or .NET IL. Most binary analysis tools are dynamic (such as Pin or Valgrind) but there do exist static binary analysis tools such as GrammaTech's CodeSurfer. This doesn't really capture the whole picture of techniques, though. Symbolic execution and its step-child concolic execution aren't best thought of as static or dynamic analysis.

skybrian 10y ago

If you consider the build system to be a language in itself, I think this is appropriately called dynamic analysis. For example, a profiler for a Makefile should record the time taken for each build step as the Makefile is running. It's not about profiling the program being built.