4 ms·

I recently did an extensive competitive analysis of SAST tools for a client. Anyone who is thinking about buying one of these tools should pay attention to what

I recently did an extensive competitive analysis of SAST tools for a client. Anyone who is thinking about buying one of these tools should pay attention to what versions of each language they support. Also try to get old release notes in order to determine when they first supported a particular language version. Many vendors take a good long time to support new versions of languages, even mainstream ones like Java and the .net family. None of them are particularly helpful in getting this information to you. They have their marketing checklists and information deeper than this can be hard to come by from the salespeople. A few were scared of letting us have this information at all once they knew it was for a competitive analysis. That's a sign that they take a long time to support new language versions. In many companies adoption of new language versions happen organically at the developer level, often within days of the new version being released. Even if the system admins try to press the brakes a bit on deploying the new version on production servers, the pressure is there for it to happen. SAST vendors typically are not going to be able to keep pace, which will make your developers unhappy or even give them an excuse for not using the expensive tool you purchased.

zxcmx 6y ago
Not just languages - Generally if the scanner isn't aware of the specific framework you're using it's unable to find entrypoints or reliably figure out sources / sinks and filters. Many modern frameworks have one or more of: dynamic configuration, compile time annotations, reflection, IoC etc which make it very hard for a "first principles" scanner to make sense of what can actually happen at runtime.
- pc86 6y ago
  What exactly is a "'first principles' scanner?"
  - zxcmx 6y ago
    Sorry, was perhaps poorly worded. By "first principles" I meant a static analyser working using only an understanding of the language (say, Java) and the application source code, looking over the ast/cfg etc using only built-in language rules. You get much better (read: maybe useful) results if you happen to have a "rule pack" for the specific framework you're using which provides specific hints on sources + sinks, "gotchas" and how things are wired together. As a somewhat obsolete example, I would not expect a sast working only from "first principles" to be able to do anything useful with a Spring XML configuration file. On the whole my experience is that these things work very well on certain types of codebases - e.g. naive PHP they can "go to town" because of the huge footgun surface and fairly direct control and data transfer. Stuff with lots of "magical" framework features and indirection (where even a human reviewer can often have trouble finding the implementation from the interface being invoked) they often silently fail to do anything useful.
    - bostik 6y ago
      > static analyser working using only an understanding of the language [...] looking over the ast The part I quoted is called "tainting". Or if you are more academically inclined, "data flow analysis". When it works right, it is an incredible force multiplier in security. You get detailed, actionable and above all helpful error messages directly from CI, because as a static analysis tool it's pretty fast and can be made part of the common linting pass. As you hinted, it does require a suitable config setup and/or code annotations to mark sources and sinks. And when it does work, it can eliminate an entire class of vulnerabilities - good taint analysis will prevent you from even accidentally using user-supplied data in anything that involves relaying, storing or displaying information. The downside is, when it doesn't work, it's a source of unhappiness. Debugging a taint failure because the AST analysis gives a false positive can be infuriating.
  - schwag09 6y ago
    I suspect this means a scanner that can derive all necessary information without any configuration. For example, consider a scanner looking for API endpoint authorization inconsistencies. Does the scanner need you to describe your authorization scheme, or can you simply run the thing and it figures it out? This can be easy or hard depending on how bespoke your application is. If you're using something like Ruby on Rails, then there's a paved road that a scanner can preconfigure to understand your application. If you're using a homegrown authorization framework, then a scanner will likely have a hard time understanding your application and will need to be configured.
- airza 6y ago
  Even Fortify and other tools which have to compile beforehand don't do so well IMO. Someone is going to come up with an actually good SAST and print money...
  - tru3_power 6y ago
    I’ve had really good experiences with Fortify so far. It took me a lot of hours to figure out how to write custom rules but once I was able to do so it became super valuable.
- Rickasaurus 6y ago
  I didn't write about this because it wasn't toward my exact point about FP, but you're totally right. This will be equally as chilling for new web frameworks.
clumsysmurf 6y ago
This. I work on an Android app written in Java. We were thinking about migrating to Kotlin years ago, but our blessed tool - Fortify - only just released a technical preview for Kotlin 1.3.50 in Fortify 20.1.0, and already Kotlin 1.4.10 is out.
- Mountain_Skies 6y ago
  My impression is that the current owners of Fortify lack the resources that HPE had and the product is suffering as a result. It's best suited for maintaining old code bases.
beh9540 6y ago
This cannot be stated enough. The company I worked for purchased "the leading gartner SAST testing solution" before they even started a project and had a framework chosen. It turned out the SAST solution didn't support the latest major version of the framework we were using at the time, and didn't support the latest version of the language we were using. Even worse, it never seemed to ever report an issue, and took hours to run at times, eating up build minutes. I ran bandit on the code base just for fun one day, and we had four hits and it took 5 minutes to run. It took a while, but I finally convinced the powers that be we were better using the tool we could verify works, rather than trusting the SAST vendor that it worked.
- acdha 6y ago
  Fortify took almost a decade to support Python 3 and there were some hefty lags on Java updates, where multiple times it was like “do you want to use the supported Java or the one which Fortify can scan?”
- chair6 6y ago
  waves, original Bandit author here. It could've done more, but pretty cool to see how useful it's been and where it's got to.. OSS is fun.
newera2016 6y ago
Wow. Great. Would you be able to publish your analysis?
- Mountain_Skies 6y ago
  Unfortunately I cannot as it is not my property. Plus one vendor made my client and myself sign an NDA even for the most basic information. What I can say is to not put too much faith in the Gartner Magic Quadrant for this space. The best thing to do is select a couple of representative projects for your company and get each vendor to let you do a POC and then compare the scan results from each. The process of making this happen will be long and painful as it is managed mostly by sales teams rather than the tech folks. In the end though, it's the best way to know what will work for your particular set of development technologies. The good news is pretty much every vendor was receptive to the idea of doing a POC even if it took a while to get each one set up.
bostik 6y ago
Pray that you never have to enter domains where the security audits and assessments must be always done by an "approved supplier". We have had to cancel engagements with suppliers who I have thoroughly enjoyed working with, and whose results were nothing short of spectacular.[ß] And all of this because certain regulators had their rules written by an established auditing firm who managed to make sure that only a select few (less than 5 globally) companies can ever perform their required assessments. Regulatory capture through security auditing companies should deserve its own circle of Hell. ß: Have you ever had a pentest report you could forward directly to your engineering teams and be certain that they understood the contents accurately? I have. But I can't use the supplier anymore, because they do not enjoy a royal charter.

zxcmx 6y ago

Not just languages - Generally if the scanner isn't aware of the specific framework you're using it's unable to find entrypoints or reliably figure out sources / sinks and filters. Many modern frameworks have one or more of: dynamic configuration, compile time annotations, reflection, IoC etc which make it very hard for a "first principles" scanner to make sense of what can actually happen at runtime.

pc86 6y ago
What exactly is a "'first principles' scanner?"
- zxcmx 6y ago
  Sorry, was perhaps poorly worded. By "first principles" I meant a static analyser working using only an understanding of the language (say, Java) and the application source code, looking over the ast/cfg etc using only built-in language rules. You get much better (read: maybe useful) results if you happen to have a "rule pack" for the specific framework you're using which provides specific hints on sources + sinks, "gotchas" and how things are wired together. As a somewhat obsolete example, I would not expect a sast working only from "first principles" to be able to do anything useful with a Spring XML configuration file. On the whole my experience is that these things work very well on certain types of codebases - e.g. naive PHP they can "go to town" because of the huge footgun surface and fairly direct control and data transfer. Stuff with lots of "magical" framework features and indirection (where even a human reviewer can often have trouble finding the implementation from the interface being invoked) they often silently fail to do anything useful.
  - bostik 6y ago
    > static analyser working using only an understanding of the language [...] looking over the ast The part I quoted is called "tainting". Or if you are more academically inclined, "data flow analysis". When it works right, it is an incredible force multiplier in security. You get detailed, actionable and above all helpful error messages directly from CI, because as a static analysis tool it's pretty fast and can be made part of the common linting pass. As you hinted, it does require a suitable config setup and/or code annotations to mark sources and sinks. And when it does work, it can eliminate an entire class of vulnerabilities - good taint analysis will prevent you from even accidentally using user-supplied data in anything that involves relaying, storing or displaying information. The downside is, when it doesn't work, it's a source of unhappiness. Debugging a taint failure because the AST analysis gives a false positive can be infuriating.
- schwag09 6y ago
  I suspect this means a scanner that can derive all necessary information without any configuration. For example, consider a scanner looking for API endpoint authorization inconsistencies. Does the scanner need you to describe your authorization scheme, or can you simply run the thing and it figures it out? This can be easy or hard depending on how bespoke your application is. If you're using something like Ruby on Rails, then there's a paved road that a scanner can preconfigure to understand your application. If you're using a homegrown authorization framework, then a scanner will likely have a hard time understanding your application and will need to be configured.
airza 6y ago
Even Fortify and other tools which have to compile beforehand don't do so well IMO. Someone is going to come up with an actually good SAST and print money...
- tru3_power 6y ago
  I’ve had really good experiences with Fortify so far. It took me a lot of hours to figure out how to write custom rules but once I was able to do so it became super valuable.
Rickasaurus 6y ago
I didn't write about this because it wasn't toward my exact point about FP, but you're totally right. This will be equally as chilling for new web frameworks.

clumsysmurf 6y ago

This. I work on an Android app written in Java. We were thinking about migrating to Kotlin years ago, but our blessed tool - Fortify - only just released a technical preview for Kotlin 1.3.50 in Fortify 20.1.0, and already Kotlin 1.4.10 is out.

Mountain_Skies 6y ago
My impression is that the current owners of Fortify lack the resources that HPE had and the product is suffering as a result. It's best suited for maintaining old code bases.

beh9540 6y ago

This cannot be stated enough. The company I worked for purchased "the leading gartner SAST testing solution" before they even started a project and had a framework chosen. It turned out the SAST solution didn't support the latest major version of the framework we were using at the time, and didn't support the latest version of the language we were using. Even worse, it never seemed to ever report an issue, and took hours to run at times, eating up build minutes. I ran bandit on the code base just for fun one day, and we had four hits and it took 5 minutes to run. It took a while, but I finally convinced the powers that be we were better using the tool we could verify works, rather than trusting the SAST vendor that it worked.

acdha 6y ago
Fortify took almost a decade to support Python 3 and there were some hefty lags on Java updates, where multiple times it was like “do you want to use the supported Java or the one which Fortify can scan?”
chair6 6y ago
waves, original Bandit author here. It could've done more, but pretty cool to see how useful it's been and where it's got to.. OSS is fun.

newera2016 6y ago

Wow. Great. Would you be able to publish your analysis?

Mountain_Skies 6y ago
Unfortunately I cannot as it is not my property. Plus one vendor made my client and myself sign an NDA even for the most basic information. What I can say is to not put too much faith in the Gartner Magic Quadrant for this space. The best thing to do is select a couple of representative projects for your company and get each vendor to let you do a POC and then compare the scan results from each. The process of making this happen will be long and painful as it is managed mostly by sales teams rather than the tech folks. In the end though, it's the best way to know what will work for your particular set of development technologies. The good news is pretty much every vendor was receptive to the idea of doing a POC even if it took a while to get each one set up.

bostik 6y ago

Pray that you never have to enter domains where the security audits and assessments must be always done by an "approved supplier". We have had to cancel engagements with suppliers who I have thoroughly enjoyed working with, and whose results were nothing short of spectacular.[ß] And all of this because certain regulators had their rules written by an established auditing firm who managed to make sure that only a select few (less than 5 globally) companies can ever perform their required assessments. Regulatory capture through security auditing companies should deserve its own circle of Hell. ß: Have you ever had a pentest report you could forward directly to your engineering teams and be certain that they understood the contents accurately? I have. But I can't use the supplier anymore, because they do not enjoy a royal charter.