5 ms·

I ran it against a few servers I'm responsible for. I'm not impressed. I got a large number of reports for a Debian system running Apache. These were all old v

by hannob 7y ago

I ran it against a few servers I'm responsible for. I'm not impressed. I got a large number of reports for a Debian system running Apache. These were all old vulnerabilities where fixes were backported to Debian's packages, so these are false positives. Also got a warning for another server about the recent XMSS issue in OpenSSH, which is code that is disabled by default (and disabled on the scanned server). It seems all this tool does is some kind of version matching (i.e. "a CVE has been reported for version x.y.z of software A, so a server running x.y.z is vulnerable"). This is a poor proxy for the actual existence of a vulnerability. It's not a "vulnerability scanner" in a sense that it actually tests for the presence of a vulnerability. (Full disclosure: Just copied over my comment I posted on reddit earlier today)

justin66 7y ago
> It seems all this tool does is some kind of version matching (i.e. "a CVE has been reported for version x.y.z of software A, so a server running x.y.z is vulnerable"). This is a poor proxy for the actual existence of a vulnerability It sounds as though it's illustrative of a problem with either the index the scanner software is referring to or the update methodology the distro relies on. Debian don't really fix the software without incrementing the version number in some fashion, do they?
- zerocrates 7y ago
  It's pretty typical for these "backported" packages to have version numbers that only differ from the original by a suffix, and that suffix often only appears in the package manager. For example, the Debian stable Apache package is currently version 2.4.38-3+deb10u3, but Apache will typically report something less than all that information, like just 2.4.38. If it's really just looking at exposed version numbers, there's plenty of other problems, like servers that are configured to not expose their version numbers.
thaumaturgy 7y ago
This has been my experience with similar tools (mostly openvas / greenbone) and companies peddling vulnerability tests for compliance purposes. There doesn't seem to be a lot of "functional scanning" in this space, like you're talking about. So I agree the false positives are a pain in networks with patched systems but this also seems to be standard behavior.
- ejcx 7y ago
  Yes it’s unfortunately standard for expensive scanners too. Our goal was to make something easy to run and deploy with similar results to something like openvas or Nessus. I personally think accuracy issues are because security vendors don’t want to show that their expensive software found nothing, so they dial down the accuracy. This works using the same methods as a Nessus, banner grabbing and indexing issues. We are going to keep working on this and look for ways to make it more accurate, but I’d say our results with this versus our expensive scanners was pretty much the same.
  - thaumaturgy 7y ago
    Yeah. FWIW those openvas reports really satisfied the C-level itch to have some kind of work product in the form of an official-looking report, and your looks identical. Overall you seem to have built an awesome toolkit to start with. Greenbone is a total pain to get working initially and openvas updates are broken out-of-the-box. If I were still doing this kind of stuff I'd be immediately trying out Flan Scan.
  - jascii 7y ago
    "I personally think accuracy issues are because security vendors don’t want to show that their expensive software found nothing" That might be a bit cynical, not sure I'd call openvas expensive.. I think it comes down to the fact that the damage of a single false negative can far outweigh the cost of the false positives. In the end, these scanners are mostly a tool to help document your security efforts, often for compliance reasons, making you think about and document the choices you make from a security perspective.
    - ejcx 7y ago
      Definitely a little cynical =]. I think the same thing about a lot of 3rd party security consulting. OpenVAS is free, but the big issue we've had with it is the complexity of setting it up and maintaining it. We are a security team, and would rather not spend our time managing servers, especially since we aren't the best people to do that at Cloudflare.
Hitton 7y ago
But what else would you propose? Only thing you could do which would be more accurate is to try to exploit all the issues it found, which isn't really practical, because POCs are not always available and if they are, you would have to build some kind of database with all of them. Maybe it could be combined with Metasploit...
- jascii 7y ago
  Tools like Metasploit can often be destructive, not sure I would want to run them against production by default.. A more reasonable approach would be to set up a lab when needed when one of the less invasive scanners flags a potential issue.
cabaalis 7y ago
I think I'd prefer a tool that identifies points of interest for my team to investigate rather than one that actually tries to execute a hack against my servers, introducing possible instability.
gwittel 7y ago
Similar experience. Many scanners seem to be designed around showing 'value' to management vs real value for engineering and security teams. Another example is Blackduck. Its slow, noisy, has tons of FPs (which you cannot globally ignore), etc. It ends up being a tool users fight against instead of work with. Another caveat -- scanners usually do not account for the current v vulnerable configuration. So while they may flag a vulnerability, its often a potential vulnerability. Its a mixed bag -- at scan time the system may be secure, but a future change may result in a vulnerable system. Unless you're religious about scanning after each build/config change; this could bite you as well. It would be nice to know these minutiae without digging into each CVE report.
- bostik 7y ago
  > scanners usually do not account for the current v vulnerable configuration. So while they may flag a vulnerability, its often a potential vulnerability. This is a straight-up artifact from the fact that the scanners (which essentially have to operate in non-destructive mode) rely on banner grabbing and very basic service behaviour pattern matching. Hence for every service they find, they report - quoting your words here - the potential vulnerability. Which basically is what you want. Trying to intelligently reduce noise would be really cool, but the moment the intelligence suppresses a finding that you WOULD have wanted to know about, you're screwed. It's bad either way, but the worst-case scenarios of the two are asymmetrical. I personally find point-in-time scans a useless exercise. Findings from individual cans are one thing, the difference over time is what really matters.

by hannob 7y ago

justin66 7y ago

> It seems all this tool does is some kind of version matching (i.e. "a CVE has been reported for version x.y.z of software A, so a server running x.y.z is vulnerable"). This is a poor proxy for the actual existence of a vulnerability It sounds as though it's illustrative of a problem with either the index the scanner software is referring to or the update methodology the distro relies on. Debian don't really fix the software without incrementing the version number in some fashion, do they?

zerocrates 7y ago
It's pretty typical for these "backported" packages to have version numbers that only differ from the original by a suffix, and that suffix often only appears in the package manager. For example, the Debian stable Apache package is currently version 2.4.38-3+deb10u3, but Apache will typically report something less than all that information, like just 2.4.38. If it's really just looking at exposed version numbers, there's plenty of other problems, like servers that are configured to not expose their version numbers.

thaumaturgy 7y ago

This has been my experience with similar tools (mostly openvas / greenbone) and companies peddling vulnerability tests for compliance purposes. There doesn't seem to be a lot of "functional scanning" in this space, like you're talking about. So I agree the false positives are a pain in networks with patched systems but this also seems to be standard behavior.

ejcx 7y ago
Yes it’s unfortunately standard for expensive scanners too. Our goal was to make something easy to run and deploy with similar results to something like openvas or Nessus. I personally think accuracy issues are because security vendors don’t want to show that their expensive software found nothing, so they dial down the accuracy. This works using the same methods as a Nessus, banner grabbing and indexing issues. We are going to keep working on this and look for ways to make it more accurate, but I’d say our results with this versus our expensive scanners was pretty much the same.
- thaumaturgy 7y ago
  Yeah. FWIW those openvas reports really satisfied the C-level itch to have some kind of work product in the form of an official-looking report, and your looks identical. Overall you seem to have built an awesome toolkit to start with. Greenbone is a total pain to get working initially and openvas updates are broken out-of-the-box. If I were still doing this kind of stuff I'd be immediately trying out Flan Scan.
- jascii 7y ago
  "I personally think accuracy issues are because security vendors don’t want to show that their expensive software found nothing" That might be a bit cynical, not sure I'd call openvas expensive.. I think it comes down to the fact that the damage of a single false negative can far outweigh the cost of the false positives. In the end, these scanners are mostly a tool to help document your security efforts, often for compliance reasons, making you think about and document the choices you make from a security perspective.
  - ejcx 7y ago
    Definitely a little cynical =]. I think the same thing about a lot of 3rd party security consulting. OpenVAS is free, but the big issue we've had with it is the complexity of setting it up and maintaining it. We are a security team, and would rather not spend our time managing servers, especially since we aren't the best people to do that at Cloudflare.

Hitton 7y ago

But what else would you propose? Only thing you could do which would be more accurate is to try to exploit all the issues it found, which isn't really practical, because POCs are not always available and if they are, you would have to build some kind of database with all of them. Maybe it could be combined with Metasploit...

jascii 7y ago
Tools like Metasploit can often be destructive, not sure I would want to run them against production by default.. A more reasonable approach would be to set up a lab when needed when one of the less invasive scanners flags a potential issue.

cabaalis 7y ago

I think I'd prefer a tool that identifies points of interest for my team to investigate rather than one that actually tries to execute a hack against my servers, introducing possible instability.

gwittel 7y ago

Similar experience. Many scanners seem to be designed around showing 'value' to management vs real value for engineering and security teams. Another example is Blackduck. Its slow, noisy, has tons of FPs (which you cannot globally ignore), etc. It ends up being a tool users fight against instead of work with. Another caveat -- scanners usually do not account for the current v vulnerable configuration. So while they may flag a vulnerability, its often a potential vulnerability. Its a mixed bag -- at scan time the system may be secure, but a future change may result in a vulnerable system. Unless you're religious about scanning after each build/config change; this could bite you as well. It would be nice to know these minutiae without digging into each CVE report.

bostik 7y ago
> scanners usually do not account for the current v vulnerable configuration. So while they may flag a vulnerability, its often a potential vulnerability. This is a straight-up artifact from the fact that the scanners (which essentially have to operate in non-destructive mode) rely on banner grabbing and very basic service behaviour pattern matching. Hence for every service they find, they report - quoting your words here - the potential vulnerability. Which basically is what you want. Trying to intelligently reduce noise would be really cool, but the moment the intelligence suppresses a finding that you WOULD have wanted to know about, you're screwed. It's bad either way, but the worst-case scenarios of the two are asymmetrical. I personally find point-in-time scans a useless exercise. Findings from individual cans are one thing, the difference over time is what really matters.