7 ms·

This is kind of a long story but the short version is that the tooling isn't yet able to differentiate between dependencies which are packaged and deployed with

This is kind of a long story but the short version is that the tooling isn't yet able to differentiate between dependencies which are packaged and deployed with your application vs dependencies which are only used in the build process. The canonical example seems to be regex DOS vulns reported in a test library. What does an "exploit" look like? One of my colleagues writes a bad regex in a test, now the test suite takes an offensively long amount of time to finish. That's not a vulnerability, that's a broken build. Even if they bypassed the automated tests and forced a deployment to production the offending regex would not be included.

d4mi3n 5y ago
While I agree that regex DoS isn't a very useful thing to highlight in a build, there are risks that come with vulnerable code running in CI/CD systems. There's a whole topic in information security and infrastructure hardening that centers around SDLC (secure development lifecycle) and it covers such thrilling topics as: 1. Is the code from your SCM the same as the code that was built and deployed? 2. Has anyone tampered with your build or deployment artifact? 3. Who all actually has access to the boxes that do all this? Can you tell when folks access your build boxes and what has changed on them? 4. Does your CI/CD system have privileged access to secrets or other internal networks? 5. How confident are you that your digital supply chain is reliable and trustworthy? Would you know if someone fed you a poisoned dependency? The list goes on and on. There be dragons here. I'd love to see less focus on regex DoS and more awareness around the fact that build systems need just as much scrutiny and security as the production environments they feed into.
- xyzzy123 5y ago
  People act like tooling that promotes useless busywork is "free security advice". It's actively harmful. Prioritising security activities that aren't useful (fixing "regex dos" in dev deps) takes time and effort that could have been spent on real bugs or product improvement.
  - kkirsche 5y ago
    It’s not actively harmful. No one is forcing you to do anything. If your team sets the standard as no security issues that’s a team workflow issue not this. Information is information. What you do with it is your choice.
    - xyzzy123 5y ago
      It's a problem at scale, Incorrect and expensive to evaluate information, broadcast to tens or hundreds of thousands of people is misinformation and yes, it's worse than no information. By notifying you of "vulnerabilities" which aren't, these tools effectively "amplify bullshit". It wastes the time of your team and in an open-source context also wastes the time of your downstream consumers. Triaging (evaluating the bullshit) takes time. It is often quicker to "fix" the non problem by upgrading but of course this pushes additional version churn on everyone downstream of you. Even ignoring it wastes time as you have to communicate why you're doing it to "helpful" third parties. In an organisational context you're often stuck with security people who don't know any better or an org policy with metrics around this. If the information were categorised correctly at the source, or the tooling were smart enough, or the rules nuanced enough to capture the reality of the situation then the time and effort of every team in the ecosystem could be saved. Unfortunately researchers are incented to produce CVEs regardless of quality, and of course the CVSS score is always calculated "worst case" despite the fact that 90% of the time these issues are in barely used parts of the codebase or exploitable only in unusual configurations. The tooling then makes this poor quality data worse by completely ignoring context. For example, basically every docker scanning tool on the market will report sev 9.8s on "linux kernel vulns" (based on installed headers) or systemd or cron privesc bugs... in a container... based on matching package versions inside the container. It's all just incredibly lazy engineering from scanning tool vendors who afaict don't QC their data feeds at all, and are also incented to maximise "findings" on every scan. I believe there's room for a startup that does "collaborative triage" of security issues to help stem the tide of this, because no vendors seem interested in fixing it.
ilammy 5y ago
> the tooling isn't yet able to differentiate between dependencies which are packaged and deployed with your application vs dependencies which are only used in the build process You can run npm audit --production to audit only the "dependencies" that are what deployed code is using, but not "devDependencies" which are what build and test infrastructure is using. > That's not a vulnerability, that's a broken build. True, but at the same time you shouldn't discount the potential issues in the build toolchain as totally harmless. A classical example is a malicious transitive dependency that steals your signing keys. Or triggers long builds, slowly (or rapidly) burning your money that you pay for CI. The thing is, with every potential vulnerability you should assess whether it actually applies to you. There are countless reasons for which a vulnerability cannot be exploited in your particular situation: you don't even use a feature, you use it with proper safeguards, you don't deploy that code to anywhere that matters, you run it in a sandbox, etc. But doing so is not that easy when you have hundreds of transitive dependencies and npm audit cries wolf daily. I have this little library with exactly one direct dependency and what do I see? found 7 vulnerabilities (4 moderate, 3 critical) in 124 scanned packages This is doable. Now multiply it by 200 and you can't be bothered to review it all and decide whether things apply to you or not. Extreme positions are to either ignore the advisories entirely (“it’s not an issue until I have a real user with a real complaint”), or treat all of them as immediate priority (“npm audit says we have 7 critical vulnerabilities, those must be real and actual, drop whatever you're doing and fix that NAO”). A middle ground is some arbitrary policy, like don't sweat it until we have more than N potential vulnerabilities, or someone happened to read their descriptions and thinks they are particularly nasty. The bad thing here is that the npm audit scale promotes this sort of middle ground position taking without involving actual security people.
- Floegipoky 5y ago
  I stand corrected, thanks! Elegance in simplicity https://github.com/npm/cli/pull/202/files#diff-2a8ed1f0d31e4c759861aa941a2dc7e4965c10d96a79bad6499301a9a0dadbc2R178 https://github.com/npm/cli/pull/202/files#diff-2a8ed1f0d31e4...
seniorsassycat 5y ago
I've used esbuild's metadata file to filter npm audit's report and only warn for packages with files that are required at runtime. It's great at filtering out denial of service in the command line parser of my web server.
Jarwain 5y ago
Wouldn't you want build tooling to be a `devDependency`? And npm audit can be configured to only run on non-devDependencies

d4mi3n 5y ago

While I agree that regex DoS isn't a very useful thing to highlight in a build, there are risks that come with vulnerable code running in CI/CD systems. There's a whole topic in information security and infrastructure hardening that centers around SDLC (secure development lifecycle) and it covers such thrilling topics as: 1. Is the code from your SCM the same as the code that was built and deployed? 2. Has anyone tampered with your build or deployment artifact? 3. Who all actually has access to the boxes that do all this? Can you tell when folks access your build boxes and what has changed on them? 4. Does your CI/CD system have privileged access to secrets or other internal networks? 5. How confident are you that your digital supply chain is reliable and trustworthy? Would you know if someone fed you a poisoned dependency? The list goes on and on. There be dragons here. I'd love to see less focus on regex DoS and more awareness around the fact that build systems need just as much scrutiny and security as the production environments they feed into.

xyzzy123 5y ago
People act like tooling that promotes useless busywork is "free security advice". It's actively harmful. Prioritising security activities that aren't useful (fixing "regex dos" in dev deps) takes time and effort that could have been spent on real bugs or product improvement.
- kkirsche 5y ago
  It’s not actively harmful. No one is forcing you to do anything. If your team sets the standard as no security issues that’s a team workflow issue not this. Information is information. What you do with it is your choice.
  - xyzzy123 5y ago
    It's a problem at scale, Incorrect and expensive to evaluate information, broadcast to tens or hundreds of thousands of people is misinformation and yes, it's worse than no information. By notifying you of "vulnerabilities" which aren't, these tools effectively "amplify bullshit". It wastes the time of your team and in an open-source context also wastes the time of your downstream consumers. Triaging (evaluating the bullshit) takes time. It is often quicker to "fix" the non problem by upgrading but of course this pushes additional version churn on everyone downstream of you. Even ignoring it wastes time as you have to communicate why you're doing it to "helpful" third parties. In an organisational context you're often stuck with security people who don't know any better or an org policy with metrics around this. If the information were categorised correctly at the source, or the tooling were smart enough, or the rules nuanced enough to capture the reality of the situation then the time and effort of every team in the ecosystem could be saved. Unfortunately researchers are incented to produce CVEs regardless of quality, and of course the CVSS score is always calculated "worst case" despite the fact that 90% of the time these issues are in barely used parts of the codebase or exploitable only in unusual configurations. The tooling then makes this poor quality data worse by completely ignoring context. For example, basically every docker scanning tool on the market will report sev 9.8s on "linux kernel vulns" (based on installed headers) or systemd or cron privesc bugs... in a container... based on matching package versions inside the container. It's all just incredibly lazy engineering from scanning tool vendors who afaict don't QC their data feeds at all, and are also incented to maximise "findings" on every scan. I believe there's room for a startup that does "collaborative triage" of security issues to help stem the tide of this, because no vendors seem interested in fixing it.

ilammy 5y ago

> the tooling isn't yet able to differentiate between dependencies which are packaged and deployed with your application vs dependencies which are only used in the build process You can run npm audit --production to audit only the "dependencies" that are what deployed code is using, but not "devDependencies" which are what build and test infrastructure is using. > That's not a vulnerability, that's a broken build. True, but at the same time you shouldn't discount the potential issues in the build toolchain as totally harmless. A classical example is a malicious transitive dependency that steals your signing keys. Or triggers long builds, slowly (or rapidly) burning your money that you pay for CI. The thing is, with every potential vulnerability you should assess whether it actually applies to you. There are countless reasons for which a vulnerability cannot be exploited in your particular situation: you don't even use a feature, you use it with proper safeguards, you don't deploy that code to anywhere that matters, you run it in a sandbox, etc. But doing so is not that easy when you have hundreds of transitive dependencies and npm audit cries wolf daily. I have this little library with exactly one direct dependency and what do I see? found 7 vulnerabilities (4 moderate, 3 critical) in 124 scanned packages This is doable. Now multiply it by 200 and you can't be bothered to review it all and decide whether things apply to you or not. Extreme positions are to either ignore the advisories entirely (“it’s not an issue until I have a real user with a real complaint”), or treat all of them as immediate priority (“npm audit says we have 7 critical vulnerabilities, those must be real and actual, drop whatever you're doing and fix that NAO”). A middle ground is some arbitrary policy, like don't sweat it until we have more than N potential vulnerabilities, or someone happened to read their descriptions and thinks they are particularly nasty. The bad thing here is that the npm audit scale promotes this sort of middle ground position taking without involving actual security people.

Floegipoky 5y ago
I stand corrected, thanks! Elegance in simplicity https://github.com/npm/cli/pull/202/files#diff-2a8ed1f0d31e4c759861aa941a2dc7e4965c10d96a79bad6499301a9a0dadbc2R178 https://github.com/npm/cli/pull/202/files#diff-2a8ed1f0d31e4...

seniorsassycat 5y ago

I've used esbuild's metadata file to filter npm audit's report and only warn for packages with files that are required at runtime. It's great at filtering out denial of service in the command line parser of my web server.

Jarwain 5y ago

Wouldn't you want build tooling to be a `devDependency`? And npm audit can be configured to only run on non-devDependencies