22 ms·

Npm Audit: broken by design?

• Macha 5y ago
  This is a general problem with many security scanning tools, and when a security team is empowered to give deadlines to fix any issue they report, leads to much frustration and poor relations in teams. Imagine if you had 3 days to fix the regex DoS issue shown there, screw your release freeze and your current sprint plans, and you have the real working environment in some companies. I've also heard reports of people trying to claim bug bounties for similar reports, or security vendors that run automated tools that detect for issues of similar (lack of) value.
  • _jal 5y ago
    Among other things, it illustrates the insufficiency of a single numeric value for assessing 'badness', which in turn masks a management issue. CVEs try to supplement with flags for remotely exploitable, etc. but it still intentionally leaves a lot of space for interpretation, which is necessary for any normal enterprise. The problem comes in when analysts (or their managers) interpret inflexibly, without appropriate technical context, or without understanding business impact and tradeoffs. If you look at the workflow, it is hard to close the loop from engineering or IT back to security. We need a set of controls for secops departments' output relevance and departmental interoperation.
  • woutr_be 5y ago
    This is actively going on where I work. Granted, it’s a financial company, so they take security pretty serious. During our last release, we had to go through 3 different teams, all doing different security scans. One if them is scanning all your dependencies, and its so frustrating. Because that team obviously has no idea what any of the dependencies do or how they’re being used. All they see is a red flag, and tell you to fix it. Good luck when they tell you this days before a release, and a week after the code is frozen. They’ll just block your release without a second thought. Funnily, in our last release, some of our NPM packages were flagged as a risk, obviously without explanation. The thing was, these packages where dependencies of another package. Obviously we can’t go around updating open source code, just because the security team in our company told us.
    • charcircuit 5y ago
      >Obviously we can’t go around updating open source code, just because the security team in our company told us. This isn't obvious to me. Most open source projects accept contribution from others.
      • woutr_be 5y ago
        > This isn't obvious to me. Most open source projects accept contribution from others. Of course they do, and I'm more than happy to help with open source projects. My point was that, we can't do it, just because a security review at my company says so. It's not just as simple as updating the version of the affected package, there's also testing involved, potentially fixing issues due to using a later version. This would almost be a full-time job.
      • Macha 5y ago
        Who says it's important to that maintainer that their project used as a build time dependency has a vulnerability if provided untrusted user input? What if it requires major upgrades of their framework or toolchain they don't want someone doing drive by? What if they require a CLA that your legal team won't let you sign?
• gampleman 5y ago
  We basically run it in CI... and then allow it to fail without failing the build. ¯\_(ツ)_/¯ It seems like a lot of this has been designed for Node (backend) development, whilst ignoring the fact that NPM is probably used more heavily for front-end development at this point.
  • mmis1000 5y ago
    Yep, some vulnerable package isn't even in the compile output. How a dev server that only binds to 127.0.0.1 a serious DOS problem? Who on the earth will want to DOS that?
    • deleted 5y ago
      [deleted]
• klodolph 5y ago
  Difficult problem to solve. It seems like the reasonable solution here is to have auditing integrated with the build system—but there are so many wacky build systems for JavaScript.
• littlecranky67 5y ago
  I've been saying this for years, especially to useless "prototype polution" notices being reported by npm audit. In a project where this "pollution" only happens on the nodejs side in our build tools, they are meaningless if our output is a browser JS bundle.
• quintex 5y ago
  As someone who only had just gotten into front end programming after years of backend work, npm has been a nightmare. I haven't experienced the same level of frustration with other package managers (pip, cargo, go mod, etc) as I have with npm. Is yarn the better option? What is our path forward?
  • protonimitate 5y ago
    I've been having a good experience with pnpm lately. It's not a silver bullet but it addresses some pain points with npm/yarn.
  • nih0 5y ago
    same xd, npm even managed to make me appreciate maven and gradle
    • vbezhenar 5y ago
      I think that there's difference between Java and JS landscapes. Java has rich standard library. Some would say - too rich. But anyway - very few people would need customized collections, standard library covers all the needs. And where it does not cover all the needs, there are few commonly accepted additions like apache-commons or google guava. So that one solves `leftpad`-like issues. Another difference is that Java is old. Most needs were covered by many libraries and few libraries survived which are good enough. It's again some commonly accepted wisdom, so you don't really need to search for many options. You have one good enough option that you'll use and move on. So it's not about maven/gradle vs npm, it's more about ecosystems. I don't think that porting maven to JS world would change anything.
  • SahAssar 5y ago
    I think the problem is not really NPM it's the way that mainstream frontend development requires very wide and deep dependencies. If you stick to using smaller frameworks and libraries without heavy build tools then it's not much of a problem.
  • mrweasel 5y ago
    I basically do no frontend development, partly due to tools npm and the current frameworks, I simply cannot wrap my head around it. I do help run a few modern javascript application however. Even a minimal app will pull in 1000+ dependecies, and I think that’s the problem. It simply don’t happen in Python, Go or Rust (or even Java) because the languages comes with a rich standard library. Javascript comes with just the basics, everything else is a dependency. It’s not uncommon for people to audit their dependencies in Python or Go, but you pull in maybe 10 or 20. A basic Javascript app easily pull in 100 times that, how are you suppose to deal with that?
• richardwhiuk 5y ago
  I think they are just saying that context matters in security vulnerabilities, and npm audit doesn't have that context. Well yes, correct, well done. By this metric, every security tool ever written is probably pointless.
  • danabramov 5y ago
    Note that `npm audit` runs _during every install_ so people who use it don't necessarily consciously understand what's happening. Many of them are beginners and have never used a security tool before (or even want to use it).
• tutuca 5y ago
  Excellent rundown of many problems I've encountered in recent versions of npm and node. I've never ever got `audit fix` or `audit fix --force` to solve any of the mentioned vulnerabilities. Ever. I even relied on downloading every dependency one by one to find that there where other offending packages. I just gave up. It's really useless and deceptive.
• cphoover 5y ago
  Vulnerabilities are just code paths that can behave in unexpected ways and be abused.... I'm not sure the author's point, that development configuration could not hide malicious code? Why not? All NPM does is scan to the dependency graph for vulnerability reports, it doesn't make any assessment of your consuming application's use-case. If you don't find this useful that is fine, don't use it. I think it's totally worthwhile to figure out which tools rely on insecure dependencies. Also looks like you can specify npm to ignore dev dependencies: > Any packages in the tree that do not have a version field in their package.json file will be ignored. If any --omit options are specified (either via the --omit config, or one of the shorthands such as --production, --only=dev, and so on), then packages will be omitted from the submitted payload as appropriate.
  • danabramov 5y ago
    >I'm not sure the author's point, that development configuration could not hide malicious code? Why not? Quite the opposite! Quoting the article: As any security professional will tell you, development dependencies actually are an attack vector, and perhaps one of the most dangerous ones because it’s so hard to detect and the code runs with high trust assumptions. This is why the situation is so bad in particular: any real issue gets buried below dozens of non-issues that npm audit is training people and maintainers to ignore. It’s only a matter of time until this happens. My point is that in the sea of non-issues, real issues are easy to miss and ignore. >If you don't find this useful that is fine, don't use it. You can't "not use it" because it's literally the default behavior built into `npm install` now. Of course there are ways to opt out, but this doesn't alleviate the confusion.
    • cphoover 5y ago
      All it does is look to see if either your direct dependencies or descendant dependencies exist in the advisory database.... It seems like a simple algorithm that works pretty well. Perhaps ignoring certain dependencies makes sense, via an ignore list. I just find the title "NPM is broken by design" to be a little hyperbolic, when it seems like the complaint is that it's tedious removing all the low-quality dependencies from your project. node security/npm-audit has at least increased the conversation around security for many around the npm ecosystem, where there wasn't much-if-any discussion prior. I think they deserve credit for this. EDIT: I'm not sure why I'm being downvoted.
• z5h 5y ago
  Seems to me that if you don't have a pure functional language with tree-shaking, and only including existing (non-generated) code, you won't know if any (3rd party) code in particular will be called in compiled output, and you'll just have to guess and err on the side of paranoia Is there a fix?
• mbesto 5y ago
  > this “vulnerability” is absurd in this context Yes that's exactly the point. The audit tool has no awareness of the context and nor do the people who create severities. If severities were absolute then there would be no reason for anyone to review them. You would simply upgrade your libraries and be done with it, but that can't always be achieved nor may make business sense. I do agree with the author's note about providing a better way to provide feedback on severity reviews. npm audit is better than no npm audit...telling people it's broken by design is going to discourage them from using it completely. smh.
  • raziel2p 5y ago
    Exactly. Why does it matter if it's absurd in this context? Just upgrade it to be on the safe side. Asking vulnerability databases to judge whether vulnerabilities are safe in devDependencies or not is a ridiculous idea, even more so when you consider that the line between static and dynamic have gone blurry long ago.
    • danabramov 5y ago
      >Just upgrade it to be on the safe side. A big part of the problem is there is no reliably way to "just upgrade it" today in npm: - `npm audit fix --force`, which is supposed to do that, is buggy and doesn't work - There is no way to override a transitive dependency with npm (there is with Yarn though, so hopefully this feature will come to npm soon) - Sometimes the fix in transitive dependency _also_ includes breaking changes (e.g. because it wasn't backported), and so updating it subtly breaks the logic >Asking vulnerability databases to judge whether vulnerabilities are safe in devDependencies or not is a ridiculous idea I don't think databases can do it, but what I'd like to be able to do is to be able to provide advisory that the way _my package_ uses a concrete transitive dependency is not affected by that vulnerability. Because as the package owner I _do_ have that context. I understand there may be significant issues with this approach though!
      • cphoover 5y ago
        > Because as the package owner I _do_ have that context. I understand there may be significant issues with this approach though! But what if one of your contributors slips in a merge that uses the vulnerable code path of your dependency... Does this "not affected" marker still exist, and now you have vulnerable code? Does it disappear with each version? What if someone maliciously adds a "not affected" marker? To a package they intend to exploit? Edit: Again why the heck am I being downvoted?
• gitgud 5y ago
  Agreed, you shouldn't see low risk security warnings by default, they're more appropriate for larger projects, with something to lose. The [1] yarn package manager is much nicer to work with in many more ways. [1] https://classic.yarnpkg.com/en/ https://classic.yarnpkg.com/en/
  • unanswered 5y ago
    TFA identifies "high risk" false positives too, so this response doesn't seem to have anything to do with them problem as stated by TFA.
• nonameiguess 5y ago
  It isn't broken by design. I sympathize and understand it's annoying to have to comb through false positives and mark them as such, but until a level of AI we're nowhere near being near exists, you can't automate this process, so the only alternative is to ignore all vulnerabilities. Complaining about this is misunderstanding the asymmetrical costs of different types of statistical error. Vulnerability scanners are sensitive by default because the cost of lots of false positives is annoyed developers who have to slow down their delivery cadence. The cost of false negatives is anything from compromising user data to losing your company to bringing down the power grid of a major city depending on what the application is. Which of those is a higher cost? npm can't possibly know the answer to that, so it has to default to assuming security actually matters to you. If it doesn't, your local policies can be more lax, but don't expect the tool to change for you.
  • bogota 5y ago
    That is not the cost of false positives. The cost of false positives is ignoring it completely. I have had jobs where we just block the security scanners because they won’t listen to any feedback about why what they are scanning was intentionally setup for the purpose of testing vulnerabilities on an internal only network. Additionally at other jobs security tickets just start to get ignored because they send too many tickets that do not matter. I feel the security field likes to ignore most feedback and play holier than though. At all companies i worked at i have only had one good security team that worked with people instead of just throwing things over the wall.
    • cratermoon 5y ago
      I agree about the false positive problem. Boy who cried wolf and all. I've also worked with security vendors who offer to run "free" vulnerability scans for you, and to absolutely nobody's surprise, they find vulnerabilities that just happen to be the ones that they can fix, if you buy what they are selling. Still, your example is problematic. Beware the "internal-only network". Such a thing has mostly lost meaning today, and it was never much more than a picket fence anyway. "All devices must be capable of maintaining their security policy on an un-trusted network." https://collaboration.opengroup.org/jericho/commandments_v1.2.pdf https://collaboration.opengroup.org/jericho/commandments_v1....
• dale_glass 5y ago
  It's a tricky problem to solve. Ideally you'd want to show only relevant alerts, but... how? You'd need to know which kind of errors are relevant for a particular project, but that'd require solving the halting problem. This is made much worse by that it's JS. Some libraries have an enormous complexity and attack surface. Take a database interface -- there probably is a vulnerability in some obscure corner the typical person may not even know exists. I think though at the very least some improvement could be made by better priorization and categorization. DoS by exploiting a regex parser isn't that big of a deal if your project is just getting started, but an exploit allowing arbitrary code execution would still be.
  • minxomat 5y ago
    You don't have to go all out doing exhaustive dynamic analysis. Data flow analysis gets rid of 99% of the most popular bugs (injection, validation, defaults, etc.). GitHub CodeQL can do this, and produces much better results than any static analysis tool I've ever used. Feeding this data back into npm (owned by GH) is just the next step.
  • scottfr 5y ago
    You just need a way for a package maintainer to flag a vulnerability in a dependency as a non-issue that does not affect that package's use of the dependency. In Dan's twitter thread, he calls this out as a viable solution.
• aerojoe23 5y ago
  It sounds like npm needs a mitigated and irrelevant flag, these flags should include an explanation field. Security teams would also have to accept this as a solved/fix status. For projects you own you'd have to flag each dependency path though, because for example, one dependency may not have the input for the regex exposed to the end user, while another dependency could. Maintainers of libraries should also flag the security issues, and an issue with these two statuses on them wouldn't be raised by default. Options should be available to list them though for auditing. For more security critical teams/projects, a per project setting to alert about any issues the maintainers have flagged irrelevant or mitigated and you'd have to accept them before it it would stop alerting about them.
• lxe 5y ago
  I agree with the author. Just like them, I only write code without any bugs that can be affected by the "vulnerabilities". I also never commit to upstream so others won't be able to edit my code. All my projects only run on my machine (which is of course also absolutely exploit-free and it's not connected to the internet).
  • cratermoon 5y ago
    You're being snarky, which is fine, but the author addresses that. If you're compromised, the attacker is not going to dig through your development folder to inject a regex that makes your build slow. They'll exploit privilege escalation bugs to install a bitcoin miner, ransomware, a DDOS bot node, or use some other vulnerability to grab and/or exploit your secrets. They'll do it the most direct way possible, not via some half-broken regex parser.
• esens 5y ago
  I found that much of the underlying cause is those mass reporting regex denial of services as being high severity bugs. So many people are reporting these in tons of different projects: https://github.com/search?q=regex+denial+of+service&type=issues https://github.com/search?q=regex+denial+of+service&type=iss... Anyhow it is just annoying and they broke NPM Audit based on these reports. It is good to fix all possible bugs, but many of these are not anywhere close to the level of bad that the reports are making them to be. But maybe this is needed to just get rid of these issues in genera? So a wave of regex vulnerability reports and then we build this type of checking into prettier or similar and we do not have these in the future? EDIT: It appears there as a project that found 100s of CVE reported Regex vulnerabilities in npm projects -- this is maybe one of the sources of mass reports. See the bottom of this resume: https://yetingli.github.io https://yetingli.github.io
  • danabramov 5y ago
    Hmm. This actually sounds really plausible. I wonder if there's any way to check that.
  • cratermoon 5y ago
    This kind of nonsense really goes back to the broken CVE process. https://opensourcesecurity.io/2021/03/30/its-time-to-fix-cve/ https://opensourcesecurity.io/2021/03/30/its-time-to-fix-cve... Linux kernel maintainer Greg Kroah-Hartman has a similar opinion. https://github.com/gregkh/presentation-cve-is-dead/blob/master/cve-linux-kernel.pdf https://github.com/gregkh/presentation-cve-is-dead/blob/mast... Edit: LWN mention https://lwn.net/Articles/801157/ https://lwn.net/Articles/801157/
    • zinekeller 5y ago
      The view of SQLite developers on CVEs is also dim: https://www.sqlite.org/cves.html https://www.sqlite.org/cves.html
      • cratermoon 5y ago
        Beautifully succinct. This quote: "Grey-hat hackers are rewarded based on the number and severity of CVEs that they write. This results in a proliferation of CVEs that have minor impact, or no impact at all, but which make exaggerated impact claims." Alignment of incentives is messed up. Goodhart-Strathern's and Campbell's laws apply.
• esprehn 5y ago
  Dan isn't the first person to notice: https://www.voitanos.io/blog/don-t-be-alarmed-by-vulnerabilities-after-running-npm-install/ https://www.voitanos.io/blog/don-t-be-alarmed-by-vulnerabili... We disable [1] audit entirely because it's not a good default behavior within a monorepo. It spams the hundreds of developers with the list of "vulnerabilities" on every install, but only a few folks should really be upgrading packages. We then run audit in non-blocking CI and track the total number of issues and mostly focus on critical ones. [1] https://docs.npmjs.com/cli/v7/using-npm/config#audit https://docs.npmjs.com/cli/v7/using-npm/config#audit
  • x0x0 5y ago
    We also wanted to use npm audit in our CI process so, instead of humans being careful, we could assert any known CVE stops a staging or prod deploy. Very annoyingly, npm audit doesn't have ignore functionality, at least when I was last forced to use it. I had to hack something together with bash scripts. For vulnerabilities that we determined weren't an issue ever (vuln in frontend framework we didn't use), or weren't high priority enough to P0 through, we needed some way to ignore either permanently or temporarily specific vulnerabilities. Given the enormous dependency sets eg react create, you'd think the tools would be better at managing them.
    • dane-pgp 5y ago
      > Very annoyingly, npm audit doesn't have ignore functionality I can't believe no one here has mentioned the fantastic tool "better-npm-audit" which can be included as an npm dependency[0] and lets you add specific vulnerabilities to an ignore list. The ignore list is actually a JSON config file stored alongside package.json in the repo, so only one developer ever needs to see the npm audit warning and can mute it for everyone else (after getting their PR approved). Even better, the config file lets you specify an expiry date for each entry in the ignore list, and provide a note, such as a link to the upstream issue being worked on, so that you can periodically be reminded to go back and check if a new version is available which can give more confidence that your code really isn't affected. I think that developers might have to be instructed to use the "--no-audit" option to "npm install" if they don't want to see the (false positive) warnings that the default behaviour produces, and that's a bad habit to learn if not all projects they work on are using "better-npm-audit". I don't know if there is a way to make that option the default on a per-project basis. [0] https://www.npmjs.com/package/better-npm-audit https://www.npmjs.com/package/better-npm-audit
• mcintyre1994 5y ago
  It doesn't solve everything, but we use npm-audit-resolver[0] and it's.. workable. It presents you vulnerabilities, offers to fix (upgrade the nested dependency) if a version exists that meets all the constraints, and gives you an option to ignore for a week/month/forever if no fix exists. Those decisions (including fixes, which is a bit silly) get recorded in a JSON file in source control. For each group of ignores we add a link to the relevant Github issue where it's been reported, so if the ignore time we chose expires we can quickly go and see what the status is. There are still problems: - The decisions file gets unweildy, mainly because every time it fixes something it writes to the file. You probably only care about ignores. It's also append only, though you could manually clear it down sometimes. - It always defaults to fixing at the deepest level, which is.. not ideal for NPM. On my machine (a not very old Macbook Pro) NPM simply can't update a dependency 20 layers deep in the tree, ie `npm update nested-dependency-from-hell --depth 20` will eventually time out and won't fix anything. So you have to manually crawl up tree yourself and find the thing that can be updated - or just ignore it until the thing right at the top of the tree gets updated. I'm not surprised to see Dan posting this though. I agree with everything he said, so I don't mean this as an attack, but a lot of the time the thing at the top of the tree we're waiting for an update on is create-react-app. It must be incredibly annoying how many Github issues get opened on that repo every time there's a new NPM advisory on some 20-dependencies-deep parser it uses for something or other. I do like the suggested fix that a maintainer can use their knowledge of the specific usage to say the vulnerability doesn't apply. Often in these threads there's a perfectly good explanation of why it isn't a real issue, and then people come back with "Okay but can you please update it anyway because I'm forced to audit and my security team/CI are yelling at me". [0] https://www.npmjs.com/package/npm-audit-resolver https://www.npmjs.com/package/npm-audit-resolver
  • deleted 5y ago
    [deleted]
• johndoe42377 5y ago
  I am Jack's absolute lack of surprise.
• _fob_ 5y ago
  Indeed, npm is not aware of the context of the vulnerabilities. That does not invalidate them, however, and mean they should be hidden. I've worked in offensive security for quite long enough (been a dev for 10+ years before) to tell you that your context, or the most common one, isn't all that exist and someone's going use it the way it makes the app vulnerable. Based on the article's example, someone's going to build an app that builds an app. A vulnerability is a vulnerability, whether it applies to your context or not. A metaphor might be: "The passenger door is broken on my car, but I'm the only one to use it". Seriously, who, in their right mind, is going to argue that the door isn't broken? - If your dependencies have security vulnerabilities, apply the updates. - If you cannot update because there's no fix available, let your org or you assess the risk and go from there. - If you cannot update because it breaks your app, {find a replacement, fix it yourself, let your org or you assess the risk and from from there}. A sensible org has a process that freezes releases until known security issues are fixed. Freezes can also be opposed by devs and are evaluated on a case-by-case basis (because sometimes they are not relevant to the product, or someone steps up to take the blame for incidents and the org agrees). We might not like it because it disturbs the "flow", but it's just part of the engineering process. More to the point though, why not take this opportunity to teach newcomers how to code properly, pick well-engineered and -written programs, and handle this vulnerability management process altogether? In any case, I hope newcoming-dev is not going to push to prod anytime soon. ;) edit: formatting
  • three14 5y ago
    A lot of these cases that npm reports are denial of service vulnerabilities (and marked high risk!). I just tried it on a project I have, and 11 out of 15 are DOS vulnerabilities in code that I run locally. When the normal user is using a project only locally, and the issue is DOS, it's hard to argue "but maybe someone will eventually put it online" and therefore I need to drop what I'm doing and patch my dependencies. (Yes, sometimes that would be the only way to satisfy npm, since the semver rules prevent it from fixing things automatically.)
  • bscphil 5y ago
    > A metaphor might be: "The passenger door is broken on my car, but I'm the only one to use it". Seriously, who, in their right mind, is going to argue that the door isn't broken? I think a better metaphor might be, the button on your key fob that opens the trunk doesn't work, so you have to open the trunk using a physical lever. Every time you start the car, a loud warning siren sounds, and a red message appears on the dashboard to tell you that there is a "high impact" problem with your car, and you need to take it to be serviced. If you were merely the owner of the car, and other people also had to drive it, you might understandably be the target of several complaints about this "high impact" problem.
• quaffapint 5y ago
  As mentioned in the article we ran into the same overload and just ended up running `npm audit --production` as part of our CI pipeline since that's what would be going out.
• est31 5y ago
  I think this is a really great article and highlights an important weakness that modern security tools in this context have: they don't distinguish between vulnerabilities that can be triggered by malicious developer code, and vulnerabilities that can be triggered by malicious users/websites. For a proper assessment, such differences need to be encoded in the security advisory, and the audit tool needs to analyze if the code is called at run time or build time, and then act accordingly.
• jgerrish 5y ago
  Correct me if I'm wrong. But, one of the goals in software engineering right now is reproducible builds. This means building from source. And of course we'll want to automate that. We've already made inroads with CI So, correct me if I'm wrong, these are still vulnerabilities. Tragedy of the commons stuff. This article might be honest. But I hope in the future we don't need devil's advocate arguments.
  • jgerrish 5y ago
    Still, the original post might inspire realignment of bug fixing incentives.
• gentleman11 5y ago
  npm is a bit nuts on its own. I started learning react this year and the course I'm taking had me install that create react app module or similar. It dragged in 1700 dependencies, and the folder for a hello world app was almost 90mb iirc. How can you possibly pretend you have any control over your app or it’s security in that situation?
  • ratww 5y ago
    Yeah, that’s the biggest issue, IMO. Even a simple app with only React as a production requirement will have dozens of issues a month. There are some packages that don’t have as many dependencies such as Typescript or Prettier, but that’s not enough, since the most popular bundlers have hundreds of of dependencies. No matter how careful you are, you get flooded by security issues.
  • TechBro8615 5y ago
    The reason it’s so big is because it’s your build chain. I’ve always found this criticism of JS annoying – you’ve also got to pull in a few hundred MB of tooling for C++, Java, or Rust — it’s just that in the JS case, the “compiler” is usually per project.
    • username90 5y ago
      Other libraries are usually self contained though, unlike JS libraries which contains dependencies and then those dependencies has dependencies etc. Probably has to do with JS lacking a standard library so it is a pain to do anything without including dependencies.
    • hsbauauvhabzb 5y ago
      That’s hardly true. Some standard libs are smaller, but in most other languages I can think of there isn’t the complete dependency hell. I want to install one package, the 50 dependencies or their ancestors create an audit he’ll (is the package secure, so I trust the developer to not inject malware, abandon the package, or add more dependencies), etc.
• AtNightWeCode 5y ago
  This is how it works by design.
• akoumjian 5y ago
  The interaction is not the worse thing about npm audit. The security model of the tool has a big hole depending on how you use it: https://mulch.dev/blog/CVE-2020-5252-python-safety-vuln/ https://mulch.dev/blog/CVE-2020-5252-python-safety-vuln/ In essence, if you are scanning an environment that is already compromised, `npm audit` results can't be relied upon if you are running it in the same environment. It should be self-evident but I'm sure plenty of people use the tool this way.
• nalekberov 5y ago
  > It makes beginners miserable Can we please tell beginners not to start programming with node.js? Teaching beginners to start with “go-to” technologies became industry standard already as it helps corporations to become more and more monopolist and dictate new industry standards.
• preinheimer 5y ago
  Honestly, this hit me. I'm not a react developer, I was experimenting with it for a new project. I finished the tic-tac-toe tutorial, then tried to throw bootstrap on top to build from there. It told me there was 97 vulnerabilities (85 moderate, 12 high)... I just deleted the directory and went back to vanilla JS. This is a fun side project, I don't need that. My tweet about it: https://twitter.com/preinheimer/status/1402785757962592256 https://twitter.com/preinheimer/status/1402785757962592256
  • jmull 5y ago
    FYI, it's probably not react-bootstrap or bootstrap or any of their dependencies being flagged. It's some of the other ~2K packages already installed.
  • alerighi 5y ago
    That is exactly the point of the article. Every JS developer knows that these numbers are stupid and doesn't look at them. However a beginner that doesn't know what impact they have of course is scared if when installing a library it tells you that there are all that vulnerabilities.
    • IggleSniggle 5y ago
      As a beginner _just to npm_ I can imagine getting totally freaked out and worried that my whole system was _potentially_ compromised after seeing a “Critical” vulnerability reported as installed on my system. After all, npm can execute any script with the users permissions on install…except often (compared to bash) it’s less easily inspected due to the common use of nested dependencies! I, too, would delete my node_modules, and if I even wanted to move forward at that point, would probably waste at least half a day looking up the Critical vulns and discovering that they are probably not at all critical in my particular scenario. Like not at all for the 99.99% use case. After experiencing something like that, it’s just like the article says. “The boy who called wolf.” Really terrible use of the labels “Critical” and “High”. The labels are fine, but the way they are applied is just stupid.
      • 5y ago
• jrochkind1 5y ago
  > Inline all dependencies during publish… From a maintainer’s point of view, the upsides are clear: you get faster boot time, smaller downloads, and — as a nice bonus — no bogus vulnerability reports from your users. And when you inlined a version that later really does have a vulnerability, it is not easily flagged or fixed by your consumers. The tension between "upgrades (especially of indirect dependencies) might break" and "upgrades (especially of indirect dependencies) might be necessary to fix bugs or patch security vulnerabilities" is real. There is no magic bullet to get around it. There are practices to try to balance it -- which generally involve ecosystem-wide commitment to backwards compatibility, reflected in semantic versioning (and minimizing major releases). That npm dependency trees are often insane doesn't help though. I'm not totally sure why they are so insane, but I increasingly think that npm's ability to have multiple versions of same dependency in the dependency tree -- often seen as a huge advantage over other platforms -- is in fact part of the problem. It makes the dependency tree even more insane, and it also accomodates a false belief that it's fine to lock to very specific versions of dependencies -- because it won't prevent other dependencies from co-existing in tree with other conflicting requirements after all. Which then accomodates a false belief that dependency maintainers don't need to worry too much about backwards compatibility, after all consumers can just lock to specific working versions... and now we wind up with insane dependency trees which depend on vulnerable versions in ways that require a whole bunch of releases to resolve.
  • theptip 5y ago
    > I'm not totally sure why they are so insane I think a big part of it is that due to much stronger pressure on bundle size than most other environments, each library tends to be small, so there have to be more to carry the same amount of functionality. Duplicates are certainly a contributing factor as well, and small bundles compound with allowed-duplication to further increase the tree size. I think that small package size also probably makes it harder to require a single version for each dep, since there are going to be more edges in the graph and therefore more relations for library maintainers to keep track of (including what would in other languages be intra-package requirements), so you're more likely to get version incompatibilities.
    • 5y ago
• genezeta 5y ago
  The title is "Broken by Design". Then the author proceeds to explain how the design is actually reasonable and meant to work ok, and how the problem lies in the relevance and context of the vulnerabilities reported. That doesn't seem to be the meaning of "broken by design".
• cphoover 5y ago
  So currently the algorithm is... check (dev)Dependencies and descendent/transient dependencies to see if they exist in a security advisory database if they do, highlight and surface them to the user. What are alternatives? A way to ignore or mark a dependency as safe? Could this be abused if an author can just mark a dependency as safe? Or perhaps, actually analyze syntax with a tool like ESLint (parse -> AST -> validate) to check that dangerous parts of libraries are not in use? This solution comes with it's own complications. Who is authoring these validations? Perhaps there are other strategies I'm not aware of.
• w3news 5y ago
  It also has some false results. Like package x has a vulnerability in version 1.x And you have a private package @company/x with version 1.x. Than npm audit will blame your private package, even if you dont have used the original package x.
• trunnell 5y ago
  A lack of imagination by the author, unfortunately... A DoS on your build machine and dev machine can be indeed be critical issues. Imagine this scenario: Your source code is somehow compromised and attackers slip in rogue code to your production site. It siphons off passwords or other PII. The attackers also take advantage of several of these RegEx DoS vulnerabilities to prevent you from quickly fixing the problem. When you discover the issue, you’ll first see that your build machine is unresponsive, so you can’t just spin a fixed build and re-deploy. You’ll sync your main branch to figure out what is going on, perhaps ready to make a build from your dev machine, but running yarn build hangs. It might take you 1 minute to solve or 5 hours - hard to guess. But every minute you’re delayed is another minute the attacker is siphoning off your production data. npm audit isn’t perfect, but I don’t agree with the author that devDependencies can’t have critical vulnerabilities. Build machines and dev machines are critical infrastructure. Recall the method of attack of SolarWinds [1]. Related: we all trust that the “many eyes” of open source contributors will keep our dependencies relatively clean, but this function is not infinite. There is some threshold of lines of code and rate of change that will outstrip the community’s natural ability to find and fix problems. I wish the npm community was more sensitive to the risks that are inherent in current practices. Efforts to limit dependencies and perhaps somehow tag which versions have completed a security audit (and by whom) would be great to see. [1] https://krebsonsecurity.com/2021/01/solarwinds-what-hit-us-could-hit-others/ https://krebsonsecurity.com/2021/01/solarwinds-what-hit-us-c...
  • askmike 5y ago
    > Your source code is somehow compromised and attackers slip in rogue code to your production site. Really at this point it's too late to do anything else, instead of trying to dos your dev machine he can instead do simpler things like delete your ssh key from the machine. But let's play along: > The attackers also take advantage of several of these RegEx DoS vulnerabilities to prevent you from quickly fixing the problem. When you discover the issue, you’ll first see that your build machine is unresponsive There is nothing any attacker can do with the static files on the server that will trigger and RegEx DoS in your local development. Aside from the fact that you wouldn't download whatever is on the server back to your machine, even if you did it would never trigger such a DoS since (in the examples in the link) these are modules related to running a dev version of a frontend project based on the raw source files. Your scenario is only true when an attacker pwned both your production server and your laptop. A regex DoS is really the last thing you worry about at that stage.
• cerved 5y ago
  the machine doesn't need to be compromised, just check it into source control
• Pxtl 5y ago
  I do wonder if a typed language shouldn't have distinct types for trusted vs untrusted strings and binaries.
• killion 5y ago
  I like his point about `npm audit --production` being a good way to cut down the noise. But Github doesn't seem to take dev dependencies into account when sending security alerts. I get emails about non-issues from them all the time.
• solatic 5y ago
  > Inlining dependencies kind of goes against the whole point of npm I mean, this is why people love language with deep, solid standard libraries. You don't have a situation where a problem in a sub-sub-sub-sub-dependency provokes five different groups of people to all issue an update, one after another. You just upgrade your underlying installation to the latest patch version and continue. Language ecosystems where a few lines of code constitutes a library fundamentally result in you being dependent on a huge number of outside people to cooperate on updates. That's what's broken. Not a tool which tells you that you have out-of-date libraries and, by the by, hooks into CVE databases. OP should stop and consider whether it might be beneficial to inline some of those dependencies before dismissing it out-of-hand. If you never use the dependency in such a way as to present a real security risk... and you don't need feature updates from upstream, i.e. the software is fine as-is when you first incorporated the dependency... then why wouldn't you inline the dependency? If anything, inlining the dependency will allow static code analyzers to point out all the parts of the dependency which you're not using (i.e. dead code) and eliminate it all. That way, even if the dependency were to be discovered to have a security fault, if the faulty code was in a section that you eliminated as dead code... then you don't have a security problem in the first place!
• riho 5y ago
  The the main problem is the fact that this audit happens with no context, and the audit results offer no information about the context an issue applies to either. Every issue should have a clear explanation about why and where it's an issue, and be tagged. Then we'd just need a way to hint npm what context a package will be used in, similarly to what we already do for devDependencies. Also going through an audit result in a CLI isn't really the best experience. I wish I could just click a link and open up the report in a browser to drill down into issues.
  • hsbauauvhabzb 5y ago
    No it’s not. The main problem is the dependency tree hell. If an ancestor version bumps, you should probably version bump too, irrespective of exploitability. Don’t like it? Try using more maintainable dependency trees.
• planb 5y ago
  Funny. I had a discussion about the same topic in context of docker image security scanners last week. 100% agree with the author here!
• thereddaikon 5y ago
  Anyone else read the title to mean the author had audited Npm and found the whole thing was broken by design? Because I've felt that way for years.
• AaronFriel 5y ago
  For those hoping to run npm audit in your CI/CD pipeline, I recommend this tool from IBM: https://github.com/IBM/audit-ci https://github.com/IBM/audit-ci In highly regulated industries, shipping code flagged as having a vuln without a manual approval could be a liability. This wrapper around npm takes an allowlist argument, and our procedure is for an engineer to review the failing build, determine if the vulnerability (ugh, usually regex ddos or prototype pollution) is present in code that runs only at build time with trusted inputs, only on the client which is by definition untrusted, or in our webserver which takes in untrusted input. As long as it's either of the first two, we document it in a commit and comment and redeploy. It's annoying, but it's far better than npm audit forcing a fix.
  • naugtur 5y ago
    Compare with npm-audit-resolver in terms of how the ignores are defined. It's important to not be too vague when ignoring things. Let me plug this as it contains a lot of references https://dev.to/naugtur/do-you-need-help-with-your-npm-audit-3olf https://dev.to/naugtur/do-you-need-help-with-your-npm-audit-... Meanwhile I'll try to get someone from IBM involved in the OpenJSF collab space
• trunnell 5y ago
  How much of the problem here is that npm audit is a little annoying and creates problems unrelated to security? Or do commenters here actually believe that npm audit should treat a DoS of a development machine as a non-vulnerability? (Please tell me it’s the former)
  • scottlamb 5y ago
    > Or do commenters here actually believe that npm audit should treat a DoS of a development machine as a non-vulnerability? I believe that npm audit should treat a DoS of a development machine by a trusted developer as a non-vulnerability. "Code I (or a fellow committer) wrote uses a lot of CPU" isn't a vulnerability. If I care to prevent this, I should run said code within a cgroup with limited resources, not panic about theoretical expense in one part of the codebase while necessarily allowing arbitrary execution elsewhere. "npm audit" is crying wolf, just as the author said. I like the proposal [1] near the end: "If I own an npm package I need to be able to tag a certain transitive vulnerability category as not affecting my usage of that transitive package." This is particularly important for npm given things like create-react-app but would also be a good idea for "cargo audit" and such. [1] https://twitter.com/dan_abramov/status/1412380714012594178 https://twitter.com/dan_abramov/status/1412380714012594178
• seemslegit 5y ago
  I recently implemented my own npm vulnerability audit tool for the CIO department of a major org - it just adds 'vulnerability' in red next to any npm-based project in their spreadsheet.
• phkahler 5y ago
  >> Five false alarms wouldn’t be too bad. >> Unfortunately, there are hundreds. This is primarily a result of the absurd number of dependencies NPM encourages (requires?) people to use. The duplicates are also there in part because of the large number of dependencies and should not be shown more than once by the tool. Stop building projects with an absurdly large dependency tree, this is just one problem that results from it.
  • TechBro8615 5y ago
    I’d imagine that in most projects, the bulk of the dependency is due to dev tooling. I don’t think it’s fair to optimize for small dependency trees when setting up your buildchain – otherwise you’re precluding any usage of create-react-app or Next or whatever development platform. This problem is further compounded by the fact that those tools encourage including dev dependencies as regular dependencies, since the output is compiled anyway. The answer here is probably some kind of static analysis to know which packages end up shipping in the actual bundle to users. I think Dan referenced some work in that regard.
    • phkahler 5y ago
      >> The answer here is probably some kind of static analysis... So even more tools?
• xmprt 5y ago
  Instead of marking dependencies as safe by the developer or by the end user, I wonder if the immediate parent can mark it as safe (because it has the appropriate context) and then npm audit can avoid reporting that "vulnerability" when it sees it.
• j1elo 5y ago
  I guess we could have a documentary series! Next up, npm link: broken by design Synopsis of the chapter: A command with broken behavior that has been reported since as early as 2015, but that "got lost" every time the winds changed and the project decided to change where to manage bugs. What will happen in the latest attempt from an affected user? Tune in and be ready for an exciting ride! https://github.com/npm/cli/issues/2372 https://github.com/npm/cli/issues/2372 Spoiler: bugs are not sentient beings that solve themselves just by closing the issue (or the whole issue tracker, for that matter). EDIT to clarify: Sorry for the snarkyness. I just find it funny in a sarcastic way that up until I reported the issue in 2020, the issue had been reported repeatedly but "lost" in the way because the project closed or ignored the issue every time it changed issue tracker. Which happened twice since 2017! so go figure the amount of reports that had gone to waste. On the flip side, this time they haven't changed platforms (yet), although the issue has been closed prematurely anyway.
  • mikewhy 5y ago
    Funny, because Yarn does do what you expect here and I absolutely loathe it.
    • winrid 5y ago
      What's to loathe about Yarn? I only used it briefly buy in comparison to NPM there were less surprises.
  • hinkley 5y ago
    npm has been buggy for so long that it is actively driving me away from NodeJS. I would like to wait to see if the rearchitecture for npm 7 actually allows them to test for regressions more productively, but at this point I don't know if I have the stamina to wait for my company to migrate to node 16. Someone offers me a job doing Elixir or non-webapp stuff and I'm out. Probably permanently.
    • earthboundkid 5y ago
      Node is the new IE.
• ur-whale 5y ago
  This should be: npm: broken by design. Or even, since it's in fact the language itself that sets the tone for the entire ecosystem: javascript: broken by design.
• dean177 5y ago
  It is just pure noise: Add `audit=false` to your ~/.npmrc to disable it
• datavirtue 5y ago
  Agreed. I have been trained by npm to ignore its audit messages.
• catears 5y ago
  So I don't work as a security professional but what I remember from IT-sec class in uni is that in order to craft an exploit you need to be vulnerable and the vulnerability needs to be exploitable. If I put a database with default credentials on the internet, there is both a vulnerability and it is exploitable. Bad. If I run a database with default credentials on my dev machine, it is vulnerable, but not exploitable. Perfectly fine. For real security work you also need to think about impact. Hacker dropping production database = we all lose our jobs. Co-worker connecting to my computer and dropping database as a joke = no real harm done. So three things to think about: - Vulnerability - Exploitability - Impact What I really don't like about npm audit is how it presents itself as "security tool" and how vulnerabilities are presented. "6 critical, 10 high vulnerabilities" with a red color screams "fix me now!!!". This is not fair to users because npm has no idea of either the exploitability or the impact of the vulnerability. Why present users with a prompt "please fix me now!!" and not even mention that exploitability and impact need to be measured first? Seems like they forgot that prompt...
  • Ayesh 5y ago
    Vulnerabilities reported to CVE carry a risk score (CVSS) that conveys this information in standard way.
• 1MachineElf 5y ago
  It's true that the signal to noise ratio is high for much of these, but whatever solution we settle on, it should take into account that forcing even beginners to learn how to use npm audit means that security will be taken into consideration from the start, which is both valuable and a net benefit.
• efitz 5y ago
  Writing my comments in the snarky tone of the article. So the article boils down to “a bunch of these vulnerabilities aren’t applicable to my app which is built using a specific NPM package”. Congratulations. Welcome to the world of practical information security. As a security engineer, we’re lucky if your favorite package manager even associates vulnerability information with your packages. Never mind that you’re pulling in code at build time from who-only-knows-where that almost certainly wasn’t security reviewed. But that’s for another post. Now you have a package manager that is kind enough to tell you that there might be a vulnerability, and you’re upset because NPM did not have specific logic to understand the mechanics of one of the packages it manages? And the upshot is that you have to apply judgment and attention to each notification? Is that a tear in my eye- no, wait, it’s just an eyelash. How many packages are there? I’m sure the NPM guys have nothing better to do than to build context awareness for every package in their repository. In all seriousness, I would love to see context awareness in vulnerability reporting. But expecting a package manager to understand that because of your specific choice of framework, that the DoS could only be conducted by an admin of your app, seems unreasonable to me.
  • overgard 5y ago
    The point is that if the feature is going to constantly produce false positives, it's useless. I concur with the author, I never check those warnings anymore.
    • efitz 5y ago
      It’s not a false positive, as described. It’s “mitigated by environment”. The vulnerability is real. The severity is arguably too high. People throw around “false positive” as a catch-all for “I don’t care about this”. But there are a number of distinct reasons one might not care: - the scanner is wrong (e.g. there’s a code bug in the scanner like detecting “printed” instead of “sprintf”. - the output is wrong because the vulnerability isn’t a vulnerability anytime, anywhere under any circumstances - the scanner is correct, but environment or mitigation’s mean it doesn’t apply to me or the severity is wrong in my environment (this is the case here) - the scanner is correct, but is giving me output I don’t care about (eg I want to filter for only high/critical but I can’t) - there is so much output that I can’t pay attention to all of it; it’s so overwhelming that I can’t stand to look at it Many security products have problems with output that is too verbose. This seems like a trivial problem to work around here; after you’ve triaged that a particular vulnerability doesn’t apply to a particular project, then filter it out with grep -v (our put a bunch of such lines in a bash script and always pipe npm audit output to the script. Also, I sympathize with concerns that the vulnerability reporter perhaps scored the vulnerability too high. But there’s no perfect solution for that, and I’d rather be aware of a vuln and choose to ignore it, than not be aware at all.
• tonetheman 5y ago
  Bad/ill formed regexes took down the internet recently so they are in fact an attack surface https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/ https://blog.cloudflare.com/details-of-the-cloudflare-outage...
• charcircuit 5y ago
  This is why something like living at head is important. If npm audit reports something you should just be able to upgrade to the latest version. Being stuck with old versions is not good. Sure a vulnerability might not effect you now, but what if someone on your team uses that dependency again in a way that ran be exploited, or what if a new vulnerability comes out that actually effects you. You will be stuck on an old version and have to struggle to update.
• bjornstar 5y ago
  It's really disappointing to hear an important member of the javascript community not maintaining their library and then blaming npm when people rightfully complain about it. This is like getting mad at the guidebook for showing which plants are weeds when your neighbors complain that your unmaintained garden is full of weeds. If Dan says "npm audit is a stain on the entire npm ecosystem", maybe it's safe to say that Create React App is a stain on the entire react ecosystem. The best time to maintain it was every single month of its existence because that's how you maintain software. Facebook has abandoned Create React App. Dan stated that he intentionally does not maintain the project. Rather than complain about npm audit, they should give Create React App over to the community who actually use it instead of keeping it shambling along as a zombie with their name on it. And if Facebook doesn't want to give it up, the best thing we can do as a community is to move on to any of the other great tools available that are actually maintained.
  • BHSPitMonkey 5y ago
    > This is like getting mad at the guidebook for showing which plants are weeds when your neighbors complain that your unmaintained garden is full of weeds. Wouldn't it be more like getting mad at a guidebook which falsely claims that every plant in your garden is a weed, forcing you to second-guess all of its assertions and therefore wasting a lot of your time?
    • bjornstar 5y ago
      It's like claiming that the weeds are not dangerous in your garden not that the guidebook got them wrong.
• naugtur 5y ago
  Let's talk about solutions. I'm late to the conversation here, responded on Twitter and went to sleep. There's a push to address the npm audit situation. It's an initiative under the OpenJS Foundation. I kinda started the whole conversation by implementing a tool that makes it acceptable instead of ditching npm audit. It's called npm-audit-resolver and I've written about it here https://dev.to/naugtur/do-you-need-help-with-your-npm-audit-3olf https://dev.to/naugtur/do-you-need-help-with-your-npm-audit-... Also check out the collab space and the tool itself https://github.com/openjs-foundation/pkg-vuln-collab-space https://github.com/openjs-foundation/pkg-vuln-collab-space https://www.npmjs.com/package/npm-audit-resolver https://www.npmjs.com/package/npm-audit-resolver
• deleted 5y ago
  [deleted]