6 ms·

What's Going on Inside Your Node_modules Folder?

64 points by swyx 5y ago

crossroadsguy 5y ago
I am an android developer and we having a react bridge, so sometimes I end up dealing with npm/yarn etc. The sight and depth of node_modules folder gives me panic attacks. The ~/.gradle/caches isn’t something to write home about either.
- nhoughto 5y ago
  yeah gradle/maven packages benefit from being much more coarse grain, so fewer large packages, fewer authors, fewer transitives, fewer risks in theory. Quicker to download and manage too, it really is just the tiny size of JS packages that causes much of the problems, having an RCE baked into NPM doesn't help either I guess.. I did just check my ~/.gradle is 17G, I can probably delete Gradle 4.0, 4.1, 4.8, 5.0... thanks for the reminder! =)
isabellat 5y ago
This post does a really great job explaining the risk we all take when downloading code that we haven't audited from the internet. I've never taken the time to audit the code I'm including in my projects. Instead, I blindly trust that someone else has. I love how socket has identified common security red flags and is automatically warning users about what they find. Great to see!
- cube00 5y ago
  For a lone dev, it's impossible but I wonder if the big players like Facebook and Google actually audit all the transitive dependencies they've selected each time they release a new version of React or Angular.
pupppet 5y ago
Kinda wish npm gave you a little more info up front, like ‘this package will download # dependencies, do you wish to continue?’
- bretc 5y ago
  There is a smattering a tools across the eco system that provide this kind of info (packagephobia, avanka etc) and it would be fantastic to surface these in a unified product UI. You will know we've met our goals when socket becomes your goto service to navigate npm!
gunapologist99 5y ago
Even though this is obviously pitching their auditing app, the article is absolutely excellent and the interactive visualization for webpack alone is extremely well done and rather terrifying. Looks like an awesome tool, actually.
- deostroll 5y ago
  The crux of the issue is really what we call the "kingdom of humanity". The earlier the population learn about this hard truth, the better.
awinter-py 5y ago
point in the middle about difficulty lining up github + npm package is smart f-droid distributes builds they run themselves from github, which depending on your trust model, is either better or worse than letting authors upload binaries, but at least shows awareness of the problem trusted CI for software will be a big deal -- it doesn't have to be open source, but in the case of a breach you should be able to have a trusted third party check the source code for your binary and rule out foul play we should also be incentivizing ($) expert code review of high-traffic packages -- I suspect this happens informally already (remember how big cos got serious about ssl post snowden), but would be nice to formalize
- bretc 5y ago
  npm and GitHub have always had an awkward overlap. One of the goals at Socket is to provide an aligned 'union' view into the data both services offer. Keen eye!
bruth 5y ago
Changelog podcast with the author: https://changelog.com/podcast/482 https://changelog.com/podcast/482
nightski 5y ago
I'd be curious to see how many packages have a pre/post install script that gets executed automatically by npm. This seems particularly problematic. It's a huge reduction in security, I'm surprised npm can justify the feature.
- photon12 5y ago
  There's a reason I'll only ever run NPM in a VM on any machine that has any data I care about
  - bretc 5y ago
    Install scripts can be disabled with a config file, or npm flag, but your paranoia is not unwarranted.
- mekster 5y ago
  Does deno (try to) solve any of this?
  - bretc 5y ago
    Deno does not have install scripts, or a lot of other npm+node features. You can turn these off in npm though if you want a hardened install step.
- bretc 5y ago
  Check out http://socket.dev/npm/issue/installScripts http://socket.dev/npm/issue/installScripts Majority are benign, but when a package without one adds one, you probably want to see why :)
mdaniel 5y ago
Based on the blog's domain, this may be a mild dupe of https://news.ycombinator.com/item?id=30515090 https://news.ycombinator.com/item?id=30515090
sylens 5y ago
What makes this problem so endemic to the Node community? I feel like I never hear about these types of scenarios with NuGet or other package managers. Not to say that it's not a problem elsewhere - it just seems like all the horror stories stem from NPM
- geostyx 5y ago
  I think it's partly because the JS ecosystem doesn't historically have a solid standard library. Doing simple things can require checking for null/undefined/does the runtime support it, etc. Why do it the hard way when you can NPM install it and call it a day? That's my understanding of it anyway.
  - vimax 5y ago
    That and an aggressive auto update with lax version constraints for transitive dependencies leads to frequently downloading untrusted code.
  - Jcampuzano2 5y ago
    The number of libraries reached for on your average JS app is humongous compared to basically any other programming language I've worked with, partially due to this. I think it's also just a footgun of the JS community. People tend to jump to "what package do I need to install for this" much quicker instead of thinking "how can I solve this". Every recent JS developer that is learning through online material is constantly berated with "just install this dep, and this dep, and then this one", to the point where it's normalized to have a dependency that comes with who knows what for something that could be a few lines of code and maybe some witty google-foo.
- bretc 5y ago
  Same reason most of the malware ecosystem focused on windows instead of macOS for many years. Scale of your potential target, and also volume of noise that is made when something bad happens. Also it supports nested dependencies and the tools can accommodate large deep trees without much pain, so they tend to grow in size over time.
whoisstan 5y ago
Would be great to run their tool as a proxy to npm whenever you install rather being triggered on a github action.
- bretc 5y ago
  We are assessing this possibility.
  - whoisstan 5y ago
    I would pay for that.
whoisstan 5y ago
NPM at work https://medium.com/hackernoon/im-harvesting-credit-card-numbers-and-passwords-from-your-site-here-s-how-9a8cb347c5b5 https://medium.com/hackernoon/im-harvesting-credit-card-numb...
CSSer 5y ago
Why can’t we solve this problem by forcing users to provide checksums? It’s particularly damning that Microsoft owns Github and Npm for their own strategic purposes and do nothing to mitigate the fact that source code hosted on Github can differ drastically from the compiled code in the registry.
roberttod 5y ago
Even if the pre/post install hooks are removed, you can still execute code in the source code files using some exec command. Doesn't catch as many people out, but anyone running an app locally has given it permissions to your shell. This is the same problem for Go and other package managers right? I guess the best defence is some npm install alternative that runs a check against a trusted registry of modules that have been audited. Does that exist already?
LinAGKar 5y ago
"Node_modules"? I don't think that exists

64 points by swyx 5y ago

I am an android developer and we having a react bridge, so sometimes I end up dealing with npm/yarn etc. The sight and depth of node_modules folder gives me panic attacks. The ~/.gradle/caches isn’t something to write home about either.

nhoughto 5y ago
yeah gradle/maven packages benefit from being much more coarse grain, so fewer large packages, fewer authors, fewer transitives, fewer risks in theory. Quicker to download and manage too, it really is just the tiny size of JS packages that causes much of the problems, having an RCE baked into NPM doesn't help either I guess.. I did just check my ~/.gradle is 17G, I can probably delete Gradle 4.0, 4.1, 4.8, 5.0... thanks for the reminder! =)

isabellat 5y ago

This post does a really great job explaining the risk we all take when downloading code that we haven't audited from the internet. I've never taken the time to audit the code I'm including in my projects. Instead, I blindly trust that someone else has. I love how socket has identified common security red flags and is automatically warning users about what they find. Great to see!

cube00 5y ago
For a lone dev, it's impossible but I wonder if the big players like Facebook and Google actually audit all the transitive dependencies they've selected each time they release a new version of React or Angular.

pupppet 5y ago

Kinda wish npm gave you a little more info up front, like ‘this package will download # dependencies, do you wish to continue?’

bretc 5y ago
There is a smattering a tools across the eco system that provide this kind of info (packagephobia, avanka etc) and it would be fantastic to surface these in a unified product UI. You will know we've met our goals when socket becomes your goto service to navigate npm!

gunapologist99 5y ago

Even though this is obviously pitching their auditing app, the article is absolutely excellent and the interactive visualization for webpack alone is extremely well done and rather terrifying. Looks like an awesome tool, actually.

deostroll 5y ago
The crux of the issue is really what we call the "kingdom of humanity". The earlier the population learn about this hard truth, the better.

awinter-py 5y ago

point in the middle about difficulty lining up github + npm package is smart f-droid distributes builds they run themselves from github, which depending on your trust model, is either better or worse than letting authors upload binaries, but at least shows awareness of the problem trusted CI for software will be a big deal -- it doesn't have to be open source, but in the case of a breach you should be able to have a trusted third party check the source code for your binary and rule out foul play we should also be incentivizing ($) expert code review of high-traffic packages -- I suspect this happens informally already (remember how big cos got serious about ssl post snowden), but would be nice to formalize

bretc 5y ago
npm and GitHub have always had an awkward overlap. One of the goals at Socket is to provide an aligned 'union' view into the data both services offer. Keen eye!

bruth 5y ago

Changelog podcast with the author: https://changelog.com/podcast/482 https://changelog.com/podcast/482

nightski 5y ago

I'd be curious to see how many packages have a pre/post install script that gets executed automatically by npm. This seems particularly problematic. It's a huge reduction in security, I'm surprised npm can justify the feature.

photon12 5y ago
There's a reason I'll only ever run NPM in a VM on any machine that has any data I care about
- bretc 5y ago
  Install scripts can be disabled with a config file, or npm flag, but your paranoia is not unwarranted.
mekster 5y ago
Does deno (try to) solve any of this?
- bretc 5y ago
  Deno does not have install scripts, or a lot of other npm+node features. You can turn these off in npm though if you want a hardened install step.
bretc 5y ago
Check out http://socket.dev/npm/issue/installScripts http://socket.dev/npm/issue/installScripts Majority are benign, but when a package without one adds one, you probably want to see why :)

mdaniel 5y ago

Based on the blog's domain, this may be a mild dupe of https://news.ycombinator.com/item?id=30515090 https://news.ycombinator.com/item?id=30515090

sylens 5y ago

What makes this problem so endemic to the Node community? I feel like I never hear about these types of scenarios with NuGet or other package managers. Not to say that it's not a problem elsewhere - it just seems like all the horror stories stem from NPM

geostyx 5y ago
I think it's partly because the JS ecosystem doesn't historically have a solid standard library. Doing simple things can require checking for null/undefined/does the runtime support it, etc. Why do it the hard way when you can NPM install it and call it a day? That's my understanding of it anyway.
- vimax 5y ago
  That and an aggressive auto update with lax version constraints for transitive dependencies leads to frequently downloading untrusted code.
- Jcampuzano2 5y ago
  The number of libraries reached for on your average JS app is humongous compared to basically any other programming language I've worked with, partially due to this. I think it's also just a footgun of the JS community. People tend to jump to "what package do I need to install for this" much quicker instead of thinking "how can I solve this". Every recent JS developer that is learning through online material is constantly berated with "just install this dep, and this dep, and then this one", to the point where it's normalized to have a dependency that comes with who knows what for something that could be a few lines of code and maybe some witty google-foo.
bretc 5y ago
Same reason most of the malware ecosystem focused on windows instead of macOS for many years. Scale of your potential target, and also volume of noise that is made when something bad happens. Also it supports nested dependencies and the tools can accommodate large deep trees without much pain, so they tend to grow in size over time.

whoisstan 5y ago

Would be great to run their tool as a proxy to npm whenever you install rather being triggered on a github action.

bretc 5y ago
We are assessing this possibility.
- whoisstan 5y ago
  I would pay for that.

whoisstan 5y ago

NPM at work https://medium.com/hackernoon/im-harvesting-credit-card-numbers-and-passwords-from-your-site-here-s-how-9a8cb347c5b5 https://medium.com/hackernoon/im-harvesting-credit-card-numb...

CSSer 5y ago

Why can’t we solve this problem by forcing users to provide checksums? It’s particularly damning that Microsoft owns Github and Npm for their own strategic purposes and do nothing to mitigate the fact that source code hosted on Github can differ drastically from the compiled code in the registry.

roberttod 5y ago

Even if the pre/post install hooks are removed, you can still execute code in the source code files using some exec command. Doesn't catch as many people out, but anyone running an app locally has given it permissions to your shell. This is the same problem for Go and other package managers right? I guess the best defence is some npm install alternative that runs a check against a trusted registry of modules that have been audited. Does that exist already?

LinAGKar 5y ago

"Node_modules"? I don't think that exists