8 ms·

Unpinned dependencies are harmful. If you aren’t reviewing the diffs of your dependencies when you update them, you’re trusting random strangers on the Interne

Unpinned dependencies are harmful. If you aren’t reviewing the diffs of your dependencies when you update them, you’re trusting random strangers on the Internet to run code on your systems. Espionage often spans multi-year timelines of preparation and trust building. No lesser solution will ever be sufficient to protect you. Either read the diffs, or pay someone like RedHat to do so and hope that you can trust them.

dmortin 8y ago
If you have tons of depedencies then it's not feasible to check every diff. You may able to do it or pay someone if you are a bigger organization, but a small shop or solo developer can't do this.
- choward 8y ago
  That's why you lock your dependencies.
  - dmortin 8y ago
    Dependencies need bugfixes and you may even want to use new features, so locking is not a permanent solution.
    - twodave 8y ago
      If you're on point enough to know which features/bugfixes you're getting then you're probably doing enough to be safe already. Just don't go around running npm -u for no reason and you should be fine. The only way to be truly safe from this attack vector is to own all of your dependencies, and nobody is willing to do that so we're all assuming some amount of risk.
  - cellis 8y ago
    That will work assuming have have audited the code already, but you will also have to audit every (changed dependency)(factorial) every time you bump a dependency!
  - sroussey 8y ago
    Hope you don’t start your project with the compromised version — and then lock it in.
- floatingatoll 8y ago
  Most users of dependencies will never consider whether to trust those dependencies at all. If you’ve considered the problem and decided to trust them as a compromise to reduce staffing and costs, that’s a fine outcome in my book. How many health/medical startups are blindly trusting dependencies today, having never thought through the harm I describe?
- twodave 8y ago
  Maybe not, but you can avoid updating unless necessary. Assuming you only make necessary updates (and at least do a cursory check at the time) and vet any newly added-dependencies as you go, you can greatly reduce your own attack surface. You're still probably vulnerable to dependency changes up the chain, but then at least you're depending on a community that is ostensibly trustworthy (i.e. if every maintainer at least feels good about their top-level dependencies then the whole tree should be trustworthy).
- chomp 8y ago
  I would caution one to not have tons of dependencies. More surface area in terms of the amount of 3rd party libraries/developers means more chances that one of them is not a responsible maintainer, as in this case. That increases the application's security risk.
  - yetanotherjosh 8y ago
    Then let's hope you're not using Webpack, which alone has several hundred of them and not small ones, mind you... super complex libraries that trying to "own" enough that you'd be able to securely review code diffs is completely infeasible.
- jerf 8y ago
  "If you have tons of depedencies then it's not feasible to check every diff." Part of bringing in a dependency is bringing in the responsibility for verifying it's not obviously being used badly. One of the things I've come to respect the Go community for is its belief that dependencies are more expensive that most developers currently realize, and so generally library authors try to minimize dependencies. Our build systems make it very easy to technically bring in lots of dependencies, but are not currently assisting us in maintaining them properly very often. (In their defense, it is not entirely clear to me what the latter would even mean at an implementation level. I have some vague ideas, but nothing solid enough to complain about not having when even I don't know what it is I want exactly.) I've definitely pulled some things in that pulled in ~10 other dependencies, but after inspection, they were generally all very reasonable. I've never pulled a Go library and gotten 250 dependencies pulled in transitively, which seems to be perfectly normal in the JS world. I won't deny I haven't auditing every single line of every single dependency... but I do look at every incoming patch when I update. It's part of the job. (And I have actually looked at the innards of a fairly significant number of the dependencies.) It's actually not that hard... malicious code tends to stick out like a sore thumb. Not always [1], but the vast majority of the time. In static languages, you see things like network activity happening where it shouldn't, and in things like JS, the obfuscation attempts themselves have a pretty obvious pattern to them (big honking random-looking string, fed to a variety of strange decoding functions and ultimately evaluated, very stereotypical look to it). And let me underline the point that there's a lot of tooling right now that actively assists you into getting into trouble on this front, but doesn't do much to help you hold the line. I'm not blaming end developers 100%. Communities have some work here to be done too. [1]: http://underhanded-c.org/ http://underhanded-c.org/
  - muraiki 8y ago
    Unfortunately, one of the things that makes JS dev pull in so many deps is that it lacks a decent standard library. Meanwhile, the Go standard library is amazing!
  - pcwalton 8y ago
    I'm not convinced that this incident argues in favor of Go's "a little copying is better than a little dependency", which I continue to strongly disagree with. Rather, it indicates that you shouldn't blindly upgrade. Dependency pinning exists for a reason, and copying code introduces more problems than it solves.
- kentm 8y ago
  I would argue that having tons of dependencies is a problem in and of itself. This has become normal in software development because it’s been made easy to include dependencies, but a lot of the costs and risks are hidden.
ttty 8y ago
I'd love to do this, but how can you review thousands of lines of code that changed?
- albertgoeswoof 8y ago
  Ideally you don’t pull in thousands of lines of code in unknown dependencies as a starting point
- twodave 8y ago
  You can't, you're forced to trust to some degree depending on factors specific to your project. If you're writing missile navigation code then you better check every last diff, but if you're writing a recipe sharing site then you don't have the same burden really.
- kd5bjo 8y ago
  You can use smaller, more targeted libraries so that the changes you need to review are actually relevant to your project.
  - sulam 8y ago
    In the Node/NPM world, this is pretty difficult. There are many (many) small libraries and everything you depend on brings in many more.
    - gkya 8y ago
      That's a big and totally objective reason to abandom the Node.js/NPM ecosystem, like its original author did. A language that doesn't have a decent standard library means that you'll have to use huge amounts of code that random strangers used, and the chain of dependencies will grow larger and larger. In languages like Ruby and Python, you have a decent standard library, and then big libraries and frameworks that are maintained by the community, like Rails, Django, SqlAlchemy, Numpy, etc. That's healthy because it minimises or zeros the amount of small libraries maintained by a single guy, thus maximising the amount of code that you can trust (because you can trust the devs and development process of a popular library backed by a foundation or with many contributors). With Node, almost every function comes from a different package. And there's no bar to entry, and no checks. If Node.js is going to stay, someone needs to take on the responsability of forming a project, that belongs to an NGO or something, where the more popular libraries are merged and fused into a standard library, like that of Python's. Personally, I'm not touching it until then.
lmm 8y ago
Lack of code signing is what's harmful here. My maven projects' dependencies are technically all pinned, but updates are just a "mvn versions:use-latest-releases" away. But, crucially, I have a file that lists which GPG key IDs I trust to publish which artifacts. If the maintainer changes, the new maintainer will sign with their key instead, my builds will (configurably) fail, and I can review and decide whether I want to trust the new maintainer or not. Of course, NPM maintainers steadfastly refuse to implement any kind of code signing support...
- raesene9 8y ago
  Even package managers where signing support nominally exists, the take-up is often poor. IIRC rubygems support package signing but almost no-one uses it, so it's effectively useless. We're seeing the same pattern again with Docker. They added support for signing (content trust) but unfortunately it's not at all designed for the use case of packages downloaded from Docker hub, so it's adoption has been poor.
  - SCHiM 8y ago
    I think browsers show how to migrate away from insecure defaults successfully. The client software should start showing big obvious warnings. Later stages should add little inconveniences such as pop-ups and user acknowledgement prompts, eg. 'I understand that what I'm doing is dangerous, wait 5 seconds to continue'. The final stage should disable access to unsigned packages without major modifications to the default settings.
    - raesene9 8y ago
      Browser security has heavily benefited from the fact that ther are a small number of companies with control over the market and an incentive to improve security. Unfortunately the development world doesn't really have the same opporunities. If, for example, npm started to get strict about managing, curating, security libs, they could just move to a new package manager. Security features (e.g. package signing, package curation) have not been prioritised by developers, so they aren't widely provided.
justhelpingout 8y ago
The problem here is that the diff is not source, but "compiled" code. We ultimately come back to "Reflections on Trusting Trust" [1] [1]: https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html
Klathmon 8y ago
Code review can't catch determined malicious actors. It just isn't a viable protection against that kind of attack. Take a look at the underhanded c contest for plenty of proof where even very senior developers told up front that there is a backdoor in the code often can't find it! And they can't all be blamed on C being C, many of them would work the same in any language. I don't know the solution, but shaming users and developers for not reviewing code enough sure as hell isn't it. All that being said, reviewing changes in dependencies is still a good idea, as it can catch many other things.
- kurtisc 8y ago
  Most malicious actors aren't determined, they're lazy and will take the path of least resistance. Code review will catch out those.
- floatingatoll 8y ago
  There’s no shame in refusing to study dependency diffs. If you consciously evaluate the risk and deem it acceptable, I agree with your judgement call. What I find shameful is the lack of advisory warnings about this risk — by the repository, by the language’s community, by the teaching material. This should have been a clearly-known risk. Instead, it was a surprise. The shame here falls on the NPM community as a whole failing to educate its users on the risks inherent in NPM, not the individual authors and users of NPM modules.
teej 8y ago
How does this work practically when the vuln exists only in the minified version of the code?
- gregknicholson 8y ago
  https://reproducible-builds.org/ https://reproducible-builds.org/
- lozenge 8y ago
  Most projects are already bundling and minifying code themselves. Any cookiecutter type tool will say that up.
jonknee 8y ago
> you’re trusting random strangers on the Internet to run code on your systems I mean that's pretty much how the world works. Even running Linux is trusting random strangers on the internet. Most of the time it works pretty well, but obviously it's not perfect. Even the largest companies in the world get caught with security issues from open source packages (remember Heartbleed?).
- nneonneo 8y ago
  When I visit a random website, it is very hard for that website to compromise my computer or my private data. The only really viable way is a zero-day in my browser, or deception (e.g. phishing, malicious download). When I install an app on my iPhone, it is very very hard for that app to compromise my phone or my private data. In both of these cases, I can download and run almost any code, and be fairly confident that it won't significantly harm me. Why? Because they're extremely locked down and sandboxed with a high degree of isolation. On the other hand, if I install random software on my desktop computer, or random packages off NPM, I don't have such safety any more. The prevalence of app stores and the Web itself speaks to the fact that it _is_ possible to trust random strangers without opening yourself up to a big security risk.
  - 83457 8y ago
    I think what you just described are platforms where you don't have to trust strangers.
    - swsieber 8y ago
      Yes, and I think they meant to. Node isn't one of those. Edit: partially because of design decisions around package permissions.
  - jonknee 8y ago
    What does that have to do with development? My point was nearly everyone uses libraries that are written by strangers from the internet. It mostly works.
joekrill 8y ago
But you also have to review the diffs of your dependency's dependencies. And their dependencies. And so on and so on.
BinaryIdiot 8y ago
Unfortunately this isn't really doable in today's world of JavaScript development. If you want to use any of the popular frameworks you are installing a metric ton of dependency code. So not only do you have to somehow review that initial set of code, but you need to know how to spot these types of things. Then, once you complete that task, you now have to look at the diffs for each update. And there will be a lot of updates. What you're suggesting is a great idea from a security perspective. But for typical workflows for JS development it just isn't practical. Now, maybe this means we need different workflows and less dependencies. But it's so ingrain I don't know that it's easy to fix / change.
a0viedo 8y ago
Your direct dependencies are not the only problem.

dmortin 8y ago

If you have tons of depedencies then it's not feasible to check every diff. You may able to do it or pay someone if you are a bigger organization, but a small shop or solo developer can't do this.

choward 8y ago
That's why you lock your dependencies.
- dmortin 8y ago
  Dependencies need bugfixes and you may even want to use new features, so locking is not a permanent solution.
  - twodave 8y ago
    If you're on point enough to know which features/bugfixes you're getting then you're probably doing enough to be safe already. Just don't go around running npm -u for no reason and you should be fine. The only way to be truly safe from this attack vector is to own all of your dependencies, and nobody is willing to do that so we're all assuming some amount of risk.
- cellis 8y ago
  That will work assuming have have audited the code already, but you will also have to audit every (changed dependency)(factorial) every time you bump a dependency!
- sroussey 8y ago
  Hope you don’t start your project with the compromised version — and then lock it in.
floatingatoll 8y ago
Most users of dependencies will never consider whether to trust those dependencies at all. If you’ve considered the problem and decided to trust them as a compromise to reduce staffing and costs, that’s a fine outcome in my book. How many health/medical startups are blindly trusting dependencies today, having never thought through the harm I describe?
twodave 8y ago
Maybe not, but you can avoid updating unless necessary. Assuming you only make necessary updates (and at least do a cursory check at the time) and vet any newly added-dependencies as you go, you can greatly reduce your own attack surface. You're still probably vulnerable to dependency changes up the chain, but then at least you're depending on a community that is ostensibly trustworthy (i.e. if every maintainer at least feels good about their top-level dependencies then the whole tree should be trustworthy).
chomp 8y ago
I would caution one to not have tons of dependencies. More surface area in terms of the amount of 3rd party libraries/developers means more chances that one of them is not a responsible maintainer, as in this case. That increases the application's security risk.
- yetanotherjosh 8y ago
  Then let's hope you're not using Webpack, which alone has several hundred of them and not small ones, mind you... super complex libraries that trying to "own" enough that you'd be able to securely review code diffs is completely infeasible.
jerf 8y ago
"If you have tons of depedencies then it's not feasible to check every diff." Part of bringing in a dependency is bringing in the responsibility for verifying it's not obviously being used badly. One of the things I've come to respect the Go community for is its belief that dependencies are more expensive that most developers currently realize, and so generally library authors try to minimize dependencies. Our build systems make it very easy to technically bring in lots of dependencies, but are not currently assisting us in maintaining them properly very often. (In their defense, it is not entirely clear to me what the latter would even mean at an implementation level. I have some vague ideas, but nothing solid enough to complain about not having when even I don't know what it is I want exactly.) I've definitely pulled some things in that pulled in ~10 other dependencies, but after inspection, they were generally all very reasonable. I've never pulled a Go library and gotten 250 dependencies pulled in transitively, which seems to be perfectly normal in the JS world. I won't deny I haven't auditing every single line of every single dependency... but I do look at every incoming patch when I update. It's part of the job. (And I have actually looked at the innards of a fairly significant number of the dependencies.) It's actually not that hard... malicious code tends to stick out like a sore thumb. Not always [1], but the vast majority of the time. In static languages, you see things like network activity happening where it shouldn't, and in things like JS, the obfuscation attempts themselves have a pretty obvious pattern to them (big honking random-looking string, fed to a variety of strange decoding functions and ultimately evaluated, very stereotypical look to it). And let me underline the point that there's a lot of tooling right now that actively assists you into getting into trouble on this front, but doesn't do much to help you hold the line. I'm not blaming end developers 100%. Communities have some work here to be done too. [1]: http://underhanded-c.org/ http://underhanded-c.org/
- muraiki 8y ago
  Unfortunately, one of the things that makes JS dev pull in so many deps is that it lacks a decent standard library. Meanwhile, the Go standard library is amazing!
- pcwalton 8y ago
  I'm not convinced that this incident argues in favor of Go's "a little copying is better than a little dependency", which I continue to strongly disagree with. Rather, it indicates that you shouldn't blindly upgrade. Dependency pinning exists for a reason, and copying code introduces more problems than it solves.
kentm 8y ago
I would argue that having tons of dependencies is a problem in and of itself. This has become normal in software development because it’s been made easy to include dependencies, but a lot of the costs and risks are hidden.

ttty 8y ago

I'd love to do this, but how can you review thousands of lines of code that changed?

albertgoeswoof 8y ago
Ideally you don’t pull in thousands of lines of code in unknown dependencies as a starting point
twodave 8y ago
You can't, you're forced to trust to some degree depending on factors specific to your project. If you're writing missile navigation code then you better check every last diff, but if you're writing a recipe sharing site then you don't have the same burden really.
kd5bjo 8y ago
You can use smaller, more targeted libraries so that the changes you need to review are actually relevant to your project.
- sulam 8y ago
  In the Node/NPM world, this is pretty difficult. There are many (many) small libraries and everything you depend on brings in many more.
  - gkya 8y ago
    That's a big and totally objective reason to abandom the Node.js/NPM ecosystem, like its original author did. A language that doesn't have a decent standard library means that you'll have to use huge amounts of code that random strangers used, and the chain of dependencies will grow larger and larger. In languages like Ruby and Python, you have a decent standard library, and then big libraries and frameworks that are maintained by the community, like Rails, Django, SqlAlchemy, Numpy, etc. That's healthy because it minimises or zeros the amount of small libraries maintained by a single guy, thus maximising the amount of code that you can trust (because you can trust the devs and development process of a popular library backed by a foundation or with many contributors). With Node, almost every function comes from a different package. And there's no bar to entry, and no checks. If Node.js is going to stay, someone needs to take on the responsability of forming a project, that belongs to an NGO or something, where the more popular libraries are merged and fused into a standard library, like that of Python's. Personally, I'm not touching it until then.

lmm 8y ago

Lack of code signing is what's harmful here. My maven projects' dependencies are technically all pinned, but updates are just a "mvn versions:use-latest-releases" away. But, crucially, I have a file that lists which GPG key IDs I trust to publish which artifacts. If the maintainer changes, the new maintainer will sign with their key instead, my builds will (configurably) fail, and I can review and decide whether I want to trust the new maintainer or not. Of course, NPM maintainers steadfastly refuse to implement any kind of code signing support...

raesene9 8y ago
Even package managers where signing support nominally exists, the take-up is often poor. IIRC rubygems support package signing but almost no-one uses it, so it's effectively useless. We're seeing the same pattern again with Docker. They added support for signing (content trust) but unfortunately it's not at all designed for the use case of packages downloaded from Docker hub, so it's adoption has been poor.
- SCHiM 8y ago
  I think browsers show how to migrate away from insecure defaults successfully. The client software should start showing big obvious warnings. Later stages should add little inconveniences such as pop-ups and user acknowledgement prompts, eg. 'I understand that what I'm doing is dangerous, wait 5 seconds to continue'. The final stage should disable access to unsigned packages without major modifications to the default settings.
  - raesene9 8y ago
    Browser security has heavily benefited from the fact that ther are a small number of companies with control over the market and an incentive to improve security. Unfortunately the development world doesn't really have the same opporunities. If, for example, npm started to get strict about managing, curating, security libs, they could just move to a new package manager. Security features (e.g. package signing, package curation) have not been prioritised by developers, so they aren't widely provided.

justhelpingout 8y ago

The problem here is that the diff is not source, but "compiled" code. We ultimately come back to "Reflections on Trusting Trust" [1] [1]: https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html

Klathmon 8y ago

Code review can't catch determined malicious actors. It just isn't a viable protection against that kind of attack. Take a look at the underhanded c contest for plenty of proof where even very senior developers told up front that there is a backdoor in the code often can't find it! And they can't all be blamed on C being C, many of them would work the same in any language. I don't know the solution, but shaming users and developers for not reviewing code enough sure as hell isn't it. All that being said, reviewing changes in dependencies is still a good idea, as it can catch many other things.

kurtisc 8y ago
Most malicious actors aren't determined, they're lazy and will take the path of least resistance. Code review will catch out those.
floatingatoll 8y ago
There’s no shame in refusing to study dependency diffs. If you consciously evaluate the risk and deem it acceptable, I agree with your judgement call. What I find shameful is the lack of advisory warnings about this risk — by the repository, by the language’s community, by the teaching material. This should have been a clearly-known risk. Instead, it was a surprise. The shame here falls on the NPM community as a whole failing to educate its users on the risks inherent in NPM, not the individual authors and users of NPM modules.

teej 8y ago

How does this work practically when the vuln exists only in the minified version of the code?

gregknicholson 8y ago
https://reproducible-builds.org/ https://reproducible-builds.org/
lozenge 8y ago
Most projects are already bundling and minifying code themselves. Any cookiecutter type tool will say that up.

jonknee 8y ago

> you’re trusting random strangers on the Internet to run code on your systems I mean that's pretty much how the world works. Even running Linux is trusting random strangers on the internet. Most of the time it works pretty well, but obviously it's not perfect. Even the largest companies in the world get caught with security issues from open source packages (remember Heartbleed?).

nneonneo 8y ago
When I visit a random website, it is very hard for that website to compromise my computer or my private data. The only really viable way is a zero-day in my browser, or deception (e.g. phishing, malicious download). When I install an app on my iPhone, it is very very hard for that app to compromise my phone or my private data. In both of these cases, I can download and run almost any code, and be fairly confident that it won't significantly harm me. Why? Because they're extremely locked down and sandboxed with a high degree of isolation. On the other hand, if I install random software on my desktop computer, or random packages off NPM, I don't have such safety any more. The prevalence of app stores and the Web itself speaks to the fact that it _is_ possible to trust random strangers without opening yourself up to a big security risk.
- 83457 8y ago
  I think what you just described are platforms where you don't have to trust strangers.
  - swsieber 8y ago
    Yes, and I think they meant to. Node isn't one of those. Edit: partially because of design decisions around package permissions.
- jonknee 8y ago
  What does that have to do with development? My point was nearly everyone uses libraries that are written by strangers from the internet. It mostly works.

joekrill 8y ago

But you also have to review the diffs of your dependency's dependencies. And their dependencies. And so on and so on.

BinaryIdiot 8y ago

Unfortunately this isn't really doable in today's world of JavaScript development. If you want to use any of the popular frameworks you are installing a metric ton of dependency code. So not only do you have to somehow review that initial set of code, but you need to know how to spot these types of things. Then, once you complete that task, you now have to look at the diffs for each update. And there will be a lot of updates. What you're suggesting is a great idea from a security perspective. But for typical workflows for JS development it just isn't practical. Now, maybe this means we need different workflows and less dependencies. But it's so ingrain I don't know that it's easy to fix / change.

a0viedo 8y ago

Your direct dependencies are not the only problem.