15 ms·

Because this is buried in the post and people don't seem to be grokking it: > Second, on November 2 we received a report to our security bug bounty program of

by nfm 5y ago

Because this is buried in the post and people don't seem to be grokking it: > Second, on November 2 we received a report to our security bug bounty program of a vulnerability that would allow an attacker to publish new versions of any npm package using an account without proper authorization. They correctly authenticated the attacker and checked they were authorised to upload a new version of their own package, but a malicious payload allowed the attacker to then upload a new version of a completely unrelated package that they weren't authorised for. Ouch!

ptx 5y ago
Also: "This vulnerability existed in the npm registry beyond the timeframe for which we have telemetry to determine whether it has ever been exploited maliciously."
jschrf 5y ago
> However, the service that performs underlying updates to the registry data determined which package to publish based on the contents of the uploaded package file Yeah, this is what's going to keep me up tonight. Yikes. I can't help but wonder if the root cause was HTTP request smuggling, or if changing package.json was enough. How do we even mitigate against these types of supply-chain attacks, aside from disabling run-scripts, using lockfiles and carefully auditing the entire dependency tree on every module update? I'm seriously considering moving to a workflow of installing dependencies in containers or VMs, auditing them there, and then perhaps commiting known safe snapshots of node_modules into my repos (YUCK). Horrible developer experience, but at least it'll help me sleep at night.
- grey-area 5y ago
  How do we even mitigate against these types of supply-chain attacks Don’t import thousands of modules from third parties just to write a simple web app. If you have 10 stable dependencies it’s no problem to vendor them and vet changes. If you have 10k you’ve entirely given up on any pretence of security.
  - vosper 5y ago
    People don't directly import thousands of modules. It's actually a lot closer to your "10 stable dependencies". But those dependencies have dependencies that have dependencies. It's a little hard to point the finger at application developers here, IMO.
    - pdonis 5y ago
      > It's a little hard to point the finger at application developers here, IMO. I disagree. Any application developer who seriously thinks that they only have 10 dependencies if they're only importing directly 10 dependencies should not be an application developer in the first place.
      - IggleSniggle 5y ago
        You sure about that? Even if you’re writing just for a vetted distribution of an OS, and you write code with zero explicit dependencies, you still have much more than zero dependencies. It’s turtles all the way down. The key is to have an entire ecosystem that you can, to some degree more or less, trust.
        marcus_holmes 5y ago
        No. we've been shouting warnings for years. There have been dozens, if not hundreds of threads on HN alone warning of supply-chain security threats. At this point if you're not actively auditing your dependencies, and reducing all of them where you can, then you're on the wrong side of history and going down with the Titanic. The frank truth is that including a dependency is, and always has been, giving a random person from the internet commit privileges to prod. The fact that "everyone else did it" doesn't make it less stupid.
        erulabs 5y ago
        > The frank truth is that including a dependency is, and always has been, giving a random person from the internet commit privileges to prod I mean, no. This is hyperbole at best and just wrong at median. A system of relative trust has worked very well for a very long time - Linus doesn’t have root access to all our systems, even if we don’t have to read every line of code.
- secondaryacct 5y ago
  It s very easy: add a dev signature in the repo that cannot be changed ever, and force the devs to sign their stuff before allowing a change of binary or a download. Like that you can have anything trying to upload but fail the signature check.
  - nathanaldensr 5y ago
    This assumes that the developers themselves are not malicious (see: left-pad) and that their signing keys can't be stolen.
- staticassertion 5y ago
  A combination of things, I think. 1. Running those builds in VMs is a good idea. 2. Monitoring for weird behavior. 3. Restricting build scripts from touching anything outside of the build directory. 4. Pressuring organizations like npm to step up their security game. It would be really nice if package repositories: 1. Produced a signed audit log 2. Supported signing keys for said audit log 3. Supported strong 2FA methods 4. Created tooling that didn't run build scripts with full system access etc etc etc I started working on a crates.io mirror and a `cargo sandbox [build|check|etc]` command that would allow crates to specify a permissions manifest for their build scripts, store the policy in a lockfile, and then warn you if a locked policy increased in scope. I'm too busy to finish it but it isn't very hard to do.
  - jschrf 5y ago
    Thanks. I was thinking of a CI step that checked the SHA-256 of yarn.lock against a "last known good" value committed by an authorized committer and enforced by a branch policy. Signed audit logs seem like a good idea. Now...how to get developers to avoid using NPM and Yarn altogether on sensitive projects...
- deleted 5y ago
  [deleted]
- 1propionyl 5y ago
  > I can't help but wonder if the root cause was HTTP request smuggling, or if changing package.json was enough. Maybe I'm just incredibly cynical from my experiences with the intersection of the JS ecosystem and security, but... ...I'd bet dimes to dollars it's the latter (just changing the package.json). My guess is they authenticate but don't actually scope the authentication properly, and no one noticed because no one thought to look. Of course, as we've seen in the past decade, there's so much inertia behind the JavaScript ecosystem that none of this is going to fundamentally change. It'll just take another decade or so for the ecosystem to reinvent all of the wheels and catch up to the rest of the space. And at that point it will probably be considered stuffy and "enterprise" and the new hotness unburdened from such concerns will repeat the cycle again.
  - mhio 5y ago
    > to reinvent all of the wheels and catch up to the rest of the space. Which of the public package systems are the state of the art that should be replicated?
    - thrashh 5y ago
      Java’s works really well. I think it makes other package managers look like a toy.
      - mhio 5y ago
        I assume you are referring to Apache Maven tooling (and compatible) and the pom repos, like Sonatype's Central Repo. PGP package signing is a huge plus. Is that a requirement for publishing? How many different repo's do you typically have to deal with in the average project? Would Sonatype react quickly to malware issue's like this in the repository? Have there been examples of similar package hijacking?
        thrashh 5y ago
        It’s a requirement for the central repo if I recall. And the best past is the signature handling is a part of Java, not the package manager, so nothing needs to be re-invented. The default class loader checks the signatures at runtime as well. Typically you need 1-2 repositories, but often just 1. But if you’re an organization, you can set up your own repository very easily and use it to store private deps and to cache deps (which also allows you to lock binaries and work offline). Repo mirroring is super easy to set up. If you have an internal repo, you can just have your internal project use your own repo and your computer never has to directly reach outside the Internet for a package. Unlike other languages, the “central repo” and the package manager tooling are independent and package resolution is distributed. When you start a project, you choose your repos. I don’t know how quickly Sonatype would react personally but they are only default by de facto. Many packages are published on several repos and mirroring is a default feature of a lot of repo software. If Sonatype started screwing up, everyone could abandon them instantly, which forces them to be better.
- snthd 5y ago
  >How do we even mitigate against these types of supply-chain attacks, aside from disabling run-scripts, using lockfiles and carefully auditing the entire dependency tree on every module update? Don't trust the package distribution system - use public key crypto.
  - hn_throwaway_99 5y ago
    Public key crypto doesn't help much if your private keys get stolen, which was essentially what happened with some of the recent hacked packages and which is why they're now starting to enforce 2FA.
    - woodruffw 5y ago
      The longer term solution to this is public key signatures with an ephemeral key, rooted to some trusted identity source (e.g., a GitHub account with strong 2FA). There’s lots of work on that front coming out of the Open Source Security Foundation.
    - onefuncman 5y ago
      are you really using private keys without a passphrase in 2021?
- 0xDEAFBEAD 5y ago
  >How do we even mitigate against these types of supply-chain attacks I know HN is usually skeptical of anything cryptocurrency/blockchain related, and I am too. But as weird as it sounds, I think blockchain might actually be the solution here. The problem with dependency auditing is it's a lot of work. And it's also duplicate work. What you'd really like to know is whether the dependency you're considering has already been audited by someone you can trust. Ideally someone with skin in the game. Someone who stands to lose something if their audit is incorrect. Imagine a DeFi app that lets people buy and sell insurance for any commit hash of any open source library. The insurance pays out if a vulnerability in that commit hash is found. * As a library user, you want to buy insurance for every library you use. If you experience a security breach, the money you get from the insurance will help you deal with the aftermath. * As an independent hacker, you can make passive income by auditing libraries and selling insurance for the ones that seem solid. If you identify a security flaw, buy up insurance for that library, then publicize the flaw for a big payday. * A distributed, anonymous marketplace is actually valuable here, because it encourages "insider trading" on the part of people who work for offensive cybersecurity orgs. Suppose Jane Hacker is working with a criminal org that's successfully penetrated a particular library. Suppose Jane wants to leave her life of crime behind. All she has to do is buy up insurance for the library that was penetrated and then anonymously disclose the vulnerability. * Even if you never trade on the insurance marketplace yourself, you can get a general idea of how risky a library is by checking how much its insurance costs. (Insurance might be subject to price manipulation by offensive cybersecurity orgs, but independent hackers would be incentivized to identify and correct such price manipulation.) The fact that there is actual value here should give the creator a huge advantage over other "Web 3.0" crypto junk.
  - jschrf 5y ago
    This is a pretty clever application of DeFi, thanks. DeSec? Can't help but wonder if there still would be incentive for lone wolves to slip backdoors and vulnerabilities into libraries though[0]. [0]: https://portswigger.net/daily-swig/smuggling-hidden-backdoors-into-javascript-with-homoglyphs-and-invisible-unicode-characters https://portswigger.net/daily-swig/smuggling-hidden-backdoor...
- Silhouette 5y ago
  I'm seriously considering moving to a workflow of installing dependencies in containers or VMs, auditing them there, and then perhaps commiting known safe snapshots of node_modules into my repos (YUCK). Horrible developer experience, but at least it'll help me sleep at night. I have had people tell me in discussions online, also entirely seriously, that running a package manager to install a dependency while developing is inherently dangerous and anyone who does it outside of a disposable sandboxed VM deserves everything they get. If the packages are inexplicably allowed to do arbitrary things with privileged access to the local system without warning at installation time then clearly the first part is correct, but victim-blaming hardly seems like a useful reaction to that danger.
firebaze 5y ago
No CVE mentioned. Hard to grok to me, could someone educate me why this is missing in the blog post?
- detaro 5y ago
  services don't get CVEs.
  - firebaze 5y ago
    Well, didn't we just experience two major npm published packages containing malware? Both had CVEs. Now we have the probable root cause, buried in a wall of text. No CVE.
    - detaro 5y ago
      Yes, because pure services don't get CVEs. CVEs are for distributed software.
      - firebaze 5y ago
        Isn't this the biggest security flaw in the package ecosystem ever? They don't even know when, if, who and when this was exploited, but maybe I didn't pay enough detail attention to the few paragraphs devoted to the real problem. So shoudn't we assume all NPM packages published prior to 2nd of November are compromised? And if so, shouldn't this deserve a CVE? (https://en.wikipedia.org/wiki/Common_Vulnerabilities_and_Exposures https://en.wikipedia.org/wiki/Common_Vulnerabilities_and_Exp...)
        detaro 5y ago
        CVEs aren't usually assigned for "there might be something wrong", but only identified specific issues.
    - cjbprime 5y ago
      CVEs alert end users that they need to take action to apply updates. That's relevant when a specific npm package contained a known vulnerability. It's not relevant when the npm server contained a known vulnerability. There's nothing a user of npm can do to update the npm server. CVEs don't just mean "this is a big security problem".
bigiain 5y ago
That one, combined with the other “ability to read names of private packages, makes for the possibility of a really really sneaky attack. I wonder how many orgs treat their private npm packages with significantly less scrutiny than the public ones they rely on?
masklinn 5y ago
Having different services trust different (and unrelated) bits of the request is an immortal classic though, great stuff.
- rbanffy 5y ago
  The part that made sure the user could update the package could have at least check if the payload is about that package before passing it to the service that trusted it.

by nfm 5y ago

ptx 5y ago

Also: "This vulnerability existed in the npm registry beyond the timeframe for which we have telemetry to determine whether it has ever been exploited maliciously."

jschrf 5y ago

> However, the service that performs underlying updates to the registry data determined which package to publish based on the contents of the uploaded package file Yeah, this is what's going to keep me up tonight. Yikes. I can't help but wonder if the root cause was HTTP request smuggling, or if changing package.json was enough. How do we even mitigate against these types of supply-chain attacks, aside from disabling run-scripts, using lockfiles and carefully auditing the entire dependency tree on every module update? I'm seriously considering moving to a workflow of installing dependencies in containers or VMs, auditing them there, and then perhaps commiting known safe snapshots of node_modules into my repos (YUCK). Horrible developer experience, but at least it'll help me sleep at night.

grey-area 5y ago
How do we even mitigate against these types of supply-chain attacks Don’t import thousands of modules from third parties just to write a simple web app. If you have 10 stable dependencies it’s no problem to vendor them and vet changes. If you have 10k you’ve entirely given up on any pretence of security.
- vosper 5y ago
  People don't directly import thousands of modules. It's actually a lot closer to your "10 stable dependencies". But those dependencies have dependencies that have dependencies. It's a little hard to point the finger at application developers here, IMO.
  - pdonis 5y ago
    > It's a little hard to point the finger at application developers here, IMO. I disagree. Any application developer who seriously thinks that they only have 10 dependencies if they're only importing directly 10 dependencies should not be an application developer in the first place.
    - IggleSniggle 5y ago
      You sure about that? Even if you’re writing just for a vetted distribution of an OS, and you write code with zero explicit dependencies, you still have much more than zero dependencies. It’s turtles all the way down. The key is to have an entire ecosystem that you can, to some degree more or less, trust.
      - marcus_holmes 5y ago
        No. we've been shouting warnings for years. There have been dozens, if not hundreds of threads on HN alone warning of supply-chain security threats. At this point if you're not actively auditing your dependencies, and reducing all of them where you can, then you're on the wrong side of history and going down with the Titanic. The frank truth is that including a dependency is, and always has been, giving a random person from the internet commit privileges to prod. The fact that "everyone else did it" doesn't make it less stupid.
        erulabs 5y ago
        > The frank truth is that including a dependency is, and always has been, giving a random person from the internet commit privileges to prod I mean, no. This is hyperbole at best and just wrong at median. A system of relative trust has worked very well for a very long time - Linus doesn’t have root access to all our systems, even if we don’t have to read every line of code.
secondaryacct 5y ago
It s very easy: add a dev signature in the repo that cannot be changed ever, and force the devs to sign their stuff before allowing a change of binary or a download. Like that you can have anything trying to upload but fail the signature check.
- nathanaldensr 5y ago
  This assumes that the developers themselves are not malicious (see: left-pad) and that their signing keys can't be stolen.
staticassertion 5y ago
A combination of things, I think. 1. Running those builds in VMs is a good idea. 2. Monitoring for weird behavior. 3. Restricting build scripts from touching anything outside of the build directory. 4. Pressuring organizations like npm to step up their security game. It would be really nice if package repositories: 1. Produced a signed audit log 2. Supported signing keys for said audit log 3. Supported strong 2FA methods 4. Created tooling that didn't run build scripts with full system access etc etc etc I started working on a crates.io mirror and a `cargo sandbox [build|check|etc]` command that would allow crates to specify a permissions manifest for their build scripts, store the policy in a lockfile, and then warn you if a locked policy increased in scope. I'm too busy to finish it but it isn't very hard to do.
- jschrf 5y ago
  Thanks. I was thinking of a CI step that checked the SHA-256 of yarn.lock against a "last known good" value committed by an authorized committer and enforced by a branch policy. Signed audit logs seem like a good idea. Now...how to get developers to avoid using NPM and Yarn altogether on sensitive projects...
deleted 5y ago
[deleted]
1propionyl 5y ago
> I can't help but wonder if the root cause was HTTP request smuggling, or if changing package.json was enough. Maybe I'm just incredibly cynical from my experiences with the intersection of the JS ecosystem and security, but... ...I'd bet dimes to dollars it's the latter (just changing the package.json). My guess is they authenticate but don't actually scope the authentication properly, and no one noticed because no one thought to look. Of course, as we've seen in the past decade, there's so much inertia behind the JavaScript ecosystem that none of this is going to fundamentally change. It'll just take another decade or so for the ecosystem to reinvent all of the wheels and catch up to the rest of the space. And at that point it will probably be considered stuffy and "enterprise" and the new hotness unburdened from such concerns will repeat the cycle again.
- mhio 5y ago
  > to reinvent all of the wheels and catch up to the rest of the space. Which of the public package systems are the state of the art that should be replicated?
  - thrashh 5y ago
    Java’s works really well. I think it makes other package managers look like a toy.
    - mhio 5y ago
      I assume you are referring to Apache Maven tooling (and compatible) and the pom repos, like Sonatype's Central Repo. PGP package signing is a huge plus. Is that a requirement for publishing? How many different repo's do you typically have to deal with in the average project? Would Sonatype react quickly to malware issue's like this in the repository? Have there been examples of similar package hijacking?
      - thrashh 5y ago
        It’s a requirement for the central repo if I recall. And the best past is the signature handling is a part of Java, not the package manager, so nothing needs to be re-invented. The default class loader checks the signatures at runtime as well. Typically you need 1-2 repositories, but often just 1. But if you’re an organization, you can set up your own repository very easily and use it to store private deps and to cache deps (which also allows you to lock binaries and work offline). Repo mirroring is super easy to set up. If you have an internal repo, you can just have your internal project use your own repo and your computer never has to directly reach outside the Internet for a package. Unlike other languages, the “central repo” and the package manager tooling are independent and package resolution is distributed. When you start a project, you choose your repos. I don’t know how quickly Sonatype would react personally but they are only default by de facto. Many packages are published on several repos and mirroring is a default feature of a lot of repo software. If Sonatype started screwing up, everyone could abandon them instantly, which forces them to be better.
snthd 5y ago
>How do we even mitigate against these types of supply-chain attacks, aside from disabling run-scripts, using lockfiles and carefully auditing the entire dependency tree on every module update? Don't trust the package distribution system - use public key crypto.
- hn_throwaway_99 5y ago
  Public key crypto doesn't help much if your private keys get stolen, which was essentially what happened with some of the recent hacked packages and which is why they're now starting to enforce 2FA.
  - woodruffw 5y ago
    The longer term solution to this is public key signatures with an ephemeral key, rooted to some trusted identity source (e.g., a GitHub account with strong 2FA). There’s lots of work on that front coming out of the Open Source Security Foundation.
  - onefuncman 5y ago
    are you really using private keys without a passphrase in 2021?
0xDEAFBEAD 5y ago
>How do we even mitigate against these types of supply-chain attacks I know HN is usually skeptical of anything cryptocurrency/blockchain related, and I am too. But as weird as it sounds, I think blockchain might actually be the solution here. The problem with dependency auditing is it's a lot of work. And it's also duplicate work. What you'd really like to know is whether the dependency you're considering has already been audited by someone you can trust. Ideally someone with skin in the game. Someone who stands to lose something if their audit is incorrect. Imagine a DeFi app that lets people buy and sell insurance for any commit hash of any open source library. The insurance pays out if a vulnerability in that commit hash is found. * As a library user, you want to buy insurance for every library you use. If you experience a security breach, the money you get from the insurance will help you deal with the aftermath. * As an independent hacker, you can make passive income by auditing libraries and selling insurance for the ones that seem solid. If you identify a security flaw, buy up insurance for that library, then publicize the flaw for a big payday. * A distributed, anonymous marketplace is actually valuable here, because it encourages "insider trading" on the part of people who work for offensive cybersecurity orgs. Suppose Jane Hacker is working with a criminal org that's successfully penetrated a particular library. Suppose Jane wants to leave her life of crime behind. All she has to do is buy up insurance for the library that was penetrated and then anonymously disclose the vulnerability. * Even if you never trade on the insurance marketplace yourself, you can get a general idea of how risky a library is by checking how much its insurance costs. (Insurance might be subject to price manipulation by offensive cybersecurity orgs, but independent hackers would be incentivized to identify and correct such price manipulation.) The fact that there is actual value here should give the creator a huge advantage over other "Web 3.0" crypto junk.
- jschrf 5y ago
  This is a pretty clever application of DeFi, thanks. DeSec? Can't help but wonder if there still would be incentive for lone wolves to slip backdoors and vulnerabilities into libraries though[0]. [0]: https://portswigger.net/daily-swig/smuggling-hidden-backdoors-into-javascript-with-homoglyphs-and-invisible-unicode-characters https://portswigger.net/daily-swig/smuggling-hidden-backdoor...
Silhouette 5y ago
I'm seriously considering moving to a workflow of installing dependencies in containers or VMs, auditing them there, and then perhaps commiting known safe snapshots of node_modules into my repos (YUCK). Horrible developer experience, but at least it'll help me sleep at night. I have had people tell me in discussions online, also entirely seriously, that running a package manager to install a dependency while developing is inherently dangerous and anyone who does it outside of a disposable sandboxed VM deserves everything they get. If the packages are inexplicably allowed to do arbitrary things with privileged access to the local system without warning at installation time then clearly the first part is correct, but victim-blaming hardly seems like a useful reaction to that danger.

firebaze 5y ago

No CVE mentioned. Hard to grok to me, could someone educate me why this is missing in the blog post?

detaro 5y ago
services don't get CVEs.
- firebaze 5y ago
  Well, didn't we just experience two major npm published packages containing malware? Both had CVEs. Now we have the probable root cause, buried in a wall of text. No CVE.
  - detaro 5y ago
    Yes, because pure services don't get CVEs. CVEs are for distributed software.
    - firebaze 5y ago
      Isn't this the biggest security flaw in the package ecosystem ever? They don't even know when, if, who and when this was exploited, but maybe I didn't pay enough detail attention to the few paragraphs devoted to the real problem. So shoudn't we assume all NPM packages published prior to 2nd of November are compromised? And if so, shouldn't this deserve a CVE? (https://en.wikipedia.org/wiki/Common_Vulnerabilities_and_Exposures https://en.wikipedia.org/wiki/Common_Vulnerabilities_and_Exp...)
      - detaro 5y ago
        CVEs aren't usually assigned for "there might be something wrong", but only identified specific issues.
  - cjbprime 5y ago
    CVEs alert end users that they need to take action to apply updates. That's relevant when a specific npm package contained a known vulnerability. It's not relevant when the npm server contained a known vulnerability. There's nothing a user of npm can do to update the npm server. CVEs don't just mean "this is a big security problem".

bigiain 5y ago

That one, combined with the other “ability to read names of private packages, makes for the possibility of a really really sneaky attack. I wonder how many orgs treat their private npm packages with significantly less scrutiny than the public ones they rely on?

masklinn 5y ago

Having different services trust different (and unrelated) bits of the request is an immortal classic though, great stuff.

rbanffy 5y ago
The part that made sure the user could update the package could have at least check if the payload is about that package before passing it to the service that trusted it.