10 ms·

Oh no, not again a meditation on NPM supply chain attacks

173 points by theycameback 1y ago

rzwitserloot 1y ago
I see two ways to fight supply chain attacks: * The endless arms race. But, nevermind. It's been 2 years since Jia Tan and the amount of such 'occurrences' in the npm ecosystem in the past 10 years are bordering on uncountable at this point. And yet this hack got through? This amateuristic and extremely obvious attempt? The injected function was literally named something like 'raidBitcoinEthWallet' or whatnot. npm has clearly done absolutely nothing in this regard. We haven't even gotten to the argument of '... but then hackers will simply use the automated tools themselves and only release stuff that doesn't get flagged'. There's nothing to talk about; npm has done nothing. Which gets us to: * Web of trust This seems to me to be a near perfect way for the big companies that have earned so, so much money using the free FOSS they rely on, to contribute. They spend the cash to hire a team that reviews FOSS stuff. Entire libraries, sure, but also updates. I think most of them _already do this_, and many will even openly publish issues they found. But they do not publish negative results (they do not publish 'our internal team had a quick look at update XYZ of project ABC and didn't see anything immediately suspicious'). They should start doing that. And repos like npm, maven, CPAN, etcetera should allow either the official maintainer of a library, or anybody, to attach 'assertments of no malicious intent'. Imagine that npm hosts the following blob of text for NPM hosted projects in addition to the javascript artefacts: > "I, google dev/security team, hereby vouch for this update in the senses: not-malicious. We have looked at it and did not see anything that we think is suspicious or worse. We make absolutely no promises whatsoever that this library is any good or that this update's changelog accurately represents the changes in it; we merely vouch for the fact that we do not think it was written with malicious intent. We explicitly disavow any legal or financial liability with this statement; we merely stake our good name. We have done this analysis on 2025-09-17 and did it for the artefact with SHA512 hash 1238498123abac. Signed, [public/private key infrastructure based signature, google.com]. And a general rule that google.com/.well-known/vouch-public-key or whatever contains the public key so anybody can check. Aside from Jia Tan/xz (which always stops any attempt; Jia Tan/xz was so legendary, exactly how the fuck THIS still happens given that massive wakeup call boggles my mind!), every supply chain attack was pretty dang easy to spot; the problem was: Nobody was looking, and everybody configures their build scripts to pick up point updates immediately. We should update these to 'pick them up after a certain 'vouch' score is reached'. Where everybody can mess with their scoring tables (don't trust google? reduce the value of their vouch to 0). I think security-crucial 0day fixing updates will not be significantly hampered by this; generally big 0-days are big news and any update that comes out gets analysed to pieces. The vouch signatures would roll in within the hour after posting them.
cube00 1y ago
> The tools we use to build software are not secure by default, and almost all of the time, the companies that provide them are not held to account for the security of their products. The companies? More like the unpaid open source community volunteers who the Fortune 500 leech off contributing nothing in return except demands for free support, fixes and more features.
- mococa 1y ago
  This.
- austin-cheney 1y ago
  [flagged]
  - pavel_lishin 1y ago
    > Consider how many JavaScript developers are completely unemployable without that free software. Can you say more about this?
    - austin-cheney 1y ago
      [flagged]
      - tcoff91 1y ago
        I think that ultimately it’s the fault of the web platform. With just a bit of retraining those engineers that could not be productive without a ton of npm packages could ship an iPhone app written in Swift. JS’ standard library is abysmal.
        austin-cheney 1y ago
        This sounds like blaming the victim. How do you on one hand call these people engineers, as if they are engineering something, and then on the other hand blame everything else for their inability to perform? That is weird. Its just a software platform. Would you really blame society for being too harsh if doctors, lawyers, police, teachers cannot do their jobs? It is weird to see so many people blame the web platform for hostility when its so much less challenging than it used to be. The most common cause of these frustrations I encountered while working in JavaScript is that developers are educated in something that looks like A, but JavaScript is not A, there is no training for JavaScript/Web, so therefore JavaScript/Web is hostile. As a self-taught developer that never made sense to me.
Ronaldoxx721 1y ago
Yes
lupusreal 1y ago
The beatings will continue until JS dev culture reforms.
thombles 1y ago
I think if somebody wants to see library distribution channels tightened up they need to be very specific about what they would like to see changed and why it would be better, since it would appear that the status quo is serving what people actually want - being able to create and upload packages and update them when you want. > But right now there are still no signed dependencies and nothing stopping people using AI agents, or just plain old scripts, from creating thousands of junk or namesquatting repositories. This is as close as we get in this particular piece. So what's the alternative here exactly - do we want uploaders to sign up with Microsoft accounts? Some sort of developer vetting process? A curated lib store? I'm sure everybody will be thrilled if Microsoft does that to the JS ecosystem. (/s) I'm not seeing a great deal of difference between having someone's NPM creds and having someone's signing key. Let's make things better but let's also be precise, please.
- lukan 1y ago
  We treat code repositories as public infrastructure, but we don't want to pay for it, so corporations run them, with their profit interest in mind. This is the fundamental conflict, that I see. And one solution, more non profits as organisations behind them.
- cube00 1y ago
  > But right now there are still no signed dependencies Considering these attacks are stealing API tokens by running code on developer's machines; I don't see how signing helps, attackers will just steal the private keys and sign their malware with those.
  - deevus 1y ago
    Could they detect code running from a new IP address or location and ask for a 2FA code?
    - cube00 1y ago
      postinstall is running on the developer's machine, from an endpoint security perspective, it's the actual developer performing the malicious actions, their machine, their IP address and their location.
amiga386 1y ago
"No Way To Prevent This" Says Only Package Manager Where This Regularly Happens
- a4isms 1y ago
  Deeply underrated comment. You can peel back the layers of sarcasm like... An onion.
  - deleted 1y ago
    [deleted]
  - TZubiri 1y ago
    Btw, it's copypasta at this stage
    - a4isms 1y ago
      Anyone familiar with The Onion knows this, The Onion themselves repost the exact same thing every time there's a school shooting. Which emphasizes how regularly it happens, and therefore in turn I have no objection to a joke like this becoming copy pasta every time an NPM supply chain attack takes place. https://duckduckgo.com/?q=site%3Atheonion.com+%22no+way+to+prevent+this%22 https://duckduckgo.com/?q=site%3Atheonion.com+%22no+way+to+p...
- andrewl-hn 1y ago
  TBF it does happen to other package managers, too. There were similar attacks on PyPI and Rubygems (and maybe others). However, since npm is the largest one and has the most packages released, updated, and downloaded, it became the primary target. Similar to how computer viruses used to target Windows first and foremost due to its popularity. Also, smaller package managers tend to learn from these attacks on npm, and by the time the malware authors try to use similar types of attacks on them the registries already have mitigations in place.
  - shakna 1y ago
2d8a875f-39a2-4 1y ago
It's a stretch to pin blame on Microsoft. They're probably the reason the service is still up at all (TFA admits as much). In hindsight it's likely that all they wanted from the purchase was AI training material. At worst they're guilty of apathy, but that's no worse than the majority of npm ecosystem participants.
- mr90210 1y ago
  > It's a stretch to pin blame on Microsoft. They're probably the reason the service is still up at all. I reckon that the ecosystem would have been much healthier if NPM had not been kept running without the care it requires.
  - 2d8a875f-39a2-4 1y ago
    I did wonder about that. Maybe yeah. It's likely that several no-better forks would have sprung up right away.
- righthand 1y ago
  It’s NOT a stretch to blame Microsoft. How many billions have we spent chasing “AI”? These issues could have been easily solved if we spent the consideration on them. This has been going on well over a decade. Microsoft isn’t any better steward than the original teams. This issue has happened Plenty under Microsoft’s ownership.
  - 2d8a875f-39a2-4 1y ago
    i would contend that they are no worse than the original teams, who also clearly didn't care. their motivations may have been growth rather than AI training data but the outcomes were the same
    - righthand 1y ago
      Microsoft has money to fix the problem is the difference. Neither side (og team, Microsoft, or npm consumers) has any capital interest in the matter.
jamesnorden 1y ago
I think the cooldown approach would make this type of attack have practically no impact anymore, if nobody ever updates to a newly published package version until, say, 2-3 days have gone by, surely there will be enough time for owner of the package to notice he got pwnd.
- matusp 1y ago
  What about cases when the update fixes a security issue? Anybody using this approach would be a target for a few more days.
  - friendzis 1y ago
    I know it sounds preposterous but there there are more ways to apply patches than npm pull
    - a_victorp 1y ago
      Update package versions manually, you say? The audacity!
- deevus 1y ago
  I've never heard of this. It sounds like a solid default to me. If you _really_ need an update you can override it, but it should remain the default and not allow opting out.
  - deevus 1y ago
    https://github.com/pnpm/pnpm/issues/9921 https://github.com/pnpm/pnpm/issues/9921
    - artursapek 1y ago
      the funny thing about this is if everyone has the same cooldown, aren’t we back in the same boat? sure there are other ways for the package maintainer to notice they were pwned, but often they will not notice.
aj_g 1y ago
Anyone have a good solution to scan all code in our Github org for uses of the affected packages? Many of the methods we've tried have dead ended. Inability to reliably search branches is quite annoying here.
- ozim 1y ago
  Have you tried Dependency Track from OWASP? Generate SBOM from each repo/projects and post it with API to DT and you have full overview. You have to hook it up so it is done automatically because of course stuff will always move.
- TZubiri 1y ago
  Any junior engineer should be able to solve this with grep in an afternoon.
  - aj_g 1y ago
    For several thousand repos? Ensuring none of the 451 package versions have been installed on any branch in any repo? I don't think it's so simple.
    - deleted 1y ago
      [deleted]
    - TZubiri 1y ago
      aikido published a list of the affected vulns. You can probably get a list of the repos with a github API or something. Git clone with org admin user credentials (can be read only) so you have access to all the repos. run grep on all package.json files, search for all of the affected repos. No need to do any code regarding versions, just filter it down and manually process versions if needed. If you have any of these packages, no matter the versions, you should already be making efforts to migrating, kill the baby with the bathwater, cut off the arm before the gangrene spreads. At any rate you can check versions manually after you have filtered it down to something reasonable, part of automating is knowing when to stop.
- deleted 1y ago
righthand 1y ago
Here is an issue from 2013 where developers are asking to fix the package signing issue. Gone fully ignored because doing so was “too hard”: https://github.com/npm/npm/pull/4016 https://github.com/npm/npm/pull/4016
1oooqooq 1y ago
funny how npm is the exact same model as maven, gopkg, cpan, pip, mix, cargo, and a million others. but only npm started with a desire to monetize it (well, npm and docker hub) and in its desire for control didn't implement (or allowed the community to implement) basic higiene.
deevus 1y ago
pnpm have already implemented a minimum age policy https://github.com/pnpm/pnpm/issues/9921 https://github.com/pnpm/pnpm/issues/9921
homebrewer 1y ago
Here's a short recap of what you can do right now, because changing the ecosystem will take years, even if "we" bother to try doing it. 1. Switch to pnpm, it's not only faster and more space efficient, but also disables post-install scripts by default. Very few packages actually need those to function, most use it for spam and analytics. When you install packages into the project for the first time, it tells you what post-install scripts were skipped, and tells you how to whitelist only those you need. In most projects I don't enable any, and everything works fine. The "worst" projects required allowing two scripts, out of a couple dozen or so. They also added this recently, which lets you introduce delays for new versions when updating packages. Combined with `pnpm audit`, I think it can replace the last suggestion of setting up a helper dependency bot with zero reliance on additional services, commercial or not: https://pnpm.io/settings#minimumreleaseage https://pnpm.io/settings#minimumreleaseage 2. If you're on Linux, wrap your package managers into bubblewrap, which is a lightweight sandbox that will block access to almost all of your system, including sensitive files like ~/.ssh, and prevent anything running under it from escalating privileges. It's used by flatpak and Steam. A fully working & slightly improved version was posted here: https://news.ycombinator.com/item?id=45271988 https://news.ycombinator.com/item?id=45271988 I posted the original here, but it was somewhat broken because some flags were sorted incorrectly (mea culpa). I still prefer using a separate cache directory instead of sharing the "global" ~/.cache because sensitive information might also end up there. https://news.ycombinator.com/item?id=45041798 https://news.ycombinator.com/item?id=45041798 3. Setup renovate or any similar bot to introduce artificial delays into your supply chain, but also to fast-track fixes for publicly known vulnerabilities. This suggestion caused some unhappiness in the previous discussion for some reason — I really don't care which service you're using, this is not an ad, just setup something to track your dependencies because you will forget it. You can fully self-host it, I don't use their commercial offering — never has, don't plan to. https://docs.renovatebot.com/configuration-options/#minimumreleaseage https://docs.renovatebot.com/configuration-options/#minimumr... https://docs.renovatebot.com/presets-default/#enablevulnerabilityalerts https://docs.renovatebot.com/presets-default/#enablevulnerab... 4. For those truly paranoid or working on very juicy targets, you can always stick your work into a virtual machine, keeping secrets out of there, maybe with one virtual machine per project.
- 1y ago
apimade 1y ago
Here’s a one-liner for node devs on MacOS, pin your versions and manually update your supply chain until your tooling supports supply chain vetting, or at least some level of protection against instantly-updated malicious upstream packages. Would love to see some default-secure package management / repo options. Even a 24 hour delayed mirror would be better than than what we have today. find . -name package.json -not -path "/node_modules/" -exec sh -c ' for pkg; do lock="$(dirname "$pkg")/package-lock.json" [ -f "$lock" ] || continue tmp="$(mktemp)" jq --argfile lock "$lock" \ ".dependencies |= with_entries(.value = $lock.dependencies[.key].version) | .devDependencies |= with_entries(.value = $lock.dependencies[.key].version // $lock.devDependencies[.key].version)" \ "$pkg" > "$tmp" && mv "$tmp" "$pkg" done ' sh {} +
- treyd 1y ago
  The expected secure workflow should not require an elaborate bash incantation, it should be the workflow the tools naturally encourage you to use organically. "You're holding it wrong" cannot be possible.
- deleted 1y ago
  [deleted]
- madeofpalk 1y ago
  ? Package lock files from npm/yarn/pnpm automatically lock all your dependencies (including transitive deps) What does this actually achieve?
  - user34283 1y ago
    Maybe the same as if "npm config set save-exact true" was enabled when adding the dependencies. Whether that's so important, I'm not sure.
  - apimade 1y ago
    Accidentally installing a malicious package in your dev environment, the concern isn’t “what’s already installed”, it’s what’s potentially going to be installed in the future by you or your colleagues. So, you pin the version and update periodically when security issues arise in your dependencies.
outsideoftime 1y ago
[dead]
tcoff91 1y ago
It seems to me like one obvious improvement is for npm to require 2fa to submit packages. The fact that malware can just automatically publish packages without a human having to go through an MFA step is crazy.
raisaguys 1y ago
[flagged]
simonw 1y ago
Question for tanepiper: what would you have Microsoft do to improve things here? My read of your article is that you don't like postinstall scripts and npx. I'm not convinced that removing those would have a particularly major impact on supply chain attacks. The nature of npm is that it distributes code that is then executed. Even without npx, an attacker could still release an updated package which, when executed as a dependency, steals environment variables or similar. And in the meantime, discarding box would break the existing workflows of practically every JavaScript-using developer in the world! You mention 2FA. npm requires that for maintainers of the top 100 packages (since 2022), would you like to see that policy increased to the top 1,000/10,000/everyone? https://github.blog/security/supply-chain-security/top-100-npm-package-maintainers-require-2fa-additional-security/ https://github.blog/security/supply-chain-security/top-100-n... You also mention code signing. I like that too, but do you think that would have a material impact on supply chain attacks given they start with compromised accounts? The investment I most want to see around this topic is in sandboxing: I think it should be the default that all code runs in a robust sandbox unless there is as very convincing reason not to. That requires investment that goes beyond a single language and package management platform - it's something that needs to be available and trustworthy for multiple operating systems.
- rs186 1y ago
  Exactly. The biggest problem with npm is that it is too popular. Nothing else. Even if you "mitigate" some of the risks by removing features like postinstall, it barely does anything at all -- if you actually use the package in any way, the threat is still there. And most of what we see recently could happen to crates.io, pypi etc as well. It is almost frustrating to see people who don't understand security talk about security. They think they have the best, smartest ideas. No, they don't, otherwise they would have been done a long time ago. Security is hard, really hard.
  - singulasar 1y ago
    There's multiple security firms by now that constantly scan updated npm packages for malware. Obviously those companies can only do this after a new package has been published. Npm could add this as an automated step during publishing. Sure, there's a manual review needed for anything flagged, but you can easily fix this as well by having smth like a trusted contributor program where let's say you'd need 5 votes to overrule a package being flagged as malware
hu3 1y ago
My non-solution years ago was to use as little dependencies as possible. And vendor node_modules then review every line of code changed when I update dependencies. Not every project and team can do that. But when feasible, it's a strong mitigation layer. What worked was splitting dependency diff review among the team so it's less of a burden. We pin exact versions and update judiciously.
- bee_rider 1y ago
  That’s the solution. The whole theory of the many-eyes model is that lots of people will read the code. You are doing the work. These automatic library installing services seem to have a massive free-rider problem.
- user34283 1y ago
  You can't realistically do that when for example you use Jest as your test runner, which alone would add 300 packages. ESLint would be another culprit, adding 80 packages. It quickly gets out of hand. To me it seems like the fewest projects could use this approach you described.
  - hu3 1y ago
    You usually can. You just gotta be a bit adventurous. https://github.com/lukeed/uvu https://github.com/lukeed/uvu is a testing library with almost no dependency. https://github.com/biomejs/biome https://github.com/biomejs/biome is a linter written in Rust which in theory has a smaller attack surface. And as long as you stay some versions behind bleeding edge, you can use time in your favor to catch supply chain attacks before they reach your codebase.
    - user34283 1y ago
      Well, can you? Maybe you can. Or you're talking about an approach you utilized in some side projects rather than moderately sized commercial web applications? I don't imagine there's many out there that have less than hundreds of dependencies.
kronicum2025 1y ago
The title does not do justice to public course of thinking which is more like "Oh no, not again! Anyway, look at this shiny new tool."
curtisszmania 1y ago
[dead]
donatj 1y ago
Every day I feel more and more like Go mod's decision to use the lowest common version of a dependency rather than the highest was pure wisdom. Not only does it prevent code breaking at rest from poor semantic versioning, it's also served to prevent automatic inclusion of supply chain attacks. npm as designed really aggressively likes to upgrade things, and the culture is basically to always blindly upgrade all dependencies as high as possible. It's sold as being safer by patching vulnerabilities, but most "vulnerabilities" are very minor or niche, whereas a lot of risk is inherent in a shifting foundation. Like it or not it's kind of a cultural problem. Recursively including thousands of dependencies, all largely updating with no review is a problem. The thing I find particularly frightful and distinctive from the other package managers I regularly use is there is zero guarantee that the code a library presents on GitHub has anything to do with it's actual content in NPM. You can easily believe you've reviewed an items code by looking at it on GitHub, but that can have absolutely zero relation to what was actually uploaded to npm. You have to actually review what's been uploaded to npm as its entirety disconnected.
- jiggawatts 1y ago
  > You have to actually review what's been uploaded to npm Crates.io and several other popular package managers have the exact same problem. Submitted packages are essentially a blob of loose files with the source code being mere metadata provided by the uploader (or attacker!) The logic behind this is that not every package comes from a source repository that is based on Git and there may not be a convenient and trustworthy "web link" back to the matching commit. Some SCM systems don't even have cryptographically hashed commits with the same level of "stability" as a Git commit id! IMHO all such public package repositories should do their own Git hosting for the package file contents. I.e.: to publish you'd have to push your code to their repo instead of uploading files. Ideally they should also scan all uploads in various ways, run reproducible builds for platforms where that makes sense, etc...
jollyllama 1y ago
The NPM monoculture is the problem. It would be absurd to suggest that all backend engineers use the same build tooling and dependency library, but here we are with frontend. It's just too big of an attack surface.
- beart 1y ago
  It would be absurd to make such a suggestion. However, the comparison is not correct. Not all front-end development uses the same build tooling or dependency libraries, or programming language for that matter. Even if you narrow to the typescript ecosystem, it's still not true.
  - jollyllama 1y ago
    And yet, 99% of front end development uses NPM.
    - beart 1y ago
      According to https://tsh.io/state-of-frontend#package-manager https://tsh.io/state-of-frontend#package-manager, it's not quite that high.
ChrisArchitect 1y ago
Related: Shai-Hulud malware attack: Tinycolor and over 40 NPM packages compromised https://news.ycombinator.com/item?id=45260741 https://news.ycombinator.com/item?id=45260741
clarkdale 1y ago
By removing ellipsis in submission title, the sentiment feels more like "not another meditation" instead of the intent "oh no!: a meditation"
jacques_chester 1y ago
I disagree with theses in this piece. 1. "2FA doesn't work". Incorrect. MFA relying on SMS or TOTP is vulnerable to phishing. Token or device based is not. And indeed GitHub sponsored a program to give such tokens away to critical developers. In 2021. 2. "There's no signing". Sigstore support shipped in like 2023. The underlying view is that "Microsoft isn't doing anything". They have been. For years. Since at least 2022, based on my literal direct interactions with the individuals directly tasked to do the things that you say aren't or haven't been done. I have no association with npm, GitHub or Microsoft. My connection was through Shopify and RubyGems. But it really steams me to see npm getting punched up with easily checked "facts".
b_lax 1y ago
[dead]
amai 1y ago
"Unfortunately, Microsoft seem to be actively hostile - in their lack of attempts to shut down an active security hole that’s almost a decade old, they have left their customers are the higest levels of risk seen in computing." Well said.

173 points by theycameback 1y ago

I see two ways to fight supply chain attacks: * The endless arms race. But, nevermind. It's been 2 years since Jia Tan and the amount of such 'occurrences' in the npm ecosystem in the past 10 years are bordering on uncountable at this point. And yet this hack got through? This amateuristic and extremely obvious attempt? The injected function was literally named something like 'raidBitcoinEthWallet' or whatnot. npm has clearly done absolutely nothing in this regard. We haven't even gotten to the argument of '... but then hackers will simply use the automated tools themselves and only release stuff that doesn't get flagged'. There's nothing to talk about; npm has done nothing. Which gets us to: * Web of trust This seems to me to be a near perfect way for the big companies that have earned so, so much money using the free FOSS they rely on, to contribute. They spend the cash to hire a team that reviews FOSS stuff. Entire libraries, sure, but also updates. I think most of them _already do this_, and many will even openly publish issues they found. But they do not publish negative results (they do not publish 'our internal team had a quick look at update XYZ of project ABC and didn't see anything immediately suspicious'). They should start doing that. And repos like npm, maven, CPAN, etcetera should allow either the official maintainer of a library, or anybody, to attach 'assertments of no malicious intent'. Imagine that npm hosts the following blob of text for NPM hosted projects in addition to the javascript artefacts: > "I, google dev/security team, hereby vouch for this update in the senses: not-malicious. We have looked at it and did not see anything that we think is suspicious or worse. We make absolutely no promises whatsoever that this library is any good or that this update's changelog accurately represents the changes in it; we merely vouch for the fact that we do not think it was written with malicious intent. We explicitly disavow any legal or financial liability with this statement; we merely stake our good name. We have done this analysis on 2025-09-17 and did it for the artefact with SHA512 hash 1238498123abac. Signed, [public/private key infrastructure based signature, google.com]. And a general rule that google.com/.well-known/vouch-public-key or whatever contains the public key so anybody can check. Aside from Jia Tan/xz (which always stops any attempt; Jia Tan/xz was so legendary, exactly how the fuck THIS still happens given that massive wakeup call boggles my mind!), every supply chain attack was pretty dang easy to spot; the problem was: Nobody was looking, and everybody configures their build scripts to pick up point updates immediately. We should update these to 'pick them up after a certain 'vouch' score is reached'. Where everybody can mess with their scoring tables (don't trust google? reduce the value of their vouch to 0). I think security-crucial 0day fixing updates will not be significantly hampered by this; generally big 0-days are big news and any update that comes out gets analysed to pieces. The vouch signatures would roll in within the hour after posting them.

cube00 1y ago

> The tools we use to build software are not secure by default, and almost all of the time, the companies that provide them are not held to account for the security of their products. The companies? More like the unpaid open source community volunteers who the Fortune 500 leech off contributing nothing in return except demands for free support, fixes and more features.

mococa 1y ago
This.
austin-cheney 1y ago
[flagged]
- pavel_lishin 1y ago
  > Consider how many JavaScript developers are completely unemployable without that free software. Can you say more about this?
  - austin-cheney 1y ago
    [flagged]
    - tcoff91 1y ago
      I think that ultimately it’s the fault of the web platform. With just a bit of retraining those engineers that could not be productive without a ton of npm packages could ship an iPhone app written in Swift. JS’ standard library is abysmal.
      - austin-cheney 1y ago
        This sounds like blaming the victim. How do you on one hand call these people engineers, as if they are engineering something, and then on the other hand blame everything else for their inability to perform? That is weird. Its just a software platform. Would you really blame society for being too harsh if doctors, lawyers, police, teachers cannot do their jobs? It is weird to see so many people blame the web platform for hostility when its so much less challenging than it used to be. The most common cause of these frustrations I encountered while working in JavaScript is that developers are educated in something that looks like A, but JavaScript is not A, there is no training for JavaScript/Web, so therefore JavaScript/Web is hostile. As a self-taught developer that never made sense to me.

Ronaldoxx721 1y ago

Yes

lupusreal 1y ago

The beatings will continue until JS dev culture reforms.

thombles 1y ago

I think if somebody wants to see library distribution channels tightened up they need to be very specific about what they would like to see changed and why it would be better, since it would appear that the status quo is serving what people actually want - being able to create and upload packages and update them when you want. > But right now there are still no signed dependencies and nothing stopping people using AI agents, or just plain old scripts, from creating thousands of junk or namesquatting repositories. This is as close as we get in this particular piece. So what's the alternative here exactly - do we want uploaders to sign up with Microsoft accounts? Some sort of developer vetting process? A curated lib store? I'm sure everybody will be thrilled if Microsoft does that to the JS ecosystem. (/s) I'm not seeing a great deal of difference between having someone's NPM creds and having someone's signing key. Let's make things better but let's also be precise, please.

lukan 1y ago
We treat code repositories as public infrastructure, but we don't want to pay for it, so corporations run them, with their profit interest in mind. This is the fundamental conflict, that I see. And one solution, more non profits as organisations behind them.
cube00 1y ago
> But right now there are still no signed dependencies Considering these attacks are stealing API tokens by running code on developer's machines; I don't see how signing helps, attackers will just steal the private keys and sign their malware with those.
- deevus 1y ago
  Could they detect code running from a new IP address or location and ask for a 2FA code?
  - cube00 1y ago
    postinstall is running on the developer's machine, from an endpoint security perspective, it's the actual developer performing the malicious actions, their machine, their IP address and their location.

amiga386 1y ago

"No Way To Prevent This" Says Only Package Manager Where This Regularly Happens

a4isms 1y ago
Deeply underrated comment. You can peel back the layers of sarcasm like... An onion.
- deleted 1y ago
  [deleted]
- TZubiri 1y ago
  Btw, it's copypasta at this stage
  - a4isms 1y ago
    Anyone familiar with The Onion knows this, The Onion themselves repost the exact same thing every time there's a school shooting. Which emphasizes how regularly it happens, and therefore in turn I have no objection to a joke like this becoming copy pasta every time an NPM supply chain attack takes place. https://duckduckgo.com/?q=site%3Atheonion.com+%22no+way+to+prevent+this%22 https://duckduckgo.com/?q=site%3Atheonion.com+%22no+way+to+p...
andrewl-hn 1y ago
TBF it does happen to other package managers, too. There were similar attacks on PyPI and Rubygems (and maybe others). However, since npm is the largest one and has the most packages released, updated, and downloaded, it became the primary target. Similar to how computer viruses used to target Windows first and foremost due to its popularity. Also, smaller package managers tend to learn from these attacks on npm, and by the time the malware authors try to use similar types of attacks on them the registries already have mitigations in place.
- shakna 1y ago

2d8a875f-39a2-4 1y ago

It's a stretch to pin blame on Microsoft. They're probably the reason the service is still up at all (TFA admits as much). In hindsight it's likely that all they wanted from the purchase was AI training material. At worst they're guilty of apathy, but that's no worse than the majority of npm ecosystem participants.

mr90210 1y ago
> It's a stretch to pin blame on Microsoft. They're probably the reason the service is still up at all. I reckon that the ecosystem would have been much healthier if NPM had not been kept running without the care it requires.
- 2d8a875f-39a2-4 1y ago
  I did wonder about that. Maybe yeah. It's likely that several no-better forks would have sprung up right away.
righthand 1y ago
It’s NOT a stretch to blame Microsoft. How many billions have we spent chasing “AI”? These issues could have been easily solved if we spent the consideration on them. This has been going on well over a decade. Microsoft isn’t any better steward than the original teams. This issue has happened Plenty under Microsoft’s ownership.
- 2d8a875f-39a2-4 1y ago
  i would contend that they are no worse than the original teams, who also clearly didn't care. their motivations may have been growth rather than AI training data but the outcomes were the same
  - righthand 1y ago
    Microsoft has money to fix the problem is the difference. Neither side (og team, Microsoft, or npm consumers) has any capital interest in the matter.

jamesnorden 1y ago

I think the cooldown approach would make this type of attack have practically no impact anymore, if nobody ever updates to a newly published package version until, say, 2-3 days have gone by, surely there will be enough time for owner of the package to notice he got pwnd.

matusp 1y ago
What about cases when the update fixes a security issue? Anybody using this approach would be a target for a few more days.
- friendzis 1y ago
  I know it sounds preposterous but there there are more ways to apply patches than npm pull
  - a_victorp 1y ago
    Update package versions manually, you say? The audacity!
deevus 1y ago
I've never heard of this. It sounds like a solid default to me. If you _really_ need an update you can override it, but it should remain the default and not allow opting out.
- deevus 1y ago
  https://github.com/pnpm/pnpm/issues/9921 https://github.com/pnpm/pnpm/issues/9921
  - artursapek 1y ago
    the funny thing about this is if everyone has the same cooldown, aren’t we back in the same boat? sure there are other ways for the package maintainer to notice they were pwned, but often they will not notice.

aj_g 1y ago

Anyone have a good solution to scan all code in our Github org for uses of the affected packages? Many of the methods we've tried have dead ended. Inability to reliably search branches is quite annoying here.

ozim 1y ago
Have you tried Dependency Track from OWASP? Generate SBOM from each repo/projects and post it with API to DT and you have full overview. You have to hook it up so it is done automatically because of course stuff will always move.
TZubiri 1y ago
Any junior engineer should be able to solve this with grep in an afternoon.
- aj_g 1y ago
  For several thousand repos? Ensuring none of the 451 package versions have been installed on any branch in any repo? I don't think it's so simple.
  - deleted 1y ago
    [deleted]
  - TZubiri 1y ago
    aikido published a list of the affected vulns. You can probably get a list of the repos with a github API or something. Git clone with org admin user credentials (can be read only) so you have access to all the repos. run grep on all package.json files, search for all of the affected repos. No need to do any code regarding versions, just filter it down and manually process versions if needed. If you have any of these packages, no matter the versions, you should already be making efforts to migrating, kill the baby with the bathwater, cut off the arm before the gangrene spreads. At any rate you can check versions manually after you have filtered it down to something reasonable, part of automating is knowing when to stop.
deleted 1y ago

righthand 1y ago

Here is an issue from 2013 where developers are asking to fix the package signing issue. Gone fully ignored because doing so was “too hard”: https://github.com/npm/npm/pull/4016 https://github.com/npm/npm/pull/4016

1oooqooq 1y ago

funny how npm is the exact same model as maven, gopkg, cpan, pip, mix, cargo, and a million others. but only npm started with a desire to monetize it (well, npm and docker hub) and in its desire for control didn't implement (or allowed the community to implement) basic higiene.

deevus 1y ago

pnpm have already implemented a minimum age policy https://github.com/pnpm/pnpm/issues/9921 https://github.com/pnpm/pnpm/issues/9921

homebrewer 1y ago

Here's a short recap of what you can do right now, because changing the ecosystem will take years, even if "we" bother to try doing it. 1. Switch to pnpm, it's not only faster and more space efficient, but also disables post-install scripts by default. Very few packages actually need those to function, most use it for spam and analytics. When you install packages into the project for the first time, it tells you what post-install scripts were skipped, and tells you how to whitelist only those you need. In most projects I don't enable any, and everything works fine. The "worst" projects required allowing two scripts, out of a couple dozen or so. They also added this recently, which lets you introduce delays for new versions when updating packages. Combined with `pnpm audit`, I think it can replace the last suggestion of setting up a helper dependency bot with zero reliance on additional services, commercial or not: https://pnpm.io/settings#minimumreleaseage https://pnpm.io/settings#minimumreleaseage 2. If you're on Linux, wrap your package managers into bubblewrap, which is a lightweight sandbox that will block access to almost all of your system, including sensitive files like ~/.ssh, and prevent anything running under it from escalating privileges. It's used by flatpak and Steam. A fully working & slightly improved version was posted here: https://news.ycombinator.com/item?id=45271988 https://news.ycombinator.com/item?id=45271988 I posted the original here, but it was somewhat broken because some flags were sorted incorrectly (mea culpa). I still prefer using a separate cache directory instead of sharing the "global" ~/.cache because sensitive information might also end up there. https://news.ycombinator.com/item?id=45041798 https://news.ycombinator.com/item?id=45041798 3. Setup renovate or any similar bot to introduce artificial delays into your supply chain, but also to fast-track fixes for publicly known vulnerabilities. This suggestion caused some unhappiness in the previous discussion for some reason — I really don't care which service you're using, this is not an ad, just setup something to track your dependencies because you will forget it. You can fully self-host it, I don't use their commercial offering — never has, don't plan to. https://docs.renovatebot.com/configuration-options/#minimumreleaseage https://docs.renovatebot.com/configuration-options/#minimumr... https://docs.renovatebot.com/presets-default/#enablevulnerabilityalerts https://docs.renovatebot.com/presets-default/#enablevulnerab... 4. For those truly paranoid or working on very juicy targets, you can always stick your work into a virtual machine, keeping secrets out of there, maybe with one virtual machine per project.

1y ago

apimade 1y ago

Here’s a one-liner for node devs on MacOS, pin your versions and manually update your supply chain until your tooling supports supply chain vetting, or at least some level of protection against instantly-updated malicious upstream packages. Would love to see some default-secure package management / repo options. Even a 24 hour delayed mirror would be better than than what we have today. find . -name package.json -not -path "/node_modules/" -exec sh -c ' for pkg; do lock="$(dirname "$pkg")/package-lock.json" [ -f "$lock" ] || continue tmp="$(mktemp)" jq --argfile lock "$lock" \ ".dependencies |= with_entries(.value = $lock.dependencies[.key].version) | .devDependencies |= with_entries(.value = $lock.dependencies[.key].version // $lock.devDependencies[.key].version)" \ "$pkg" > "$tmp" && mv "$tmp" "$pkg" done ' sh {} +

treyd 1y ago
The expected secure workflow should not require an elaborate bash incantation, it should be the workflow the tools naturally encourage you to use organically. "You're holding it wrong" cannot be possible.
deleted 1y ago
[deleted]
madeofpalk 1y ago
? Package lock files from npm/yarn/pnpm automatically lock all your dependencies (including transitive deps) What does this actually achieve?
- user34283 1y ago
  Maybe the same as if "npm config set save-exact true" was enabled when adding the dependencies. Whether that's so important, I'm not sure.
- apimade 1y ago
  Accidentally installing a malicious package in your dev environment, the concern isn’t “what’s already installed”, it’s what’s potentially going to be installed in the future by you or your colleagues. So, you pin the version and update periodically when security issues arise in your dependencies.

outsideoftime 1y ago

[dead]

tcoff91 1y ago

It seems to me like one obvious improvement is for npm to require 2fa to submit packages. The fact that malware can just automatically publish packages without a human having to go through an MFA step is crazy.

raisaguys 1y ago

[flagged]

simonw 1y ago

Question for tanepiper: what would you have Microsoft do to improve things here? My read of your article is that you don't like postinstall scripts and npx. I'm not convinced that removing those would have a particularly major impact on supply chain attacks. The nature of npm is that it distributes code that is then executed. Even without npx, an attacker could still release an updated package which, when executed as a dependency, steals environment variables or similar. And in the meantime, discarding box would break the existing workflows of practically every JavaScript-using developer in the world! You mention 2FA. npm requires that for maintainers of the top 100 packages (since 2022), would you like to see that policy increased to the top 1,000/10,000/everyone? https://github.blog/security/supply-chain-security/top-100-npm-package-maintainers-require-2fa-additional-security/ https://github.blog/security/supply-chain-security/top-100-n... You also mention code signing. I like that too, but do you think that would have a material impact on supply chain attacks given they start with compromised accounts? The investment I most want to see around this topic is in sandboxing: I think it should be the default that all code runs in a robust sandbox unless there is as very convincing reason not to. That requires investment that goes beyond a single language and package management platform - it's something that needs to be available and trustworthy for multiple operating systems.

rs186 1y ago
Exactly. The biggest problem with npm is that it is too popular. Nothing else. Even if you "mitigate" some of the risks by removing features like postinstall, it barely does anything at all -- if you actually use the package in any way, the threat is still there. And most of what we see recently could happen to crates.io, pypi etc as well. It is almost frustrating to see people who don't understand security talk about security. They think they have the best, smartest ideas. No, they don't, otherwise they would have been done a long time ago. Security is hard, really hard.
- singulasar 1y ago
  There's multiple security firms by now that constantly scan updated npm packages for malware. Obviously those companies can only do this after a new package has been published. Npm could add this as an automated step during publishing. Sure, there's a manual review needed for anything flagged, but you can easily fix this as well by having smth like a trusted contributor program where let's say you'd need 5 votes to overrule a package being flagged as malware

hu3 1y ago

My non-solution years ago was to use as little dependencies as possible. And vendor node_modules then review every line of code changed when I update dependencies. Not every project and team can do that. But when feasible, it's a strong mitigation layer. What worked was splitting dependency diff review among the team so it's less of a burden. We pin exact versions and update judiciously.

bee_rider 1y ago
That’s the solution. The whole theory of the many-eyes model is that lots of people will read the code. You are doing the work. These automatic library installing services seem to have a massive free-rider problem.
user34283 1y ago
You can't realistically do that when for example you use Jest as your test runner, which alone would add 300 packages. ESLint would be another culprit, adding 80 packages. It quickly gets out of hand. To me it seems like the fewest projects could use this approach you described.
- hu3 1y ago
  You usually can. You just gotta be a bit adventurous. https://github.com/lukeed/uvu https://github.com/lukeed/uvu is a testing library with almost no dependency. https://github.com/biomejs/biome https://github.com/biomejs/biome is a linter written in Rust which in theory has a smaller attack surface. And as long as you stay some versions behind bleeding edge, you can use time in your favor to catch supply chain attacks before they reach your codebase.
  - user34283 1y ago
    Well, can you? Maybe you can. Or you're talking about an approach you utilized in some side projects rather than moderately sized commercial web applications? I don't imagine there's many out there that have less than hundreds of dependencies.

kronicum2025 1y ago

The title does not do justice to public course of thinking which is more like "Oh no, not again! Anyway, look at this shiny new tool."

curtisszmania 1y ago

[dead]

donatj 1y ago

Every day I feel more and more like Go mod's decision to use the lowest common version of a dependency rather than the highest was pure wisdom. Not only does it prevent code breaking at rest from poor semantic versioning, it's also served to prevent automatic inclusion of supply chain attacks. npm as designed really aggressively likes to upgrade things, and the culture is basically to always blindly upgrade all dependencies as high as possible. It's sold as being safer by patching vulnerabilities, but most "vulnerabilities" are very minor or niche, whereas a lot of risk is inherent in a shifting foundation. Like it or not it's kind of a cultural problem. Recursively including thousands of dependencies, all largely updating with no review is a problem. The thing I find particularly frightful and distinctive from the other package managers I regularly use is there is zero guarantee that the code a library presents on GitHub has anything to do with it's actual content in NPM. You can easily believe you've reviewed an items code by looking at it on GitHub, but that can have absolutely zero relation to what was actually uploaded to npm. You have to actually review what's been uploaded to npm as its entirety disconnected.

jiggawatts 1y ago
> You have to actually review what's been uploaded to npm Crates.io and several other popular package managers have the exact same problem. Submitted packages are essentially a blob of loose files with the source code being mere metadata provided by the uploader (or attacker!) The logic behind this is that not every package comes from a source repository that is based on Git and there may not be a convenient and trustworthy "web link" back to the matching commit. Some SCM systems don't even have cryptographically hashed commits with the same level of "stability" as a Git commit id! IMHO all such public package repositories should do their own Git hosting for the package file contents. I.e.: to publish you'd have to push your code to their repo instead of uploading files. Ideally they should also scan all uploads in various ways, run reproducible builds for platforms where that makes sense, etc...

jollyllama 1y ago

The NPM monoculture is the problem. It would be absurd to suggest that all backend engineers use the same build tooling and dependency library, but here we are with frontend. It's just too big of an attack surface.

beart 1y ago
It would be absurd to make such a suggestion. However, the comparison is not correct. Not all front-end development uses the same build tooling or dependency libraries, or programming language for that matter. Even if you narrow to the typescript ecosystem, it's still not true.
- jollyllama 1y ago
  And yet, 99% of front end development uses NPM.
  - beart 1y ago
    According to https://tsh.io/state-of-frontend#package-manager https://tsh.io/state-of-frontend#package-manager, it's not quite that high.

ChrisArchitect 1y ago

Related: Shai-Hulud malware attack: Tinycolor and over 40 NPM packages compromised https://news.ycombinator.com/item?id=45260741 https://news.ycombinator.com/item?id=45260741

clarkdale 1y ago

By removing ellipsis in submission title, the sentiment feels more like "not another meditation" instead of the intent "oh no!: a meditation"

jacques_chester 1y ago

I disagree with theses in this piece. 1. "2FA doesn't work". Incorrect. MFA relying on SMS or TOTP is vulnerable to phishing. Token or device based is not. And indeed GitHub sponsored a program to give such tokens away to critical developers. In 2021. 2. "There's no signing". Sigstore support shipped in like 2023. The underlying view is that "Microsoft isn't doing anything". They have been. For years. Since at least 2022, based on my literal direct interactions with the individuals directly tasked to do the things that you say aren't or haven't been done. I have no association with npm, GitHub or Microsoft. My connection was through Shopify and RubyGems. But it really steams me to see npm getting punched up with easily checked "facts".

b_lax 1y ago

[dead]

amai 1y ago

"Unfortunately, Microsoft seem to be actively hostile - in their lack of attempts to shut down an active security hole that’s almost a decade old, they have left their customers are the higest levels of risk seen in computing." Well said.