10 ms·
GitLab discovers widespread NPM supply chain attack
- ChrisArchitect 11mo agoDiscussion: https://news.ycombinator.com/item?id=46032539 https://news.ycombinator.com/item?id=46032539
- ares623 11mo agoPhew, thought it was another one.
- gchamonlive 11mo ago> Our internal monitoring system has uncovered multiple infected packages containing what appears to be an evolved version of the "Shai-Hulud" malware. Although it's not entirely new, it's something else.
- prophesi 11mo agoGitlab's post and the linked discussion thread are both from November 24th 2025. I may be misreading the parent comment, but I'm personally thankful there isn't a Return of the Return of Shai-Hulud, as I assumed this was a third recent incident. For those concerned about these attacks, Helixguard's post (from the linked discussion) lists out the packages they found to be effected, while Gitlab's post gives more information on how the attack works. Since it's self-propagating though, assume the list of affected packages might be longer as more NPM tokens are compromised.
- TZubiri 11mo agoNot all the npm packages, but always an npm package
- cyanydeez 11mo agoWhile you think this is a producer problem, it's simply a userland market. Just like in the 90s when viruses primarily went to windows, it' wasn't some magical property of windows, it was the market of users available. Also, following this logic, it then becomes survivorship bias, in that the more attacks they get, the more researchers spend time looking & documenting.
- TZubiri 11mo agoright, npm users. The extreme demand for simple packages and the absent consideration creates an opportunity for attackers to insert "free" solutions. The problem are the 'npm install' happy developers no doubt.
- KevinMS 11mo ago> it' wasn't some magical property of windows no, it really was windows
- foobiekr 11mo agoIt really wasn't. MacOS classic was full of vulnerabilities as was OS/2 and Linux up through 2004. Windows dominated because it was the biggest ecosystem.
- elwebmaster 11mo agoAnd had the highest proportion of ignorant users.
- ndsipa_pomu 11mo agoWhat made Windows easy to exploit was that it enabled a bunch of network services by default. I don't know about MacOS, but Linux disabled network services by default and generally had a better grasp of network security such as requiring authentication for services (e.g. compare telnet and ssh). Also, Windows had the ridiculous default of immediately running things when a user put in a CD or USB stick - that behaviour led to many infections and is obviously a stupid default option. I'm not even going to mention the old Windows design of everyone running with admin privileges on their desktop.
- Incipient 11mo agoSurely in this day and age we can fairly trivially find out these come from the usual suspects - China, Russia, Iran, etc. Being in such a digital age, where our economies are built on this tech...is this not effectively (economic) warfare? Why are so many governments blase about it?
- Nextgrid 11mo agoProving the attack is state-sponsored is difficult (as any attack you attribute to a country can very well be a false-flag operation), and “state sponsorship” is itself a spectrum; for example, you could argue India’s insufficient action against tech-support scammers is effectively state-sanctioned. This can of course be resolved, but here’s the kicker: our own governments equally enjoy this ambiguity to do their own bidding; so no government truly has an incentive to actually improve cross-border identity verification and cybercrime enforcement. Not to mention, even besides government involvement, these malicious actors still “engage” or induce “engagement” which happens to be the de-facto currency of the technology industry, so even businesses don’t actually have any incentive of fighting them.
- mc32 11mo agoA one or two off can be a false flag, thousand upon thousands is not going to be a false flag.
- halJordan 11mo agoIt shouldn't be a "get the foreigners!" situation. Sure that is a method of solving the symptoms. But what you're really asking for is ... a software bill of materials. Why dont we have that yet? Bc it's cheaper to get ripped off than it is to pay for a bom. Thats the real problem
- c0balt 11mo agoSBOMs exist. You can get them generated for most software via package managers in standard forms like cyclonedx. It's just not that effective when the SBOM becomes unmanageable. For example, our JS project at $work has 2.3k dependencies just from npm. I can give you that SBOM (and even include the system deps with nix) but that won't really help you. They are only really effective when the size is reasonable.
- yupyupyups 11mo agoSomething helpful here would be to enable developers to optionally identify themselves. Not Discord-style where only the platform knows their real identity, but publically as well.
- gruez 11mo agoSo, EV code signing certificates? Windows has that, and it'll verify that right in the OS. Git for instance, shows as being signed by CN = Johannes Schindelin O = Johannes Schindelin S = Nordrhein-Westfalen C = DE Downside is the cost. Certificates cost hundreds of dollars per year. There's probably some room to reduce cost, but not by much. You also run into issues of paying some homeless person $50 to use their identity for cyber crimes.
- wiradikusuma 11mo agoDoes anyone know why NPM seems to be the only attractive target? Python and Java are very popular, but I haven't heard anything in those ecosystems for a while. Is it because something inherently "weak" about NPM, or simply because, like Windows or JavaScript, everyone uses it?
- parliament32 11mo agoLarger attack surface (JS has been the #1 language on GitHub for years now) and more amateur developers (who are more likely to blindly install dependencies, not harden against dev attack vectors, etc).
- dboreham 11mo agoAlso: a culture of constant churn in libraries which in combination with the potential for security bugs to be fixed in any new release leads to a common practice of ingesting a continual stream of mystery meat. That makes filtering out malware very hard. Too much noise to see the signal. None of the above cultural factors is present in the other ecosystems.
- Sophira 11mo agoUnfortunately, blindly installing dependencies at compile-time is something that many projects will do by default nowadays. It's not just "more amateur developers" who are at risk here. I've even seen "setup scripts" for projects that will use root (with your permission) to install software. Such scripts are less common now with containers, but unfortunately containers aren't everything.
- 1718627440 11mo ago> blindly installing dependencies at compile-time is something that many projects will do by default nowadays. I consider this to be a sign that someone is still an amateur, and this is a reason to not use the software and quickly delete it. If you need a dependency, you can call the OS package manager, or tell me to compile it myself. If you start a network connection, you are malware in my eyes.
- thepasswordapp 11mo agoThe credential harvesting aspect is what concerns me most for the average developer. If you've ever run `npm install` on an affected package, your environment variables, .npmrc tokens, and potentially other cached credentials may have been exfiltrated. The action item for anyone potentially affected: rotate your npm tokens, GitHub PATs, and any API keys that were in environment variables. And if you're like most developers and reused any of those passwords elsewhere... rotate those too. This is why periodic credential rotation matters - not just after a breach notification, but proactively. It reduces the window where any stolen credential is useful.
- Towaway69 11mo ago> anyone potentially affected How does one know one is affected? What's the point of rotating tokens if I'm not sure that I've been affected - the new tokens will just be ex-filtrated as well. First step would be to identify infection, then clean up and then rotate tokens.
- mcintyre1994 11mo agoThe article has some indicators of compromise, the main one locally would be .truffler-cache/ in the home directory. It’s more obvious for package maintainers with exposed credentials, who will have a wormed version of their own packages deployed. From what I’ve read so far (and this definitely could change), it doesn’t install persistent malware, it relies on a postinstall script. So new tokens wouldn’t be automatically exfiltrated, but if you npm install any of an increasing number of packages then it will happen to you again.
- sierra1011 11mo agoIt does install a GitHub runner and registers the infected machine as a runner, so remote code execution remains possible. It might be a stretch to call it persistent but it definitely tries.
- dawnerd 11mo agoAlso a good reminder that you should be storing secrets in some kind of locker, not in plain text via environment variables or config files. Impossible to get everyone on board but if you can you should as much as possible. I hate that high profile services still default to plain text for credential storage.
- dmitrygr 11mo agoLucky for us C programmers. Each distro provides its own trusted libc, and my code has no other dependencies. :)
- TheTxT 11mo agoBut how do you left pad a string?
- deleted 11mo ago[deleted]
- 1718627440 11mo agochar * left_pad (const char * string, unsigned int pad) { char tmp[strlen (string)+pad+1]; memset (tmp, ' ', pad); strcpy (tmp+pad, string); return strdup (tmp); } Doesn't sound too hard in my opinion. This only works for strings, that fit on the stack, so if you want to make it robust, you should check for the string size. It (like everything in C) can of course fail. Also it is a quite naive implementation, since it calculates the string size three times.
- brabel 11mo agoNot a C expert but you’re using a dynamic array right on the stack, and then returning the duplicate of that. Shouldn’t that be Malloc’ed instead?? Is it safe to return the duplicate of a stack allocated array, wouldn’t the copy be heap allocated anyway? Not to mention it blows the stack and you get segmentation fault?
- Aeolun 11mo agoI thought this was a really insightful post, until they used it to try and sell me on Gitlab’s security features.
- jaggirs 11mo agoWhy would that make it any less insightfull?
- hu3 11mo agoBecause bias and incentives matter. There's a reason disclosures are obligatory in academic papers.
- baq 11mo agoIt’s published on gitlab.com, not arxiv
- rockskon 11mo agoIt's almost like the speakers are motivated by advertising a product to solve a problem in their own garden.
- serial_dev 11mo agoThey pulled a little sneaky on ya, mentioning GitLab security features available to GitLab users in a GitLab Security blog post with GitLab logos everywhere. Call me a conspiracy theorist, but I start to think these people might be affiliated with GitLab.
- TeMPOraL 11mo agoIt's more like, you don't know where honest technical evaluation ends, and an ad starts.
- hiccuphippo 11mo ago
- xyzal 11mo agoOkay ... what best practices should I as a mere dev follow to be protected? Is the "cooldown" approach enough, or should every npm command be run in bubblewrap ... ?
- mcintyre1994 11mo agoIn this narrow case, using pnpm or something similar that blocks postinstall scripts by default should be sufficient. In general, you probably want to use a container/vm/sandbox of some sort so dev stuff can’t access anything else on your machine.
- dawnerd 11mo agoEveryone is blaming npm but GitHub should be put on blast too for allowing the repos to be created and not quickly flagged. GitHub has a massive malware problem as it is and it doesn’t get enough attention.
- benatkin 11mo agoThey're part of the same company, but that's a good point. They both have mediocre security.
- princevegeta89 11mo agoI love! how Github, as a corporate company now owned by Microsoft, is directly tied to GoLang as the main repository of the vast majority of packages/dependencies. Imagine the number of things that can go wrong when they try to regulate or introduce restrictions for build workflows for the purpose of making some extra money... lol The original Java platform is a good example to think about.
- oefrha 11mo agoGolang builds pulling a github.com/foo/bar/baz module don't rely on any GitHub "build workflow", so unless you mean they're going to start restricting or charging for git clones for public repos (before you mention Docker Hub, yes I know), nothing's gonna change. And even if they're crazy enough to do that, Go module downloads default to a proxy (proxy.golang.org by default, can be configured and/or self-hosted) and only fall back to vcs if the module's not available, so a module only needs to be downloaded once from GitHub anyway. Oh and once a module is cached in the proxy, the proxy will keep serving it even if the repo/tag is removed from GitHub.
- Cthulhu_ 11mo ago"The original Java platform" had no package management though, that came with Maven and later Gradle, that have similar vectors for supply chain attacks (that is, nobody reviews anything before it's made available on package repositories). And (to put on my Go defender hat), the Go ecosystem doesn't like having many dependencies, in part because of supply chain attack vectors and the fact that Node's ecosystem went a bit overboard with libraries.
- AmbroseBierce 11mo agoMicrosoft should just bite the bullet and make a huge JS standard library and then send GitHub notifications to all the project maintainers who are using anything that could be replaced by something from there suggesting them to do such replacement. This would likely significantly reduce the number of supply chain attacks on the npm ecosystem.
- testdelacc1 11mo agoThis is harder than it sounds. Look at the amount of effort it took to standardise temporal (new time library) and then for all the runtimes to implement it. It’s a lot of work. And what’s more, people have proposed a standard library through tc39 without success - https://github.com/tc39/proposal-built-in-modules https://github.com/tc39/proposal-built-in-modules Of course any large company could create a massive standard library on their own without going through the standards process but it might not be adopted by developers.
- nottorp 11mo agoThere's an xckd for that :) The one with 12 competing standards going to 13 competing standards, or something like that.
- latexr 11mo agohttps://xkcd.com/927/ https://xkcd.com/927/
- AmbroseBierce 11mo agoPretty sure Microsoft is exponentially bigger than 99% of the library authors out there, and add to that the giant communication channel that GitHub gives it over developers, so the analogy breaks pretty fast.
- nottorp 11mo agoOr it's worse, because there's a good bunch of devs that don't trust MS by default?
- wonderfuly 11mo agoI'm a victim of this. In addition to concerns about npm, I'm now hesitant to use the GitHub CLI, which stores a highly privileged OAuth token in plain text in the HOME directory. After the attacker accesses it, they can do almost anything on behalf of me, for example, they turned many of my private repos to public.
- febusravenga 11mo agothis, this, this All our tokens should be in is protected keychain and there are no proper cross-platform solutions for this. All gclouds, was aww sdks, gh and other tools just store them in dotfile. And worst thing, afaik there is no way do do it correctly in MacOS for example. I'd like to be corrected though.
- mcny 11mo agoWhat is a proper solution for this? I don't imagine gpg can help if you encrypt it but decrypt it when you login to gnome, right? However, it would be too much of a hassle to have to authenticate each time you need a token. I imagine macOS people have access to the secure enclave using touch ID but then even that is not available on all devices. I feel like we are barking up the wrong tree here. The plain text token thing can't be fixed. We have to protect our computers from malware to begin with. Maybe Microsoft was right to use secure admin workstations (saw) for privileged access but then again it is too much of a hassle.
- L-four 11mo agoI think the correct solution is to use a keyring. On Linux there's gnome keyring and last time I worked on a IOS app there was something similar. This does mean entering your keyring password a lot. https://en.wikipedia.org/wiki/GNOME_Keyring https://en.wikipedia.org/wiki/GNOME_Keyring
- 1718627440 11mo ago> This does mean entering your keyring password a lot. Not when you put that keyrings password into the user keyring. I think it is also cached by default.
- mrklol 11mo agoIs there any reason to keep using postinstall scripts allowed instead of asking e.g. the user? Are they even needed in most cases?
- Cthulhu_ 11mo agoIf you ask the user "should I run this script" after installing, they will just hit yes every time. But also, a lot (I'm confident it's "most") of NPM install operations are done on a CI server, which need to run without human interaction.
- arkh 11mo agoMost of those attacks do the same kind of things. So I'm surprised to never see something akin to "our AI systems flagged a possible attack" in those posts. Or the fact Github from AI pusher fame Microsoft does not already use their AI to find this kind of attacks before they become a problem. Where is this miracle AI for cybersecurity when you need it?
- nottorp 11mo agoCurrent "AI" is generative "AI". It can generate bullshit not evaluate anything. Edit: see the curl posts about them being bombarded with "AI" generated security reports that mean nothing and waste their time.
- michaelt 11mo agoThe security product marketers ruined “a possible attack” as a brag 25 years ago. Every time a firewall blocks something, it’s a possible attack being blocked, and imagine how often that happens.
- firesteelrain 11mo agoSonaType Lifecycle has some magic to prevent these types of attacks. They claim it is AI based. Not sure how it all works as it is proprietary but it is one of the things we use at work. SonaType IQ server powers it
- akdor1154 11mo agoJesus Christ, i can't even get my own package to reliably self-publish in CI without ending up with a fragile pile of twigs, I'm awed they are able to automate infection like that.
- Yokohiii 11mo agoI have an friend that starts an project next month that will rely on npm. He is quite a noob and didn't code in ages. He will have almost no clue how to harden against this, he will probably not even notice if he becomes a victim until something really bad happens. Pretty sad.
- mkesper 11mo agoAt least make them run pnpm instead of npm, disabling post-install scripts. https://pnpm.io/supply-chain-security https://pnpm.io/supply-chain-security
- newsoftheday 11mo ago"a friend" because friend starts with a consonant sound, not a vowel sound. "a project" for the same reason. HTH.
- Yokohiii 11mo agoLike an egregious comment?
- zx8080 11mo agoEveryone wanted to centralise as much as possible to save every cent. No wonder what it got us all into. Enjoy it while saving your cent!
- Flere-Imsaho 11mo agoAlso layer upon layer of abstractions - to the point where no single person understands the stack from top to bottom. Perhaps there is a light at the end of the tunnel: with AI coding assistance, the whole application can be written from scratch (like the old days). All the code is there, not buried deep within someone else's codebase.
- efortis 11mo agoMitigate this attack vector by adding: ignore-scripts=true to your .npmrc https://blog.uxtly.com/getting-rid-of-npm-scripts https://blog.uxtly.com/getting-rid-of-npm-scripts
- philipwhiuk 11mo agoOr use pnpm
- jMyles 11mo agoTo delay updates, you mean? I'm curious though: how do you avoid being stuck on the _vulnerable_ versions, delaying updates?
- homebrewer 11mo agopnpm disables all install scripts by default and makes it trivial to whitelist the few you need. It's usually just one or two, or sometimes zero, depending on the project. Even without malware, most postinstall scripts are used for spam and analytics, and running them makes your life worse. npm should have died long ago, I don't know why it's still being used.
- seanwilson 11mo agoOnce you run the JavaScript of the npm library you just installed, if it's Node, what's to stop it accessing environment variables and any file it wants, and sending data to any domain it wants?
- MetaWhirledPeas 11mo agoNothing, but at least you'll have time to see the audit if it's aware.
- efortis 11mo agofs and net can be mitigated with `--permission` https://nodejs.org/api/permissions.html https://nodejs.org/api/permissions.html Regardless, it’s worth using `--ignore-scripts=true` because that’s the common vector these supply chain attacks target. Consider that when automating the attack, adding it to the application code is more difficult than injecting it into life-cycle scripts, which have well-known config lines.
- qubex 11mo agoAbout a month ago I had a rather annoying task to perform, and I found an NPM package that handled it. I threw “brew install NPM” or whatever onto the terminal and watched a veritable deluge of dependencies download and install. Then I typed in ‘npm ’ and my hand hovered on the keyboard after the space as I suddenly thought long and hard about where I was on the risk/benefit curve and then I backspaced and typed “brew uninstall npm” instead, and eventually strung together an oldschool unix utilities pipeline with some awk thrown in. Probably the best decision of my life, in retrospect.
- kubafu 11mo agoSame story from a month ago. The moment I saw the sheer number of dependencies artillery wanted to pull I gave up.
- sigmoid10 11mo agoThis is why you want containerisation or, even better, full virtualisation. Running programs built on node, python or any other ecosystem that makes installing tons of dependencies easy (and thus frustratingly common) on your main system where you keep any unrelated data is a surefire way to get compromised by the supply chain eventually. I don't even have the interpreters for python and js on my base system anymore - just so I don't accidentally run something in the host terminal that shouldn't run there.
- baq 11mo ago...but the github runners already are virtualized; you'd need to virtualize the secrets they have access to instead.
- Glemkloksdjf 11mo agoNo thats not what i want, that whats i need when i use something like npm. Which can't be the right way.
- godzillabrennus 11mo agoThe right way (technically) and the commercially viable way are often diametrically opposed. Ship first, ask questions later, or, move fast and break things, wins.
- austin-cheney 11mo agoAre there any good alternatives to ESLint? ESLint is now my only dev dependency with hundreds of dependencies of its own.
- tuzemec 11mo agoBiome: https://biomejs.dev/ https://biomejs.dev/ Also the whole ecosystem around OXS looks very promising: https://oxc.rs/ https://oxc.rs/
- jackwilsdon 11mo agoBoth of those have over >400 dependencies each [0] [1] but just in Rust instead - there hasn't been a Rust supply chain attack yet but is this any better? [2] Admittedly you're not normally downloading the dependencies to your machine as you're often using pre-built binaries, but a malicious package could still run if a version was shipped with it. [0]: https://github.com/biomejs/biome/blob/93182ea8e9d479fd0187ce21ff8fdcdf143d64cf/Cargo.lock https://github.com/biomejs/biome/blob/93182ea8e9d479fd0187ce... [1]: https://github.com/oxc-project/oxc/blob/65bd5584bfce0c7da90ff46f8e1052861e14b7eb/Cargo.lock https://github.com/oxc-project/oxc/blob/65bd5584bfce0c7da90f... [2]: https://users.rust-lang.org/t/yet-another-npm-supply-chain-attack-is-cargo-any-safer/133766 https://users.rust-lang.org/t/yet-another-npm-supply-chain-a...
- brabel 11mo agoWow that’s terrifying.
- xomodo 11mo agoI think I found some repos here: https://github.com/search?q=in:description+Sha1-Hulud&type=repositories https://github.com/search?q=in:description+Sha1-Hulud&type=r...
- ksynwa 11mo agoWhat are the "sha1-hulud" github repositories for exactly? I see files like secrets.json but the contents seems to not be valid json. Are these encrypted?
- hiccuphippo 11mo agoI looked at one and it was doubly encoded base64.
- mikkupikku 11mo ago> "This creates a dangerous scenario. If GitHub mass-deletes the malware's repositories or npm bulk-revokes compromised tokens, thousands of infected systems could simultaneously destroy user data." Pop quiz, hot shot! A terrorist is holding user data hostage, got enough malware strapped to his chest to blow a data center in half. Now what do you do? Shoot the hostage.
- hsbauauvhabzb 11mo agoThe hostage naively walked past all the police and into the data centre, and you’re shooing them in the leg. They’ll probably survive, but they knowingly or incompetently made their choice. Sucks to be them.
- hakcermani 11mo agopardon the naive question. What i don't get is these injected payload are js files, isn't there some scanning at npm upload level to look for exfiltration behaviour, bash executions of dangerous commands like rm or shred ?
- Barry-Perkins 11mo ago[dead]
- hresvelgr 11mo agoWhile this does appear to be getting worse, I'm in the camp of letting it happen. The Node/JS ecosystem is imho completely unsuitable for serious work and this is merely the natural consequence. Let it burn, and perhaps something better will come from the ashes.
- deleted 11mo ago[deleted]
- newsoftheday 11mo agoAs a Java dev, seems like only a matter of time before Maven Nexus repo attacks become commonplace.
- loginatnine 11mo agoSend them a request to have Trusted publishers support at central-support (at) sonatype.com I did that a couple of weeks ago and received an acknowledgment "Another request on Trusted Publishing option. Assigning to Product for review and further action." so this is a bit encouraging. At least Maven dependencies don't execute scripts on install, but Maven plugins could have a big blast radius.
- jonhohle 11mo agoOver a decade ago at Amazon, all third party dependencies needed to be manually imported. On the one hand, it makes importing new versions or packages slow. On the other hand, there is a very explicit intention and log of every external change that made it into internal projects. At my previous company, I implemented staged dependencies with artifactory so that production could never get packages that had never gone through CR, or staging environments first. They just were never replicated. That eliminated fuzzy dependency matches that showed up for the first time in production (something that did happen). Because dev to production was about 1 week, it also afforded time to identify packages before they had a chance to be deployed. Obviously it was less robust than manually importing. Maybe self-hosted package caches support these features now, but 6-7 years ago, that was all manual work.
- arresin 11mo agoOh look, another day and another NPM supply chain attack.
- csutil-com 11mo agoCan't GitHub just block/make private all https://github.com/search?q=Sha1-Hulud%3A%20The%20Second%20Coming&type=repositories https://github.com/search?q=Sha1-Hulud%3A%20The%20Second%20C... repos as a first step?
- noobcoder 10mo agoThe brutal part is how rotate secrets and move on has become the default hygiene advice when the real pattern is that npm keeps being the soft underbelly of modern stacks It should be mandatory for a build process to have some tool like Prismor scan for these
- rkagerer 10mo agoOnce upon a time I would download the source code of a library, unzip it, and personally vet the code before adding it to my project. With some package managers these days I don't even know how to do that (and I'm not necessarily talking about Node, specifically). How do you figure out what the install process does to your computer, without becoming an expert on the manifest syntax? For those of us who care about what goes on under the hood, it is definitely not easier than the days of following well-formed (or even semi-formed) documentation by hand.
- Traubenfuchs 10mo agoStill? Again?