7 ms·

NPM to implement staged publishing after turbulent shift off classic tokens

205 points by feross 9mo ago

woodruffw 9mo ago
> In its current form, however, trusted publishing applies to a limited set of use cases. Support is restricted to a small number of CI providers, it cannot be used for the first publish of a new package, and it does not yet offer enforcement mechanisms such as mandatory 2FA at publish time. Those constraints have led maintainer groups to caution against treating trusted publishing as a universal upgrade, particularly for high-impact or critical packages. This isn't strictly accurate: when we designed Trusted Publishing for PyPI, we designed it to be generic across OIDC IdPs (typically CI providers), and explicitly included an accommodation for creating new projects via Trusted Publishing (we called it "pending" publishers[1]). The latter is something that not all subsequent adopters of the Trusted Publishing technique have adopted, which is IMO both unfortunate and understandable (since it's a complication over the data model/assumptions around package existence). I think a lot of the pains here are self-inflicted on GitHub's part here: deciding to remove normal API credentials entirely strikes me as extremely aggressive, and is completely unrelated to implementing Trusted Publishing. Combining the two together in the same campaign has made things unnecessarily confusing for users and integrators, it seems. [1]: https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/ https://docs.pypi.org/trusted-publishers/creating-a-project-...
- deleted 9mo ago
  [deleted]
  - deleted 9mo ago
    [deleted]
- epage 9mo ago
  As I'm not familiar with the npm ecosystem so maybe I'm misunderstanding this but it sounds like they removed support for local publishes (via a token) in favor of CI publishing using Trusted Publishing. If that is correct, I thought this was discussed when Trusted Publishing was proposed for Rust that it was not meant to replace local publishing, only harden CI publishing.
  - woodruffw 9mo ago
    > If that is correct, I thought this was discussed when Trusted Publishing was proposed for Rust that it was not meant to replace local publishing, only harden CI publishing. Yes, that's right, and that's how it was implemented for both Rust and Python. NPM seems to have decided to do their own thing here. (More precisely, I think NPM still allows local publishing with an API token, they just won't grant long-lived ones anymore.)
    - the_mitsuhiko 9mo ago
      I think the path to dependency on closed publishers was opened wide with the introduction of both attestations and trusted publishing. People now have assigned extra qualities to such releases and it pushes the ecosystem towards more dependency on closed CI systems such as github and gitlab. It was a good intention, but the ramifications of it I don't think are great.
      - woodruffw 9mo ago
        > People now have assigned extra qualities to such releases and it pushes the ecosystem towards more dependency on closed CI systems such as github and gitlab. I think this is unfortunately true, but it's also a tale as old as time. I think PyPI did a good job of documenting why you shouldn't treat attestations as evidence of security modulo independent trust in an identity[1], but the temptation to verify a signature and call it a day is great for a lot of people. Still, I don't know what a better solution is -- I think there's general agreement that packaging ecosystems should have some cryptographically sound way for responsible parties to correlate identities to their packages, and that previous techniques don't have a great track record. (Something that's noteworthy is that PyPI's implementation of attestations uses CI/CD identities because it's easy, but that's not a fundamental limitation: it could also allow email identities with a bit more work. I'd love to see more experimentation in that direction, given that it lifts the dependency on CI/CD platforms.) [1]: https://docs.pypi.org/attestations/security-model/ https://docs.pypi.org/attestations/security-model/
      - blibble 9mo ago
- pamcake 9mo ago
  "In its current form" is in context of NPM, where I think it's accurate. Great to see PyPi taking a more reasonable path.
- veeti 9mo ago
  Yet in practice, only the big boys are allowed to become "Trusted Publishers": > In the interest of making the best use of PyPI's finite resources, we only plan to support platforms that have a reasonable level of usage among PyPI users for publishing. Additionally, we have high standards for overall reliability and security in the operation of a supported Identity Provider: in practice, this means that a home-grown or personal use IdP will not be eligible. How long until everyone is forced to launder their artifacts using Microsoft (TM) GitHub (R) to be "trusted"? [1] https://docs.pypi.org/trusted-publishers/internals/#how-do-i-become-a-trusted-publishing-provider https://docs.pypi.org/trusted-publishers/internals/#how-do-i...
  - woodruffw 9mo ago
    I wrote a good chunk of those docs, and I can assure you that the goal is always to add more identity providers, and not to enforce support for any particular provider. GitHub was only the first because it’s popular; there’s no grand evil theory beyond that.
    - VorpalWay 9mo ago
      So if I self host my own gitea/forgejo instance, will trusted publishing work for me?
      - eesmith 9mo ago
        According to their docs, they have a "have high standards for overall reliability and security in the operation of a supported Identity Provider: in practice, this means that a home-grown or personal use IdP will not be eligible." If you think your setup meets those standards, you'll need to use Microsoft (TM) GitHub (R) to contact them.
        VorpalWay 9mo ago
        In other words, it is a clear centralization drive. No two ways about it.
- jacquesm 9mo ago
  > I think a lot of the pains here are self-inflicted on GitHub's part here It is spelled 'Microsoft'. What did you think would happen long term? I remember when that acquisition happened and there were parties thrown all around, MS finally 'got' open source. And never mind feeding all of the GitHub contents to their AI.
  - woodruffw 9mo ago
    My point was that these are political and logistical problems latent to GitHub/Microsoft/whatever, not to Trusted Publishing as a design. I don't think I materially disagree with you about Microsoft not having a sterling reputation.
    - jacquesm 9mo ago
      Yes, but I think that that more than anything is the driver behind these decisions.
      - woodruffw 9mo ago
        Which ones? It wasn't a driver behind our decisions when we designed Trusted Publishing originally; the fact that GitHub has been such a mess has been a consistent source of tsuris in my life.
spankalee 9mo ago
I maintain some very highly used npm package and this situation just has me on edge. In our last release of dozens of packages, I was manually reading though our package-lock and package.json changes and reviewing every dependency change. Luckily our core libraries have no external dependencies, but our tooling has a ton. We were left with a tough choice of moving to Trusted Publishers or allowing a few team members to publish locally with 2FA. We decided on Trusted Publishers because we've had an automated process with review steps for years, but we understand there's still a chance of a hack, so we're just extremely cautious with any PRs right now. Turning on Trusted Publishers was a huge pain with so many package. The real thing we want for publishing is for us is to be able to continue to use our CI-based publishing setup, with Trusted Publishers, but with a human-in-the-loop 2FA step. But that's only part of a complete solution. HITL is only guaranteed to slow down malicious code propagating. It doesn't actually protect our project against compromised dependencies, and doesn't really help prevent us from spreading it. All of that is still a manual responsibility of the humans. We need tools to lock down and analyze our dependencies better, and tools to analyze our our packages before publishing. I also want better tools for analyzing and sandboxing 3rd party PRs before running CI. Right now we have HITL there, but we have to manually investigate each PR before running tests.
m4rtink 9mo ago
Can we finally decare this (and other incomplete language specific package namanegers) to be a failed experiment and go back to robust and secure distro based package management workflow, with maintainers separate from upstream developpers ?
- ashishb 9mo ago
  I run them inside a sandbox. The npm community is too big that one can never discard it for frontend development.
- arccy 9mo ago
  where do you get all these trusted people to review your dependencies from? it can't be anyone, because you're essentially delegating trust. no way there's enough trustworthy volunteers (and how do you vet them all?) and who's going to pay them if they're not volunteers?
- Macha 9mo ago
  The robust and secure distro based package management workflow that shipped the libxz backdoor to everyone, and broke openssh key generation, and most of the functionality of keepassxc?
  - TZubiri 9mo ago
    >workflow that shipped the libxz backdoor to everyone Isn't it the case that it didn't ship the backdoor? Precisely because of the thorough testing and vetting process?
    - Macha 9mo ago
      No, it shipped in Debian Sid, OpenSUSE Tumbleweed and Fedora Rawhide, along with beta versions of Ubuntu 24.04 and Fedora 40. Arch also shipped it but the code looked for rpm/apt distros so the payload didn’t trigger. It was caught by a Postgres developer who noticed strange performance on their Debian Sid system, not by anyone involved with the distro packaging process.
lloydatkinson 9mo ago
Just to be clear, "trusted publishing" means a type of reverse vendor lock in? Only some CI systems are allowed to be used for it.
- LtWorf 9mo ago
  Yes. You cannot set up your own.
- woodruffw 9mo ago
  "Trusted Publishing" is just a term of art for OIDC. NPM can and should support federating with CI/CD platforms other than GitHub Actions, to avoid even the appearance of impropriety. (It makes sense that they'd target GHA first, since that's where the majority of their users probably are. But the technique itself is fundamentally platform agnostic and interoperable.)
  - thayne 9mo ago
    Currently only GHA and Gitlab are supported.
herpdyderp 9mo ago
The shift wouldn't have been so turbulent if npm had simply updated their CLI in tandem. I still can't use 2FA to publish because their CLI simply cannot handle it.
- pamcake 9mo ago
  CLI publishing with TOTP 2FA worked fine until they broke it.
ajross 9mo ago
I try to make this point when the subject comes up, but IMHO this is a lipstick-on-a-pig solution. The problem with npm security isn't stability or attestation, it's the impossibility of auditing all that garbage. The software is too fine-grained. Too many (way too many) packages from small projects or obscure single authors doing way too many things that are being picked up for one trivial feature. That's just never going to work. If you don't know who's writing your software the answer will always end up being "Your Enemies" at some point. And the solution is to stop the madness. Conglomerate the development. No more tiny things. Use big packages[1] from projects with recognized governance. Audit their releases and inclusion in the repository from a separate project with its own validation and testing. No more letting the bad guys push a button to publish. Which is to say: this needs to be Debian. Or some other Linux distro. But really the best thing is for the JS community (PyPI and Cargo are dancing on the edge of madness too) to abandon its mistake and move everything into a bunch of Debian packages. Won't happen, but it's the solution nonetheless. [1] c.f. the stuff done under the Apache banner, or C++ Boost, etc...
- 12345ieee 9mo ago
  The fact is that being Debian is boring, and JS (python/rust/...) is *cool*. Give it a few more decades, hopefully it'll be boring by then, the same way, say, making a house is boring.
  - immibis 9mo ago
    > the same way, say, making a house is boring. The government will mostly ban it to keep prices high?
- gr4vityWall 9mo ago
  Agreed on the "this needs to be Debian" part. If some of the most popular JS packages were available through the system package manager as normal *.deb packages, I think people would be more likely to build on top of stable versions/releases. Stability (in the "it doesn't change" sense) is underrated.
pyrolistical 9mo ago
Seems like requiring 2FA to publish or trusted publishing should prevent the vast majority of this issue. The only tricky bit would be to disallow approval own pull request when using trusted publishing. That should fall back to requiring 2FA
- thayne 9mo ago
  It also make it impossible to publish using CI, which is problematic for projects with frequent releases. And trusted publishing doesn't solve that if you use self-hosted CI.
  - fc417fc802 9mo ago
    > trusted publishing doesn't solve that if you use self-hosted CI Is there any particular reason for the whitelist approach? Standing on the sidelines it appears wholly unnecessary to me. Authentication that an artifact came from a given CI system seems orthogonal to the question of how much trust you place in a given CI system.
    - thayne 9mo ago
      Well, given that Github owns NPM, one potential reason could be vendor lock in. Also, from an implementation standpoint it is probably easier to make a system that just works for a handful of OIDC providers, than a more general solution. In particular, a general solution would require having a UI and maybe an API for registering NPM as a service provider for an identity provider of the package owner's choice.
      - fc417fc802 9mo ago
        Is OIDC federation really so involved as to require a nontrivial registration step? Shouldn't this be an OAuth style flow where you initiate with the third party and then confirm the requested permissions with the service? Where did it all go wrong?
        9mo ago
bikelang 9mo ago
Why is JS, in particular, so deeply afflicted with these issues? Why hasn’t there been an effort to create a more robust standard library? Or at least first party libraries maintained by the JavaScript team? That way folks can pull in trusted deps instead of all the hodgepodge. Go did a lot wrong. It was just awful before they added Go modules. But it’s puzzling to me to understand why as a community and ecosystem its 3rd party dependencies seem so much less bloated. Part of it I think is because the standard library is pretty expansive. Part of it is because of things like golang.org/x. But there’s also a lot of corporate maintainers - and I feel like part of that is because packages are namespaces to the repository - which itself is namespaced to ownership. Technically that isn’t even a requirement - but the community adopted it pretty evenly - and it makes me wonder why others haven’t.
- bblaylock 9mo ago
  There has been an effort to both replace npm with a better model and to have a stable standard library. See https://jsr.io/@std https://jsr.io/@std
  - bikelang 9mo ago
    Is the ambition to lift this out of Deno? Bun brings its own standard lib too right? Are the two coordinating at all?
    - WorldMaker 9mo ago
      The stuff on JSR is lifted out of Deno. JSR can install packages for Node and Bun [0]. Most of the "@std" packages in the link above claim support for Bun (the right hand of the package list "stack of avatars" will have the Bun avatar; easier to read on individual package pages where it becomes a header, though), and there is a Bun test matrix in the GitHub Actions CI. (Right now it looks like it just has Bun latest in the matrix, though.) In terms of coordination, I don't see any obvious Bun contributors in a quick skim [1], but it seems open to contribution and is MIT licensed. [0] https://jsr.io/docs/using-packages#adding-a-package https://jsr.io/docs/using-packages#adding-a-package [1] https://github.com/denoland/std/graphs/contributors https://github.com/denoland/std/graphs/contributors
cvbnmb 9mo ago
I think this turbulent shift is going to push a lot of node devs elsewhere. I understand things need to be safe, but this is a recklessly fast transition.
TZubiri 9mo ago
I've been thinking, Java doesn't have many supply chain issues, their model is based of namespacing with the DNS system. If I want a library from vendor.com, the library to import is somewhere under com.vendor.* Simple enough, things like npm and pip reinvent a naming authority, have no cost associated (so it's weak to sybil attacks), all for not much, what do you get in exchange? You create equality by letting everyone contribute their wonderful packages, even those that don't have 15$/yr? I'm sorry was the previous leading internet mechanism not good and decentralized enough for you? Java's package naming system is great in design, the biggest vuln in dependencies that I can think of on java was not a supply chain specific vuln, but rather a general weakness of a library (log4j). But maybe someone with more java experience can point to some disadvantage of the java system that explains why we are not all copying that
- woodruffw 9mo ago
  I think Java’s DNS namespacing is, at best, only a weak benefit to the supply chain security posture of Java packaging as a whole. I think it’s more that Java is (1) a batteries-included language, (2) lacks the same pervasive open source packaging culture that Python, Rust, JS, etc. have, (3) is much more conservative around dependency updates as a community, and (4) lacks a (well-known?) build time code execution vector similar to JS’s install scripts or Python’s setup.py. (Most of these are good things, to be clear!)
  - TZubiri 9mo ago
    There being a compile/runtime difference at all seems quite impactful to dependency mgmt as a whole apparently, I've seen impacts in bc, build times and now security.
  - rectang 9mo ago
    Thanks for this insight-dense comment — and for all the efforts you have put into Trusted Publishing.
  - fc417fc802 9mo ago
    > lacks a (well-known?) build time code execution vector similar to JS’s install scripts or Python’s setup.py How is that leveraged by attackers in practice? Naively I would expect the actual issue to be insufficient sandboxing (network access in particular).
jonkoops 9mo ago
I really think that the main issue is that NPM itself will execute any script that is in the "postinstall" section of a package, without asking the user for permission. This is a solved problem in other package managers, e.g. PNPM will only run scripts if the user allows them to, and store the allowlist in the package.json file for future reference. In this scenario, if a dependency were to add a "postinstall" script because it was compromised, it would not execute, and the user can review whether it should, greatly reducing the attack surface.
- m90 9mo ago
  Wouldn't this just make the number of packages that can be targeted smaller? E.g. I publish a testrunner that needs to install Headless Chrome if not present via postinstall. People trust me and put the package on their allowlist. My account gets compromised and a malicious update is published. People execute malicious code they have never vetted. I do understand this is still better than npm right now, but it's still broken.
  - acdha 9mo ago
    Security is usually full of incremental improvements like that, however. Reducing the scope from all of NPM to the handful of things like test runners would be an enormous benefit for auditors and would encourage consolidation (e.g. most testing frameworks could consolidate on a single headless chrome package), and in the future this could be further improved by things like restricting the scope of those scripts using the operating system sandbox features.
  - jonkoops 9mo ago
    Security is layered, no layer will conclusively keep you safe, but each one make it harder to pierce to the core. For example, the impact of the recent SHA1-Hulud attack would be much less, as compromised packages (that previously did not have any scripts executing at install time), would not suddenly start executing, as they are not allowlisted.
- fergie 9mo ago
  There are a large subset of security problems that are solved by simply eliminating compilation steps typically included in "postinstall". If you want a more secure, more debuggable, more extensible lib, then you should definitely publish it in pure js (rather than, say, Typescript), so that there is no postinstall attack surface.
fergie 9mo ago
In all of this, people forget that NPM packages are largely maintained by volunteers. If you are going to put up hurdles and give us extra jobs, you need to start paying us. Open source licenses explicitly state some variation of "use at your own risk". A big motivation for most maintainers is that we can create without being told what to do. I had 25 million downloads on NPM last year. Not a huge amount compared to the big libs, but OTOH, people actually use my stuff. For this I have received exactly $0 (if they were Spotify or YouTube streams I would realistically be looking at ~$100,000). I propose that we have two NPMs. A non-commercial NPM that is 100% use at your own risk, and a commerical NPM that has various guarantees that authors and maintainers are paid to uphold.
- justarandomname 9mo ago
  I agree with you here, it feels like management said: "well, we have to do SOMETHING!" and this is what they chose: push more of the burden on to the developers giving away stuff for free when the burden should be on the developers and companies consuming the stuff for free.
  - LtWorf 9mo ago
    But the management who decided that gets rewarded for pushing work to someone else.
- pamcake 9mo ago
  Not looking forward to the mandatory doxxing that would probably come along if this was introduced today.
  - fergie 9mo ago
    This makes no sense, maintainers are not exactly operating under a cloak of anonymity. Quite the opposite in fact.
- jeroenhd 9mo ago
  NPM has to decide between either being a friendly place for hobbyists to explore their passions or being the backbone for a significant slice of the IT industry. Every time someone pulls/messes with/uploads malware to NPM, people complain and blame NPM. Every time NPM takes steps to prevent pulling/messing with/uploading malware to NPM, people complain and blame NPM. I don't think splitting NPM will change that. Current NPM is already the "100% use at your own risk" NPM and still people complain when a piece of protestware breaks their build.
sarreph 9mo ago
I really don't think this should be a registry-level issue. As in, the friction shouldn't be introduced into _publishing_ workflows, it should be introduced into _subscription_ workflows where there is an easy fix. Just stop supporting auto-update (through wildcard patch or minor versions) by default... Make the default behaviour to install whatever version you load at install time (like `npm ci` does)
- anon7000 9mo ago
  No, it does need help at the publishing side. Most places I know are not updating by default. Everything has a lock file. But the nx attack a few months ago happened because the VS Code extension for nx was always ran the @latest version to check for updates or something like that. So yeah… people will always have these workflows which are either stupid or don’t have an easy way to use a lock file. So I’d sure as hell like npm to also take some steps to secure things better. As far as I know, using a lock file with npm install is both the default behavior and also doesn’t randomly updates things unless you ask it to… though it’s definitely best practice to pin dependencies too
dmarwicke 9mo ago
2FA publishing still doesn't work for me. just use legacy tokens at this point, gave up trying to figure out what's wrong
yoan9224 9mo ago
The comment about maintainers not getting paid resonates. I'm a solo founder and these security changes, while necessary, add real friction to shipping. The irony is that trusted publishing pushes everyone toward GitHub Actions, which centralizes risk. If GHA gets compromised, the blast radius is enormous. Meanwhile solo devs who publish from their local machine with 2FA are arguably more secure (smaller attack surface, human in the loop) but are being pushed toward automation. What I'd like to see: a middle ground where trusted publishing works but requires a 2FA confirmation before the publish actually goes through. Keep the automation, keep the human gate. Best of both worlds. The staged publishing approach mentioned in the article is a good step - at least you'd catch malicious code before it hits everyone's node_modules.
irishmanlondon 9mo ago
[dead]

205 points by feross 9mo ago

> In its current form, however, trusted publishing applies to a limited set of use cases. Support is restricted to a small number of CI providers, it cannot be used for the first publish of a new package, and it does not yet offer enforcement mechanisms such as mandatory 2FA at publish time. Those constraints have led maintainer groups to caution against treating trusted publishing as a universal upgrade, particularly for high-impact or critical packages. This isn't strictly accurate: when we designed Trusted Publishing for PyPI, we designed it to be generic across OIDC IdPs (typically CI providers), and explicitly included an accommodation for creating new projects via Trusted Publishing (we called it "pending" publishers[1]). The latter is something that not all subsequent adopters of the Trusted Publishing technique have adopted, which is IMO both unfortunate and understandable (since it's a complication over the data model/assumptions around package existence). I think a lot of the pains here are self-inflicted on GitHub's part here: deciding to remove normal API credentials entirely strikes me as extremely aggressive, and is completely unrelated to implementing Trusted Publishing. Combining the two together in the same campaign has made things unnecessarily confusing for users and integrators, it seems. [1]: https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/ https://docs.pypi.org/trusted-publishers/creating-a-project-...

deleted 9mo ago
[deleted]
- deleted 9mo ago
  [deleted]
epage 9mo ago
As I'm not familiar with the npm ecosystem so maybe I'm misunderstanding this but it sounds like they removed support for local publishes (via a token) in favor of CI publishing using Trusted Publishing. If that is correct, I thought this was discussed when Trusted Publishing was proposed for Rust that it was not meant to replace local publishing, only harden CI publishing.
- woodruffw 9mo ago
  > If that is correct, I thought this was discussed when Trusted Publishing was proposed for Rust that it was not meant to replace local publishing, only harden CI publishing. Yes, that's right, and that's how it was implemented for both Rust and Python. NPM seems to have decided to do their own thing here. (More precisely, I think NPM still allows local publishing with an API token, they just won't grant long-lived ones anymore.)
  - the_mitsuhiko 9mo ago
    I think the path to dependency on closed publishers was opened wide with the introduction of both attestations and trusted publishing. People now have assigned extra qualities to such releases and it pushes the ecosystem towards more dependency on closed CI systems such as github and gitlab. It was a good intention, but the ramifications of it I don't think are great.
    - woodruffw 9mo ago
      > People now have assigned extra qualities to such releases and it pushes the ecosystem towards more dependency on closed CI systems such as github and gitlab. I think this is unfortunately true, but it's also a tale as old as time. I think PyPI did a good job of documenting why you shouldn't treat attestations as evidence of security modulo independent trust in an identity[1], but the temptation to verify a signature and call it a day is great for a lot of people. Still, I don't know what a better solution is -- I think there's general agreement that packaging ecosystems should have some cryptographically sound way for responsible parties to correlate identities to their packages, and that previous techniques don't have a great track record. (Something that's noteworthy is that PyPI's implementation of attestations uses CI/CD identities because it's easy, but that's not a fundamental limitation: it could also allow email identities with a bit more work. I'd love to see more experimentation in that direction, given that it lifts the dependency on CI/CD platforms.) [1]: https://docs.pypi.org/attestations/security-model/ https://docs.pypi.org/attestations/security-model/
    - blibble 9mo ago
pamcake 9mo ago
"In its current form" is in context of NPM, where I think it's accurate. Great to see PyPi taking a more reasonable path.
veeti 9mo ago
Yet in practice, only the big boys are allowed to become "Trusted Publishers": > In the interest of making the best use of PyPI's finite resources, we only plan to support platforms that have a reasonable level of usage among PyPI users for publishing. Additionally, we have high standards for overall reliability and security in the operation of a supported Identity Provider: in practice, this means that a home-grown or personal use IdP will not be eligible. How long until everyone is forced to launder their artifacts using Microsoft (TM) GitHub (R) to be "trusted"? [1] https://docs.pypi.org/trusted-publishers/internals/#how-do-i-become-a-trusted-publishing-provider https://docs.pypi.org/trusted-publishers/internals/#how-do-i...
- woodruffw 9mo ago
  I wrote a good chunk of those docs, and I can assure you that the goal is always to add more identity providers, and not to enforce support for any particular provider. GitHub was only the first because it’s popular; there’s no grand evil theory beyond that.
  - VorpalWay 9mo ago
    So if I self host my own gitea/forgejo instance, will trusted publishing work for me?
    - eesmith 9mo ago
      According to their docs, they have a "have high standards for overall reliability and security in the operation of a supported Identity Provider: in practice, this means that a home-grown or personal use IdP will not be eligible." If you think your setup meets those standards, you'll need to use Microsoft (TM) GitHub (R) to contact them.
      - VorpalWay 9mo ago
        In other words, it is a clear centralization drive. No two ways about it.
jacquesm 9mo ago
> I think a lot of the pains here are self-inflicted on GitHub's part here It is spelled 'Microsoft'. What did you think would happen long term? I remember when that acquisition happened and there were parties thrown all around, MS finally 'got' open source. And never mind feeding all of the GitHub contents to their AI.
- woodruffw 9mo ago
  My point was that these are political and logistical problems latent to GitHub/Microsoft/whatever, not to Trusted Publishing as a design. I don't think I materially disagree with you about Microsoft not having a sterling reputation.
  - jacquesm 9mo ago
    Yes, but I think that that more than anything is the driver behind these decisions.
    - woodruffw 9mo ago
      Which ones? It wasn't a driver behind our decisions when we designed Trusted Publishing originally; the fact that GitHub has been such a mess has been a consistent source of tsuris in my life.

spankalee 9mo ago

I maintain some very highly used npm package and this situation just has me on edge. In our last release of dozens of packages, I was manually reading though our package-lock and package.json changes and reviewing every dependency change. Luckily our core libraries have no external dependencies, but our tooling has a ton. We were left with a tough choice of moving to Trusted Publishers or allowing a few team members to publish locally with 2FA. We decided on Trusted Publishers because we've had an automated process with review steps for years, but we understand there's still a chance of a hack, so we're just extremely cautious with any PRs right now. Turning on Trusted Publishers was a huge pain with so many package. The real thing we want for publishing is for us is to be able to continue to use our CI-based publishing setup, with Trusted Publishers, but with a human-in-the-loop 2FA step. But that's only part of a complete solution. HITL is only guaranteed to slow down malicious code propagating. It doesn't actually protect our project against compromised dependencies, and doesn't really help prevent us from spreading it. All of that is still a manual responsibility of the humans. We need tools to lock down and analyze our dependencies better, and tools to analyze our our packages before publishing. I also want better tools for analyzing and sandboxing 3rd party PRs before running CI. Right now we have HITL there, but we have to manually investigate each PR before running tests.

m4rtink 9mo ago

Can we finally decare this (and other incomplete language specific package namanegers) to be a failed experiment and go back to robust and secure distro based package management workflow, with maintainers separate from upstream developpers ?

ashishb 9mo ago
I run them inside a sandbox. The npm community is too big that one can never discard it for frontend development.
arccy 9mo ago
where do you get all these trusted people to review your dependencies from? it can't be anyone, because you're essentially delegating trust. no way there's enough trustworthy volunteers (and how do you vet them all?) and who's going to pay them if they're not volunteers?
Macha 9mo ago
The robust and secure distro based package management workflow that shipped the libxz backdoor to everyone, and broke openssh key generation, and most of the functionality of keepassxc?
- TZubiri 9mo ago
  >workflow that shipped the libxz backdoor to everyone Isn't it the case that it didn't ship the backdoor? Precisely because of the thorough testing and vetting process?
  - Macha 9mo ago
    No, it shipped in Debian Sid, OpenSUSE Tumbleweed and Fedora Rawhide, along with beta versions of Ubuntu 24.04 and Fedora 40. Arch also shipped it but the code looked for rpm/apt distros so the payload didn’t trigger. It was caught by a Postgres developer who noticed strange performance on their Debian Sid system, not by anyone involved with the distro packaging process.

lloydatkinson 9mo ago

Just to be clear, "trusted publishing" means a type of reverse vendor lock in? Only some CI systems are allowed to be used for it.

LtWorf 9mo ago
Yes. You cannot set up your own.
woodruffw 9mo ago
"Trusted Publishing" is just a term of art for OIDC. NPM can and should support federating with CI/CD platforms other than GitHub Actions, to avoid even the appearance of impropriety. (It makes sense that they'd target GHA first, since that's where the majority of their users probably are. But the technique itself is fundamentally platform agnostic and interoperable.)
- thayne 9mo ago
  Currently only GHA and Gitlab are supported.

herpdyderp 9mo ago

The shift wouldn't have been so turbulent if npm had simply updated their CLI in tandem. I still can't use 2FA to publish because their CLI simply cannot handle it.

pamcake 9mo ago
CLI publishing with TOTP 2FA worked fine until they broke it.

ajross 9mo ago

I try to make this point when the subject comes up, but IMHO this is a lipstick-on-a-pig solution. The problem with npm security isn't stability or attestation, it's the impossibility of auditing all that garbage. The software is too fine-grained. Too many (way too many) packages from small projects or obscure single authors doing way too many things that are being picked up for one trivial feature. That's just never going to work. If you don't know who's writing your software the answer will always end up being "Your Enemies" at some point. And the solution is to stop the madness. Conglomerate the development. No more tiny things. Use big packages[1] from projects with recognized governance. Audit their releases and inclusion in the repository from a separate project with its own validation and testing. No more letting the bad guys push a button to publish. Which is to say: this needs to be Debian. Or some other Linux distro. But really the best thing is for the JS community (PyPI and Cargo are dancing on the edge of madness too) to abandon its mistake and move everything into a bunch of Debian packages. Won't happen, but it's the solution nonetheless. [1] c.f. the stuff done under the Apache banner, or C++ Boost, etc...

12345ieee 9mo ago
The fact is that being Debian is boring, and JS (python/rust/...) is *cool*. Give it a few more decades, hopefully it'll be boring by then, the same way, say, making a house is boring.
- immibis 9mo ago
  > the same way, say, making a house is boring. The government will mostly ban it to keep prices high?
gr4vityWall 9mo ago
Agreed on the "this needs to be Debian" part. If some of the most popular JS packages were available through the system package manager as normal *.deb packages, I think people would be more likely to build on top of stable versions/releases. Stability (in the "it doesn't change" sense) is underrated.

pyrolistical 9mo ago

Seems like requiring 2FA to publish or trusted publishing should prevent the vast majority of this issue. The only tricky bit would be to disallow approval own pull request when using trusted publishing. That should fall back to requiring 2FA

thayne 9mo ago
It also make it impossible to publish using CI, which is problematic for projects with frequent releases. And trusted publishing doesn't solve that if you use self-hosted CI.
- fc417fc802 9mo ago
  > trusted publishing doesn't solve that if you use self-hosted CI Is there any particular reason for the whitelist approach? Standing on the sidelines it appears wholly unnecessary to me. Authentication that an artifact came from a given CI system seems orthogonal to the question of how much trust you place in a given CI system.
  - thayne 9mo ago
    Well, given that Github owns NPM, one potential reason could be vendor lock in. Also, from an implementation standpoint it is probably easier to make a system that just works for a handful of OIDC providers, than a more general solution. In particular, a general solution would require having a UI and maybe an API for registering NPM as a service provider for an identity provider of the package owner's choice.
    - fc417fc802 9mo ago
      Is OIDC federation really so involved as to require a nontrivial registration step? Shouldn't this be an OAuth style flow where you initiate with the third party and then confirm the requested permissions with the service? Where did it all go wrong?
      - 9mo ago

bikelang 9mo ago

Why is JS, in particular, so deeply afflicted with these issues? Why hasn’t there been an effort to create a more robust standard library? Or at least first party libraries maintained by the JavaScript team? That way folks can pull in trusted deps instead of all the hodgepodge. Go did a lot wrong. It was just awful before they added Go modules. But it’s puzzling to me to understand why as a community and ecosystem its 3rd party dependencies seem so much less bloated. Part of it I think is because the standard library is pretty expansive. Part of it is because of things like golang.org/x. But there’s also a lot of corporate maintainers - and I feel like part of that is because packages are namespaces to the repository - which itself is namespaced to ownership. Technically that isn’t even a requirement - but the community adopted it pretty evenly - and it makes me wonder why others haven’t.

bblaylock 9mo ago
There has been an effort to both replace npm with a better model and to have a stable standard library. See https://jsr.io/@std https://jsr.io/@std
- bikelang 9mo ago
  Is the ambition to lift this out of Deno? Bun brings its own standard lib too right? Are the two coordinating at all?
  - WorldMaker 9mo ago
    The stuff on JSR is lifted out of Deno. JSR can install packages for Node and Bun [0]. Most of the "@std" packages in the link above claim support for Bun (the right hand of the package list "stack of avatars" will have the Bun avatar; easier to read on individual package pages where it becomes a header, though), and there is a Bun test matrix in the GitHub Actions CI. (Right now it looks like it just has Bun latest in the matrix, though.) In terms of coordination, I don't see any obvious Bun contributors in a quick skim [1], but it seems open to contribution and is MIT licensed. [0] https://jsr.io/docs/using-packages#adding-a-package https://jsr.io/docs/using-packages#adding-a-package [1] https://github.com/denoland/std/graphs/contributors https://github.com/denoland/std/graphs/contributors

cvbnmb 9mo ago

I think this turbulent shift is going to push a lot of node devs elsewhere. I understand things need to be safe, but this is a recklessly fast transition.

TZubiri 9mo ago

I've been thinking, Java doesn't have many supply chain issues, their model is based of namespacing with the DNS system. If I want a library from vendor.com, the library to import is somewhere under com.vendor.* Simple enough, things like npm and pip reinvent a naming authority, have no cost associated (so it's weak to sybil attacks), all for not much, what do you get in exchange? You create equality by letting everyone contribute their wonderful packages, even those that don't have 15$/yr? I'm sorry was the previous leading internet mechanism not good and decentralized enough for you? Java's package naming system is great in design, the biggest vuln in dependencies that I can think of on java was not a supply chain specific vuln, but rather a general weakness of a library (log4j). But maybe someone with more java experience can point to some disadvantage of the java system that explains why we are not all copying that

woodruffw 9mo ago
I think Java’s DNS namespacing is, at best, only a weak benefit to the supply chain security posture of Java packaging as a whole. I think it’s more that Java is (1) a batteries-included language, (2) lacks the same pervasive open source packaging culture that Python, Rust, JS, etc. have, (3) is much more conservative around dependency updates as a community, and (4) lacks a (well-known?) build time code execution vector similar to JS’s install scripts or Python’s setup.py. (Most of these are good things, to be clear!)
- TZubiri 9mo ago
  There being a compile/runtime difference at all seems quite impactful to dependency mgmt as a whole apparently, I've seen impacts in bc, build times and now security.
- rectang 9mo ago
  Thanks for this insight-dense comment — and for all the efforts you have put into Trusted Publishing.
- fc417fc802 9mo ago
  > lacks a (well-known?) build time code execution vector similar to JS’s install scripts or Python’s setup.py How is that leveraged by attackers in practice? Naively I would expect the actual issue to be insufficient sandboxing (network access in particular).

jonkoops 9mo ago

I really think that the main issue is that NPM itself will execute any script that is in the "postinstall" section of a package, without asking the user for permission. This is a solved problem in other package managers, e.g. PNPM will only run scripts if the user allows them to, and store the allowlist in the package.json file for future reference. In this scenario, if a dependency were to add a "postinstall" script because it was compromised, it would not execute, and the user can review whether it should, greatly reducing the attack surface.

m90 9mo ago
Wouldn't this just make the number of packages that can be targeted smaller? E.g. I publish a testrunner that needs to install Headless Chrome if not present via postinstall. People trust me and put the package on their allowlist. My account gets compromised and a malicious update is published. People execute malicious code they have never vetted. I do understand this is still better than npm right now, but it's still broken.
- acdha 9mo ago
  Security is usually full of incremental improvements like that, however. Reducing the scope from all of NPM to the handful of things like test runners would be an enormous benefit for auditors and would encourage consolidation (e.g. most testing frameworks could consolidate on a single headless chrome package), and in the future this could be further improved by things like restricting the scope of those scripts using the operating system sandbox features.
- jonkoops 9mo ago
  Security is layered, no layer will conclusively keep you safe, but each one make it harder to pierce to the core. For example, the impact of the recent SHA1-Hulud attack would be much less, as compromised packages (that previously did not have any scripts executing at install time), would not suddenly start executing, as they are not allowlisted.
fergie 9mo ago
There are a large subset of security problems that are solved by simply eliminating compilation steps typically included in "postinstall". If you want a more secure, more debuggable, more extensible lib, then you should definitely publish it in pure js (rather than, say, Typescript), so that there is no postinstall attack surface.

fergie 9mo ago

In all of this, people forget that NPM packages are largely maintained by volunteers. If you are going to put up hurdles and give us extra jobs, you need to start paying us. Open source licenses explicitly state some variation of "use at your own risk". A big motivation for most maintainers is that we can create without being told what to do. I had 25 million downloads on NPM last year. Not a huge amount compared to the big libs, but OTOH, people actually use my stuff. For this I have received exactly $0 (if they were Spotify or YouTube streams I would realistically be looking at ~$100,000). I propose that we have two NPMs. A non-commercial NPM that is 100% use at your own risk, and a commerical NPM that has various guarantees that authors and maintainers are paid to uphold.

justarandomname 9mo ago
I agree with you here, it feels like management said: "well, we have to do SOMETHING!" and this is what they chose: push more of the burden on to the developers giving away stuff for free when the burden should be on the developers and companies consuming the stuff for free.
- LtWorf 9mo ago
  But the management who decided that gets rewarded for pushing work to someone else.
pamcake 9mo ago
Not looking forward to the mandatory doxxing that would probably come along if this was introduced today.
- fergie 9mo ago
  This makes no sense, maintainers are not exactly operating under a cloak of anonymity. Quite the opposite in fact.
jeroenhd 9mo ago
NPM has to decide between either being a friendly place for hobbyists to explore their passions or being the backbone for a significant slice of the IT industry. Every time someone pulls/messes with/uploads malware to NPM, people complain and blame NPM. Every time NPM takes steps to prevent pulling/messing with/uploading malware to NPM, people complain and blame NPM. I don't think splitting NPM will change that. Current NPM is already the "100% use at your own risk" NPM and still people complain when a piece of protestware breaks their build.

sarreph 9mo ago

I really don't think this should be a registry-level issue. As in, the friction shouldn't be introduced into _publishing_ workflows, it should be introduced into _subscription_ workflows where there is an easy fix. Just stop supporting auto-update (through wildcard patch or minor versions) by default... Make the default behaviour to install whatever version you load at install time (like `npm ci` does)

anon7000 9mo ago
No, it does need help at the publishing side. Most places I know are not updating by default. Everything has a lock file. But the nx attack a few months ago happened because the VS Code extension for nx was always ran the @latest version to check for updates or something like that. So yeah… people will always have these workflows which are either stupid or don’t have an easy way to use a lock file. So I’d sure as hell like npm to also take some steps to secure things better. As far as I know, using a lock file with npm install is both the default behavior and also doesn’t randomly updates things unless you ask it to… though it’s definitely best practice to pin dependencies too

dmarwicke 9mo ago

2FA publishing still doesn't work for me. just use legacy tokens at this point, gave up trying to figure out what's wrong

yoan9224 9mo ago

The comment about maintainers not getting paid resonates. I'm a solo founder and these security changes, while necessary, add real friction to shipping. The irony is that trusted publishing pushes everyone toward GitHub Actions, which centralizes risk. If GHA gets compromised, the blast radius is enormous. Meanwhile solo devs who publish from their local machine with 2FA are arguably more secure (smaller attack surface, human in the loop) but are being pushed toward automation. What I'd like to see: a middle ground where trusted publishing works but requires a 2FA confirmation before the publish actually goes through. Keep the automation, keep the human gate. Best of both worlds. The staged publishing approach mentioned in the article is a good step - at least you'd catch malicious code before it hits everyone's node_modules.

irishmanlondon 9mo ago

[dead]