3 ms·

An owner being compromised is absolutely survivable on a responsibly run FOSS project with proper commit/review/push signing. This and every other recent suppl

by lrvick 7mo ago

An owner being compromised is absolutely survivable on a responsibly run FOSS project with proper commit/review/push signing. This and every other recent supply chain attack was completely preventable. So much so I am very comfortable victim blaming at this point. This is absolutely on the Axios team. Go setup some smartcards for signing git push/commit and publish those keys widely, and mandate signed merge commits so nothing lands on main without two maintainer sigs, and no more single points of failure.

TheTaytay 7mo ago
It wasn’t done through git. It was a direct npm publish from the compromised machine. If you read further down in the comments (https://github.com/axios/axios/issues/10636#issuecomment-4181516546 https://github.com/axios/axios/issues/10636#issuecomment-418...), it seems difficult to pick the right npm settings to prevent this attack. If I understand it correctly, your suggestions wouldn’t have prevented it, which is evidence that this is not as trivially fixable as you believe it is.
- lrvick 7mo ago
  To prevent supply chain attacks you need multi party cryptographic attestation at every layer, which is pretty straight forward, but you are correct, NPM and GitHub controls absolutely will not save you. Microsoft insists their centralized approach can work, but we have plenty of evidence it does not. Operate under the assumption all accounts will be taken over because centralized corporate auth systems are fundamentally vulnerable. This is how you actually fix it: 1. Every commit must be signed by a maintainer key listed in the MAINTAINERS file or similar 2. Every review/merge must be signed by a -second- maintainer key 3. Every artifact must be build deterministically and be signed by multiple maintainers. 4. Have only one online npm publish key maintained in a deterministic and remotely attestable enclave that validates multiple valid maintainer signatures 5. Automatically sound the alarm if an NPM release is pushed any other way, and automatically revoke it.
  - charcircuit 7mo ago
    And for 5 there should be help on the NPM end to make it so that the alarms can fire before the new update is actually revealed to the public. There could be a short staging time where it could be revoked before any harm has been done. During this staging time NPM should also scan the package through a malware scanner before allowing it to go public.
    - lrvick 7mo ago
      I agree that would be nice, but NPM absolutely will not do any basic supply chain integrity work. They are actively opposed to it citing concerns that it might turn off lower skill developers that would be too annoyed by tapping a yubikey to sign releases or code. I have talked to them enough times over the years to have completely given up here. Whats even more stupid is they actually started mandating 2FA for high risk packages, and FIDO2 supports being used to actually sign artifacts, but they instead simply use it for auth, and let releases stay unsigned. Even the developers they insisted hold cryptographic signing keys, they insist on only throw-away signatures for auth, but not using them for artifact signing to prevent impersonation. It is golf clap level stupid. Consider them a CDN that wants to analyze your code for AI training for their employer and nothing more. Any security controls that might restrict the flow of publishing even a little bit will be rejected.
      - deleted 7mo ago
        [deleted]
fortuitous-frog 7mo ago
Did you investigate the maintainer compromise and publication path? The malicious version was never committed or pushed via git. The maintainer signs his commits, and v1 releases were using OIDC and provenance attestations. The malicious package versions were published locally using the npm cli after the maintainer's machine was compromised via a RAT; there's no way for package maintainers to disable/forbid local publication on npmjs. It seems the Axios team was largely practicing what you're preaching. To the extent they aren't: it still wouldn't have prevented this compromise.
- vaginaphobic 7mo ago
  [dead]
- lrvick 7mo ago
  I can not find a single signed recent commit on the axios repo. It is totally yolo mode. Those "signed by github" signatures are meaningless. I stand by my comment in full. One must sign commits -universally- and -also- sign reviews/merges (multi-party) and then -also- do multi party signing on releases. Doing only one step of basic supply chain security unfortunately buys you about as much defense as locking only a single door. I do however certainly assign significant blame to the NPM team though for repeatedly refusing optional package signing support so packages with signing enabled can be refused at the server and client if unsigned by a quorum of pinned keys, but even aside from that if packages were signed manually then canary tools could have detected this immediately.
  - dns_snek 7mo ago
    What you sign or don't sign in your Git repo doesn't matter because NPM doesn't publish from a Git repo. Signing commits is still useful for your contributors and downstream forks but it won't have any effect on the users who use your package via NPM. I think NPM is fully to blame here. Packages that exceed a certain level of popularity should require signing/strong 2FA. They should implement more schemes that publishers can optionally enable, like requiring mandatory sign-off from more than 1 maintainer before the package is available to download. Then on the package page it should say: "[Warning] Weak publishing protection" or "[Checkmark] This package requires sign-off from accountA and accountB to publish".
Zopieux 7mo ago
This only works up to a point. Some human needs some way of changing the publication setup in case something goes wrong or changes. What you're asking is blowing a proverbial e-fuse once the setup is known to be working. This is software, shit will go wrong at some point and you need a way to make changes.
- lrvick 7mo ago
  Of course, which is why all the (decent) tooling for this is provider agnostic, and provides documentation for multi-party-sharded backups so a quorum of maintainers can always re-assemble the key by hand for any reason if needed.
patrakov 7mo ago
The "nothing gets on main without two signatures" rule would not have prevented the xz story, where a comaintainer was able to smuggle malicious code past the review as "binary data for new tests" and, effectively, get it signed.

by lrvick 7mo ago

TheTaytay 7mo ago

It wasn’t done through git. It was a direct npm publish from the compromised machine. If you read further down in the comments (https://github.com/axios/axios/issues/10636#issuecomment-4181516546 https://github.com/axios/axios/issues/10636#issuecomment-418...), it seems difficult to pick the right npm settings to prevent this attack. If I understand it correctly, your suggestions wouldn’t have prevented it, which is evidence that this is not as trivially fixable as you believe it is.

lrvick 7mo ago
To prevent supply chain attacks you need multi party cryptographic attestation at every layer, which is pretty straight forward, but you are correct, NPM and GitHub controls absolutely will not save you. Microsoft insists their centralized approach can work, but we have plenty of evidence it does not. Operate under the assumption all accounts will be taken over because centralized corporate auth systems are fundamentally vulnerable. This is how you actually fix it: 1. Every commit must be signed by a maintainer key listed in the MAINTAINERS file or similar 2. Every review/merge must be signed by a -second- maintainer key 3. Every artifact must be build deterministically and be signed by multiple maintainers. 4. Have only one online npm publish key maintained in a deterministic and remotely attestable enclave that validates multiple valid maintainer signatures 5. Automatically sound the alarm if an NPM release is pushed any other way, and automatically revoke it.
- charcircuit 7mo ago
  And for 5 there should be help on the NPM end to make it so that the alarms can fire before the new update is actually revealed to the public. There could be a short staging time where it could be revoked before any harm has been done. During this staging time NPM should also scan the package through a malware scanner before allowing it to go public.
  - lrvick 7mo ago
    I agree that would be nice, but NPM absolutely will not do any basic supply chain integrity work. They are actively opposed to it citing concerns that it might turn off lower skill developers that would be too annoyed by tapping a yubikey to sign releases or code. I have talked to them enough times over the years to have completely given up here. Whats even more stupid is they actually started mandating 2FA for high risk packages, and FIDO2 supports being used to actually sign artifacts, but they instead simply use it for auth, and let releases stay unsigned. Even the developers they insisted hold cryptographic signing keys, they insist on only throw-away signatures for auth, but not using them for artifact signing to prevent impersonation. It is golf clap level stupid. Consider them a CDN that wants to analyze your code for AI training for their employer and nothing more. Any security controls that might restrict the flow of publishing even a little bit will be rejected.
    - deleted 7mo ago
      [deleted]

fortuitous-frog 7mo ago

Did you investigate the maintainer compromise and publication path? The malicious version was never committed or pushed via git. The maintainer signs his commits, and v1 releases were using OIDC and provenance attestations. The malicious package versions were published locally using the npm cli after the maintainer's machine was compromised via a RAT; there's no way for package maintainers to disable/forbid local publication on npmjs. It seems the Axios team was largely practicing what you're preaching. To the extent they aren't: it still wouldn't have prevented this compromise.

vaginaphobic 7mo ago
[dead]
lrvick 7mo ago
I can not find a single signed recent commit on the axios repo. It is totally yolo mode. Those "signed by github" signatures are meaningless. I stand by my comment in full. One must sign commits -universally- and -also- sign reviews/merges (multi-party) and then -also- do multi party signing on releases. Doing only one step of basic supply chain security unfortunately buys you about as much defense as locking only a single door. I do however certainly assign significant blame to the NPM team though for repeatedly refusing optional package signing support so packages with signing enabled can be refused at the server and client if unsigned by a quorum of pinned keys, but even aside from that if packages were signed manually then canary tools could have detected this immediately.
- dns_snek 7mo ago
  What you sign or don't sign in your Git repo doesn't matter because NPM doesn't publish from a Git repo. Signing commits is still useful for your contributors and downstream forks but it won't have any effect on the users who use your package via NPM. I think NPM is fully to blame here. Packages that exceed a certain level of popularity should require signing/strong 2FA. They should implement more schemes that publishers can optionally enable, like requiring mandatory sign-off from more than 1 maintainer before the package is available to download. Then on the package page it should say: "[Warning] Weak publishing protection" or "[Checkmark] This package requires sign-off from accountA and accountB to publish".

Zopieux 7mo ago

This only works up to a point. Some human needs some way of changing the publication setup in case something goes wrong or changes. What you're asking is blowing a proverbial e-fuse once the setup is known to be working. This is software, shit will go wrong at some point and you need a way to make changes.

lrvick 7mo ago
Of course, which is why all the (decent) tooling for this is provider agnostic, and provides documentation for multi-party-sharded backups so a quorum of maintainers can always re-assemble the key by hand for any reason if needed.

patrakov 7mo ago

The "nothing gets on main without two signatures" rule would not have prevented the xz story, where a comaintainer was able to smuggle malicious code past the review as "binary data for new tests" and, effectively, get it signed.