4 ms·

Two-factor auth for publishing is helpful, but requiring cryptographically signed approval by multiple authors would be more helpful. Then compromising a single

Two-factor auth for publishing is helpful, but requiring cryptographically signed approval by multiple authors would be more helpful. Then compromising a single author wouldn't be enough.

tcoff91 1y ago
Many packages have only 1 author.
- chrisweekly 1y ago
  and (as in this case), that 1 author may use a single token to authz publishing many packages
- deleted 1y ago
  [deleted]
- rectang 1y ago
  The conclusion I'm coming to is that depending on packages which only have a single author is problematic. There are too many ways that packages published by one person can be compromised. Packages which don't have approval and review by a reliable third party shouldn't be visible by default in a package manager.
  - Hackbraten 1y ago
    How are you supposed to gain collaborators for a project that no one can possibly find?
    - rectang 1y ago
      There are ways, but at a high level, I don't care. I hate how modern package managers have come to value author convenience over downstream user security.
      - Hackbraten 1y ago
        Fair enough. In the meantime, I'm trying to do my part through occasional random spot inspections when there's an update to a package, and encourage others to do the same for swarm coverage.
      - whatevaa 1y ago
        Ahh, the classic I don't care. What if other people don't care about your problems? What if both sides don't care about each other? What then?
        rectang 1y ago
        We wait and see whether the supply chain attacks crescendo to a crisis and force NPM's hand. In the meantime I'm doing everything I can to avoid NPM and to uphold "just don't use the software if you don't like it"... but people like myself don't always have a choice.
  - x0x0 1y ago
    That's a lot of entitlement for things you haven't paid a cent for; not just multiple authors but trusted 3rd parties; approval and review; etc.
    - rectang 1y ago
      I’ve done all those things myself (past ASF member where all that and more was SOP), so I realize what I’m asking for. It’s not crazy for authors of small packages to form small collectives and serve as each others’ trusted third parties. In any case, if the choice is “frequent supply chain compromise, take it or leave it”, the answer is of course “leave it”. If we need to pay for curated packages because the problems with NPM are endemic, that’s not unreasonable.
      - x0x0 1y ago
        > It’s not crazy for authors of small packages to form small collectives and serve as each others’ trusted third parties. Yeah, there's that insane entitlement. More demands for others' time and labor, plus the conflation between you demanding labor vs if people don't agree to your free labor demands, they're pro supply chain compromise.
        rectang 1y ago
        In a general discussion forum, I have floated some approaches for hardening distribution which have proven effective in other communities. If NPM can harden their systems using other mechanisms, then more power to them.
      - practorz 1y ago
        >In any case, if the choice is “frequent supply chain compromise, take it or leave it”, the answer is of course “leave it”. There's another choice: vendor your dependencies and manually review and vet updates. That solves all your problems, no need for "trusted third parties", you are the one vetting it, only need to trust yourself.
        1y ago
  - bigiain 1y ago
    How many of your dependencies have 2nd level dependencies which have even deeper dependencies on ZX Utils, or NX (or left_pad.js)? (right now I don't know the answer to that for the stuff I'm responsible for, but I'm in the process of researching and setting up and configuring the sort of tools needed to automate that.)
- KronisLV 1y ago
  I'm not sure why we never got around to more human in the loop with 2FA when it comes to this sort of stuff: "Oh, you want to publish a new package? Okay, confirm it on this app on your device/phone to make sure." Surely a button press on a pre-approved device wouldn't be too hard, pretty much how every user initiated online banking payment over here goes like. I once heard from a sysadmin that didn't want to automate certificate renewal and other things, because he believed that doing so would take away useful skills or some inner knowledge of how the system works. Because of the human error risk, I thought that was stupid, but when it comes to approval processes, I think it makes sense. Especially because pushing code doesn't necessarily mean the same thing as such an approval, or the main device that you push code from could also get compromised, using your phone as 2FA could save you. Then again, maybe I'm also stupid and the way we build our software is messed up on a fundamental level with all of the dependencies and nobody being able to practically audit all of the code they import, given deadlines, limited skills and resources and so on. Maybe it's all just fighting against a windmill.
  - rectang 1y ago
    I don’t think the current state of software development is irredeemable. Ongoing downstream review of all dependency code is practical for only a tiny fraction of projects; for most projects using publisher reputation as a proxy for package safety is reasonable. What’s not working is the low-standards package managers where inconveniencing authors is never acceptable because the whole enterprise is built on popularity with authors — you can’t trust that what those package managers give you reflects author intent.
cyphar 1y ago
While multiple authors' signatures would be nice, a lot of these kinds of attacks would be solved if there was any signature verification being done of the commits, tags, or generated artefacts at all. People like to complain about distribution packaging being obtuse, but most distributions have rich support for verifying that package sources were signed by a key in a keyring that is maintained by the distribution. My (somewhat biased) view is that language package managers still do not provide the same set of features for validation that (for instance) rpmbuild does. The release process for runc has the following safeguards: * As the upstream maintainer of runc, our releases are all signed with one of a set of keys that are maintained in our repo[1]. Our tags are also signed by one of the same keys. In my case, my key is stored in a Yubikey and so cannot easily be exfiltrated. * Our release scripts include a step which validate that all of the keys in that keyring file are valid (sub)keys registered to the GitHub account of a maintainer[2]. They also prompt the person doing the signing to check that the list looks reasonable before signing anything[3]. * Distributions such as openSUSE have a copy of the keyring file[4] and the build system will automatically reject the build if the source code archive is not signed. Our official binary releases are also signed and so can be validated in a similar manner. Maybe there are still gaps in this setup, and I would love to hear them. But I think this setup would have blocked this kind of attack at several stages. I personally don't like the idea of signing releases in CI -- if you really want to build your binaries in CI, that's fine, but you should always require a maintainer to personally sign the binaries at the end of the process. For language package managers that do not support such a workflow, trusted publishing is a less awful setup than having long-lived publishing keys that may be incorrectly scoped (as happened in this case) but it still allows someone who gains access to your GitHub account (such as by stealing your cookies) to publish updated versions of your package with very little resource. GitHub supports setting a mandatory timeout for trusted publishing but the attacker could easily disable that. If someone got access to my GitHub account, it would be a very bad day but distributions would not accept the new releases because their copy of our keyring would not include the attackers keys (even if they added them to my account). Disclaimer: I work at SUSE, though I will say that I would like for OBS to have nice support for validating checksums of artefacts like Arch and Gentoo do (you can /theoretically/ do it with OBS services or emulate it with forcelocal -- and most packages actually store the archive in OBS rather than pulling it at build time -- but it would be nice to do both). [1]: https://github.com/opencontainers/runc/blob/v1.4.0-rc.1/runc.keyring https://github.com/opencontainers/runc/blob/v1.4.0-rc.1/runc... [2]: https://github.com/opencontainers/runc/blob/v1.4.0-rc.1/script/keyring_validate.sh https://github.com/opencontainers/runc/blob/v1.4.0-rc.1/scri... [3]: https://github.com/opencontainers/runc/blob/v1.4.0-rc.1/script/release_sign.sh#L120 https://github.com/opencontainers/runc/blob/v1.4.0-rc.1/scri... [4]: https://build.opensuse.org/projects/openSUSE:Factory/packages/runc/files/runc.keyring?expand=1 https://build.opensuse.org/projects/openSUSE:Factory/package...

Two-factor auth for publishing is helpful, but requiring cryptographically signed approval by multiple authors would be more helpful. Then compromising a single author wouldn't be enough.

tcoff91 1y ago

Many packages have only 1 author.

chrisweekly 1y ago
and (as in this case), that 1 author may use a single token to authz publishing many packages
deleted 1y ago
[deleted]
rectang 1y ago
The conclusion I'm coming to is that depending on packages which only have a single author is problematic. There are too many ways that packages published by one person can be compromised. Packages which don't have approval and review by a reliable third party shouldn't be visible by default in a package manager.
- Hackbraten 1y ago
  How are you supposed to gain collaborators for a project that no one can possibly find?
  - rectang 1y ago
    There are ways, but at a high level, I don't care. I hate how modern package managers have come to value author convenience over downstream user security.
    - Hackbraten 1y ago
      Fair enough. In the meantime, I'm trying to do my part through occasional random spot inspections when there's an update to a package, and encourage others to do the same for swarm coverage.
    - whatevaa 1y ago
      Ahh, the classic I don't care. What if other people don't care about your problems? What if both sides don't care about each other? What then?
      - rectang 1y ago
        We wait and see whether the supply chain attacks crescendo to a crisis and force NPM's hand. In the meantime I'm doing everything I can to avoid NPM and to uphold "just don't use the software if you don't like it"... but people like myself don't always have a choice.
- x0x0 1y ago
  That's a lot of entitlement for things you haven't paid a cent for; not just multiple authors but trusted 3rd parties; approval and review; etc.
  - rectang 1y ago
    I’ve done all those things myself (past ASF member where all that and more was SOP), so I realize what I’m asking for. It’s not crazy for authors of small packages to form small collectives and serve as each others’ trusted third parties. In any case, if the choice is “frequent supply chain compromise, take it or leave it”, the answer is of course “leave it”. If we need to pay for curated packages because the problems with NPM are endemic, that’s not unreasonable.
    - x0x0 1y ago
      > It’s not crazy for authors of small packages to form small collectives and serve as each others’ trusted third parties. Yeah, there's that insane entitlement. More demands for others' time and labor, plus the conflation between you demanding labor vs if people don't agree to your free labor demands, they're pro supply chain compromise.
      - rectang 1y ago
        In a general discussion forum, I have floated some approaches for hardening distribution which have proven effective in other communities. If NPM can harden their systems using other mechanisms, then more power to them.
    - practorz 1y ago
      >In any case, if the choice is “frequent supply chain compromise, take it or leave it”, the answer is of course “leave it”. There's another choice: vendor your dependencies and manually review and vet updates. That solves all your problems, no need for "trusted third parties", you are the one vetting it, only need to trust yourself.
      - 1y ago
- bigiain 1y ago
  How many of your dependencies have 2nd level dependencies which have even deeper dependencies on ZX Utils, or NX (or left_pad.js)? (right now I don't know the answer to that for the stuff I'm responsible for, but I'm in the process of researching and setting up and configuring the sort of tools needed to automate that.)
KronisLV 1y ago
I'm not sure why we never got around to more human in the loop with 2FA when it comes to this sort of stuff: "Oh, you want to publish a new package? Okay, confirm it on this app on your device/phone to make sure." Surely a button press on a pre-approved device wouldn't be too hard, pretty much how every user initiated online banking payment over here goes like. I once heard from a sysadmin that didn't want to automate certificate renewal and other things, because he believed that doing so would take away useful skills or some inner knowledge of how the system works. Because of the human error risk, I thought that was stupid, but when it comes to approval processes, I think it makes sense. Especially because pushing code doesn't necessarily mean the same thing as such an approval, or the main device that you push code from could also get compromised, using your phone as 2FA could save you. Then again, maybe I'm also stupid and the way we build our software is messed up on a fundamental level with all of the dependencies and nobody being able to practically audit all of the code they import, given deadlines, limited skills and resources and so on. Maybe it's all just fighting against a windmill.
- rectang 1y ago
  I don’t think the current state of software development is irredeemable. Ongoing downstream review of all dependency code is practical for only a tiny fraction of projects; for most projects using publisher reputation as a proxy for package safety is reasonable. What’s not working is the low-standards package managers where inconveniencing authors is never acceptable because the whole enterprise is built on popularity with authors — you can’t trust that what those package managers give you reflects author intent.

cyphar 1y ago

While multiple authors' signatures would be nice, a lot of these kinds of attacks would be solved if there was any signature verification being done of the commits, tags, or generated artefacts at all. People like to complain about distribution packaging being obtuse, but most distributions have rich support for verifying that package sources were signed by a key in a keyring that is maintained by the distribution. My (somewhat biased) view is that language package managers still do not provide the same set of features for validation that (for instance) rpmbuild does. The release process for runc has the following safeguards: * As the upstream maintainer of runc, our releases are all signed with one of a set of keys that are maintained in our repo[1]. Our tags are also signed by one of the same keys. In my case, my key is stored in a Yubikey and so cannot easily be exfiltrated. * Our release scripts include a step which validate that all of the keys in that keyring file are valid (sub)keys registered to the GitHub account of a maintainer[2]. They also prompt the person doing the signing to check that the list looks reasonable before signing anything[3]. * Distributions such as openSUSE have a copy of the keyring file[4] and the build system will automatically reject the build if the source code archive is not signed. Our official binary releases are also signed and so can be validated in a similar manner. Maybe there are still gaps in this setup, and I would love to hear them. But I think this setup would have blocked this kind of attack at several stages. I personally don't like the idea of signing releases in CI -- if you really want to build your binaries in CI, that's fine, but you should always require a maintainer to personally sign the binaries at the end of the process. For language package managers that do not support such a workflow, trusted publishing is a less awful setup than having long-lived publishing keys that may be incorrectly scoped (as happened in this case) but it still allows someone who gains access to your GitHub account (such as by stealing your cookies) to publish updated versions of your package with very little resource. GitHub supports setting a mandatory timeout for trusted publishing but the attacker could easily disable that. If someone got access to my GitHub account, it would be a very bad day but distributions would not accept the new releases because their copy of our keyring would not include the attackers keys (even if they added them to my account). Disclaimer: I work at SUSE, though I will say that I would like for OBS to have nice support for validating checksums of artefacts like Arch and Gentoo do (you can /theoretically/ do it with OBS services or emulate it with forcelocal -- and most packages actually store the archive in OBS rather than pulling it at build time -- but it would be nice to do both). [1]: https://github.com/opencontainers/runc/blob/v1.4.0-rc.1/runc.keyring https://github.com/opencontainers/runc/blob/v1.4.0-rc.1/runc... [2]: https://github.com/opencontainers/runc/blob/v1.4.0-rc.1/script/keyring_validate.sh https://github.com/opencontainers/runc/blob/v1.4.0-rc.1/scri... [3]: https://github.com/opencontainers/runc/blob/v1.4.0-rc.1/script/release_sign.sh#L120 https://github.com/opencontainers/runc/blob/v1.4.0-rc.1/scri... [4]: https://build.opensuse.org/projects/openSUSE:Factory/packages/runc/files/runc.keyring?expand=1 https://build.opensuse.org/projects/openSUSE:Factory/package...