6 ms·

Sigstore: Making sure your software is what it claims to be

119 points by saikatsg 2y ago

linkregister 2y ago
Has anyone implemented this end-to-end? This seems production ready for smaller shops where it's feasible for developers to sign artifacts individually. For a system where you'd want CI to publish artifacts, and then use the k8s policy controller to only run verified artifacts, it seems incomplete. It appears the reason to include this system in a toolchain would be to meet compliance requirements, but even the GCP, AWS, and Azure implementations of artifact signing & verification are in beta.
- arccy 2y ago
  yes, i've implemented it in multiple companies. cosign supports using generated keys and kms services, that's been pretty stable and usable for a long time. keyless signing is different and you need to think a bit more carefully about what you're trusting.
  - eikenberry 2y ago
    I recently implemented a software updating system using [The Update Framework](https://theupdateframework.io/ https://theupdateframework.io/) directly, with [go-tuf](https://github.com/theupdateframework/go-tuf https://github.com/theupdateframework/go-tuf). It required a lot of design work around how we were going to do package management on top of using it for a secure updating system. This was due to TUF's designing around the capability for existing package management systems to adopt it and integrate it into their system. So TUF is very unopinionated and flexible. Given how TUF made it particularly hard to implement a system from scratch... How was your experience using Sigstore? Is it designed more around building systems from scratch? I.E. Is it more opinionated? Thanks.
    - arccy 2y ago
      TUF is much more comprehensive than what sigstore/cosign offers. at the core for sigtore/cosign are just the primitives for sign a blob / container and maybe some extra metadata, and for verifying the blob / container / metadata. there are other integrations that will also attach and sign a SBOM etc, but it's not necessary, so you can build something very simple such as: artifacts only signed by this key when it's run via ci on master, and deployments must run using artifacts signed with said key. sigstore is also a transparency log which you can check the signature against, but it's not really necessary, good for public things, you probably don't need it for private / company stuff.
  - linkregister 2y ago
    I designed a system using Sigstore where the signing key is in a secret store, and the CI shells out to the cosign CLI to perform the signing. Is this an antipattern? For verification, did you use the policy controller in kubernetes? Or are you manually performing the verification at runtime?
    - arccy 2y ago
      i used OPA in one org, and kyverno in another for verifying (reused whichever was already in place for other purposes). our teams always chose to go with cloud kms services for the signing keys, we thought they offered stronger access controls, and less need to revoke / rotate keys when access changes (team member leaves).
- remram 2y ago
  End-to-end it would require something like a web-of-trust or similar. There is little benefit in knowing that your package was definitely built by GitHub Actions definitely from the code that definitely came from the fingers of the random guy who maintains that particular tool. Unless you have some trust relationship with the author, or with someone that audited the code, the whole cryptographically-authenticated chain hangs from nothing. Tools like Crev did a lot of work in that area but it never really took off, people don't want to think about trust: https://github.com/crev-dev/cargo-crev https://github.com/crev-dev/cargo-crev
- woodruffw 2y ago
  > Has anyone implemented this end-to-end? Yes; I (along with a bunch of other fantastic folks) implemented it end-to-end for both Homebrew[1] and PyPI[2]. This is at a "lower" level than most corporate uses, however: the goal with integrating Sigstore into these OSS ecosystems is not to build up complex verifiable policies (which OSS maintainers don't want to deal with), but to enable signing with misuse-resistant machine identities. [1]: https://blog.trailofbits.com/2023/11/06/adding-build-provenance-to-homebrew/ https://blog.trailofbits.com/2023/11/06/adding-build-provena... [2]: https://blog.pypi.org/posts/2024-11-14-pypi-now-supports-digital-attestations/ https://blog.pypi.org/posts/2024-11-14-pypi-now-supports-dig...
  - KennyBlanken 2y ago
    [flagged]
    - woodruffw 2y ago
      Apart from the snark (which is unwarranted), I can't even parse what you're saying. (Mentioning sudo in the context of Homebrew suggests that you're one of those incoherent threat model people, so I'm going to assume it's that. So I'll say what Homebrew's maintainers have been saying for years: having a user writable Homebrew prefix is no more or less exploitable in the presence of attacker code execution than literally anything else. The attacker can always modify your shell initialization script, or your local Python bin directory, or anything else.)
      - saidinesh5 2y ago
        I'm not much of a Mac user but I'm guessing the parent comment wanted readonly user install path so accidental/malicious rm -rf only affects the user data and not the installed programs? Nothing to do with exploits as part of homebrew etc .
        woodruffw 2y ago
        > I'm not much of a Mac user but I'm guessing the parent comment wanted readonly user install path so accidental/malicious rm -rf only affects the user data and not the installed programs? That's one possible interpretation, but it's at odds with how most people use the `$PATH` anyways -- there's lots of local read-writable stuff on it in typical cases. And of course, even without that, persistence is trivial for an attacker with local code execution.
- firesteelrain 2y ago
  We use CodeLocker then oPA/Gatekeeper as the Admission Controller. Only signed artifacts are allowed to be executed. This is on AKS with ACR connected.
rough-sea 2y ago
JSR supports sigstore https://jsr.io/docs/trust https://jsr.io/docs/trust
deleted 2y ago
[deleted]
djhn 2y ago
Somewhat adjacent question: are there people working on ways to verify that a particular server or API backend are running the specific signed release that is open sourced? Can a company somehow cryptographically prove to its users that the running build is derived from the source unmodified?
- cperciva 2y ago
  You can do this with e.g. EC2 enclaves. Of course that's kind of begging the question, since you need to trust the enclaves.
- formerly_proven 2y ago
  That's what remote attestation in Intel SGX does. There's similar features in other platforms as well.
  - sublimefire 2y ago
    Yes Intel SGX gives you the enclave measurement which you could reproduce (if source code code is reproducible), the measurement can be verified against Intel keys to prove it came from that hardware. Similarly AMD SEV-SNP gives you that, it is preferred to SGX due to the ability to run VMs as opposed to smaller applications. AWS has their firmware image in OSS to be able to reproduce it and then compare the measurements in their Nitro instances: https://github.com/aws/uefi https://github.com/aws/uefi Azure has confidential compute offerings as well, their attestation, mhsm, ledger services rely on it. But it is easy to talk about confidential compute and the link between the measurement and the source code. Such link does not exist in regular non-encrypted services and you basically need to trust the service provider about the proofs they give you.
  - udev4096 2y ago
    https://xcancel.com/_markel___/status/1828112469010596347#m https://xcancel.com/_markel___/status/1828112469010596347#m https://sgx.fail https://sgx.fail
- shortsunblack 2y ago
deleted 2y ago
[deleted]
croes 2y ago
Does this help when a project change ownership or in cases like the xz backdoor?
- sublimefire 2y ago
  Transparency does not prevent it but rather adds an additional anchor to make it harder to spoof packages/binaries and detracts from doing it because it will be publicly logged. Somebody still needs to verify if all is good, e.g. if PR in the official repo adds some malicious code (think xz) then it might get published and logged in this transparency log system.
- blueflow 2y ago
  No. Malicious upstreams will have their software properly signed as theirs.
udev4096 2y ago
It seems really difficult to actually use it. For instance, a standard linux distro probably has thousands of packages, components, etc. How can you verify all of them? Even if you can, does it defend against attacks like xz where the trusted source itself is compromised
- eptcyka 2y ago
  It protects against someone making rogue builds - it should be obvious when a build is made using valid keys. So if you steal my keys, you won’t be able to covertly make a build and get one user of mine to trust it without making publishing the build. If you publish it, everyone knows, and can try and see where it came fron. Prevent against another xz it will not, but it can help against directed attacks.
- arccy 2y ago
  it's not much more difficult (maybe even easier) than the gpg signing / checking that distros generally like to do. with gpg, you get a root set of public keys that you want to trust. with sigstore, depending on the signing method, you either trust public keys, or identities (some oauth2 identity provider, like email, or your ci system).
eadmund 2y ago
It sounds neat, but I am uncomfortable with a central CA (Fulcio) and central log (Rekor). And I trust OIDC providers about as far as I can throw them. Granted, the whole point of a central audit log is to make misbehaviour apparent, but it still strikes me as the wrong direction. I don’t have a useful proposal for a decentralised version, so I’m just kvetching at this point. Also, neither X.509 nor JSON is great. We can do better. We should do better.
- tuananh 2y ago
  you can deploy your own fulcio & rekor.
- kfreds 2y ago
  Check out sigsum.org for a simpler design with a stronger threat model. As for building a decentralized append-only log, that would complicate the design and the threat model quite a bit. In particular it would make proofs of inclusion and consistency much less efficient.
hulitu 2y ago
> Sigstore: Making sure your software is what it claims to be > sign. verify. protect. Making sure your software is what it claims to be. I guess you never used Windows. No amount of signing will prevent it from phoning home. Spyware at its finest.

119 points by saikatsg 2y ago

Has anyone implemented this end-to-end? This seems production ready for smaller shops where it's feasible for developers to sign artifacts individually. For a system where you'd want CI to publish artifacts, and then use the k8s policy controller to only run verified artifacts, it seems incomplete. It appears the reason to include this system in a toolchain would be to meet compliance requirements, but even the GCP, AWS, and Azure implementations of artifact signing & verification are in beta.

arccy 2y ago
yes, i've implemented it in multiple companies. cosign supports using generated keys and kms services, that's been pretty stable and usable for a long time. keyless signing is different and you need to think a bit more carefully about what you're trusting.
- eikenberry 2y ago
  I recently implemented a software updating system using [The Update Framework](https://theupdateframework.io/ https://theupdateframework.io/) directly, with [go-tuf](https://github.com/theupdateframework/go-tuf https://github.com/theupdateframework/go-tuf). It required a lot of design work around how we were going to do package management on top of using it for a secure updating system. This was due to TUF's designing around the capability for existing package management systems to adopt it and integrate it into their system. So TUF is very unopinionated and flexible. Given how TUF made it particularly hard to implement a system from scratch... How was your experience using Sigstore? Is it designed more around building systems from scratch? I.E. Is it more opinionated? Thanks.
  - arccy 2y ago
    TUF is much more comprehensive than what sigstore/cosign offers. at the core for sigtore/cosign are just the primitives for sign a blob / container and maybe some extra metadata, and for verifying the blob / container / metadata. there are other integrations that will also attach and sign a SBOM etc, but it's not necessary, so you can build something very simple such as: artifacts only signed by this key when it's run via ci on master, and deployments must run using artifacts signed with said key. sigstore is also a transparency log which you can check the signature against, but it's not really necessary, good for public things, you probably don't need it for private / company stuff.
- linkregister 2y ago
  I designed a system using Sigstore where the signing key is in a secret store, and the CI shells out to the cosign CLI to perform the signing. Is this an antipattern? For verification, did you use the policy controller in kubernetes? Or are you manually performing the verification at runtime?
  - arccy 2y ago
    i used OPA in one org, and kyverno in another for verifying (reused whichever was already in place for other purposes). our teams always chose to go with cloud kms services for the signing keys, we thought they offered stronger access controls, and less need to revoke / rotate keys when access changes (team member leaves).
remram 2y ago
End-to-end it would require something like a web-of-trust or similar. There is little benefit in knowing that your package was definitely built by GitHub Actions definitely from the code that definitely came from the fingers of the random guy who maintains that particular tool. Unless you have some trust relationship with the author, or with someone that audited the code, the whole cryptographically-authenticated chain hangs from nothing. Tools like Crev did a lot of work in that area but it never really took off, people don't want to think about trust: https://github.com/crev-dev/cargo-crev https://github.com/crev-dev/cargo-crev
woodruffw 2y ago
> Has anyone implemented this end-to-end? Yes; I (along with a bunch of other fantastic folks) implemented it end-to-end for both Homebrew[1] and PyPI[2]. This is at a "lower" level than most corporate uses, however: the goal with integrating Sigstore into these OSS ecosystems is not to build up complex verifiable policies (which OSS maintainers don't want to deal with), but to enable signing with misuse-resistant machine identities. [1]: https://blog.trailofbits.com/2023/11/06/adding-build-provenance-to-homebrew/ https://blog.trailofbits.com/2023/11/06/adding-build-provena... [2]: https://blog.pypi.org/posts/2024-11-14-pypi-now-supports-digital-attestations/ https://blog.pypi.org/posts/2024-11-14-pypi-now-supports-dig...
- KennyBlanken 2y ago
  [flagged]
  - woodruffw 2y ago
    Apart from the snark (which is unwarranted), I can't even parse what you're saying. (Mentioning sudo in the context of Homebrew suggests that you're one of those incoherent threat model people, so I'm going to assume it's that. So I'll say what Homebrew's maintainers have been saying for years: having a user writable Homebrew prefix is no more or less exploitable in the presence of attacker code execution than literally anything else. The attacker can always modify your shell initialization script, or your local Python bin directory, or anything else.)
    - saidinesh5 2y ago
      I'm not much of a Mac user but I'm guessing the parent comment wanted readonly user install path so accidental/malicious rm -rf only affects the user data and not the installed programs? Nothing to do with exploits as part of homebrew etc .
      - woodruffw 2y ago
        > I'm not much of a Mac user but I'm guessing the parent comment wanted readonly user install path so accidental/malicious rm -rf only affects the user data and not the installed programs? That's one possible interpretation, but it's at odds with how most people use the `$PATH` anyways -- there's lots of local read-writable stuff on it in typical cases. And of course, even without that, persistence is trivial for an attacker with local code execution.
firesteelrain 2y ago
We use CodeLocker then oPA/Gatekeeper as the Admission Controller. Only signed artifacts are allowed to be executed. This is on AKS with ACR connected.

rough-sea 2y ago

JSR supports sigstore https://jsr.io/docs/trust https://jsr.io/docs/trust

deleted 2y ago

[deleted]

djhn 2y ago

Somewhat adjacent question: are there people working on ways to verify that a particular server or API backend are running the specific signed release that is open sourced? Can a company somehow cryptographically prove to its users that the running build is derived from the source unmodified?

cperciva 2y ago
You can do this with e.g. EC2 enclaves. Of course that's kind of begging the question, since you need to trust the enclaves.
formerly_proven 2y ago
That's what remote attestation in Intel SGX does. There's similar features in other platforms as well.
- sublimefire 2y ago
  Yes Intel SGX gives you the enclave measurement which you could reproduce (if source code code is reproducible), the measurement can be verified against Intel keys to prove it came from that hardware. Similarly AMD SEV-SNP gives you that, it is preferred to SGX due to the ability to run VMs as opposed to smaller applications. AWS has their firmware image in OSS to be able to reproduce it and then compare the measurements in their Nitro instances: https://github.com/aws/uefi https://github.com/aws/uefi Azure has confidential compute offerings as well, their attestation, mhsm, ledger services rely on it. But it is easy to talk about confidential compute and the link between the measurement and the source code. Such link does not exist in regular non-encrypted services and you basically need to trust the service provider about the proofs they give you.
- udev4096 2y ago
  https://xcancel.com/_markel___/status/1828112469010596347#m https://xcancel.com/_markel___/status/1828112469010596347#m https://sgx.fail https://sgx.fail
shortsunblack 2y ago

deleted 2y ago

[deleted]

croes 2y ago

Does this help when a project change ownership or in cases like the xz backdoor?

sublimefire 2y ago
Transparency does not prevent it but rather adds an additional anchor to make it harder to spoof packages/binaries and detracts from doing it because it will be publicly logged. Somebody still needs to verify if all is good, e.g. if PR in the official repo adds some malicious code (think xz) then it might get published and logged in this transparency log system.
blueflow 2y ago
No. Malicious upstreams will have their software properly signed as theirs.

udev4096 2y ago

It seems really difficult to actually use it. For instance, a standard linux distro probably has thousands of packages, components, etc. How can you verify all of them? Even if you can, does it defend against attacks like xz where the trusted source itself is compromised

eptcyka 2y ago
It protects against someone making rogue builds - it should be obvious when a build is made using valid keys. So if you steal my keys, you won’t be able to covertly make a build and get one user of mine to trust it without making publishing the build. If you publish it, everyone knows, and can try and see where it came fron. Prevent against another xz it will not, but it can help against directed attacks.
arccy 2y ago
it's not much more difficult (maybe even easier) than the gpg signing / checking that distros generally like to do. with gpg, you get a root set of public keys that you want to trust. with sigstore, depending on the signing method, you either trust public keys, or identities (some oauth2 identity provider, like email, or your ci system).

eadmund 2y ago

It sounds neat, but I am uncomfortable with a central CA (Fulcio) and central log (Rekor). And I trust OIDC providers about as far as I can throw them. Granted, the whole point of a central audit log is to make misbehaviour apparent, but it still strikes me as the wrong direction. I don’t have a useful proposal for a decentralised version, so I’m just kvetching at this point. Also, neither X.509 nor JSON is great. We can do better. We should do better.

tuananh 2y ago
you can deploy your own fulcio & rekor.
kfreds 2y ago
Check out sigsum.org for a simpler design with a stronger threat model. As for building a decentralized append-only log, that would complicate the design and the threat model quite a bit. In particular it would make proofs of inclusion and consistency much less efficient.

hulitu 2y ago

> Sigstore: Making sure your software is what it claims to be > sign. verify. protect. Making sure your software is what it claims to be. I guess you never used Windows. No amount of signing will prevent it from phoning home. Spyware at its finest.