8 ms·

Improving software supply chain security with tamper-proof builds

52 points by _-david-_ 5y ago

rmah 5y ago
Am I the only one annoyed that they use the term "supply chain" for a software build process?
- firstSpeaker 5y ago
  Yes.
- bushbaba 5y ago
  It's a fair term for supplying software to other organizations. Where there's chance for it to be tampered with between the two orgs.
- beams_of_light 5y ago
  The verbiage is being extended from hardware supply chain security efforts.
firstSpeaker 5y ago
Supply chain protection and security is making its way into thoughtworks radar as well: https://www.thoughtworks.com/radar/techniques https://www.thoughtworks.com/radar/techniques
- mistrial9 5y ago
  they say "Software Bill of Materials" in that newsletter, apparently
ris 5y ago
There is so much going on in the world of SBOM that seems so wrong-headed to me, and this is a good example. So much of the effort seems to be going into processes and systems that do little more than build a chain of trust back to someone or something that says "yeah just trust me". Here, from what I can tell (and I may have got this wrong), the "guarantee" of an un-tamperable build seems to come from some behaviour that some product (GitHub Actions) happens to exhibit. Today. That feature subtly changes tomorrow or someone finds a(nother) hole in GitHub Actions security model, it means nothing. I've generally come to believe that supply chain security comes from reproducible builds - though, more importantly, the build transparency that enables those builds to be performed reproducibly. If I can get two disparate, differently-hosted build machines to generate an identical artifact for me, that gives me far more confidence than relying on any behaviours that GitHub Actions is supposed to have. If I can then decide to ignore the pre-built package and choose to very easily perform the full scratch-build on my own systems and inspect every part of that process, that's even more convincing. A lot of the efforts around SBOM give me the impression that they're trying to solve the problem using lightly-digitized bureaucracy (which, as pointed out by Luis Villa in https://blog.tidelift.com/pay-to-play-dont-expect-maintainers-to-solve-your-supply-chain-issues-for-free https://blog.tidelift.com/pay-to-play-dont-expect-maintainer... is a game that many FOSS maintainers may not want to play), whereas, as a Nix person myself (oh hadn't you guessed by now?) it's always struck me that this is really a build system problem and most of the work should be doable automatically. (I guess the approaches I've been seeing are also related to the fixation we seem to have in ops/ci/cd standard practices these days with the "push" model of many individual build processes & steps "pushing" an opaque artifact to the next in a pipeline. It's an idea baked into most components from Docker to .. really every ci pipeline syntax I've seen. But it's an idea that's strewn with subtle complexities - from stale artifacts of previous failed runs to race conditions with dire consequences. In my experience life becomes a lot simpler when your destination instead understands exactly what it needs, can "pull" in exactly those resources and have them built to their specifications or build them itself..)
- skybrian 5y ago
  Sure, but it seems like the next steps would be (1) make a local build tool that works like GitHub Actions and (2) make your builds reproducible. It seems doable?
  - ris 5y ago
    No I think that's a perfect example of why this doesn't work. Once you remove the build tool from the assumed-impenetrable GitHub, where are your guarantees of non-tamperability for it? How is what you have now not "just" another build tool?
    - skybrian 5y ago
      GitHub Actions is somewhat trustworthy, and other people can verify that it's doing the right thing by comparing results. If enough other people vouch for a build (and I don't think you need that many), you don't need to do it yourself. This doesn't seem all that different from how certificate transparency helps to build trust in DNS registries. Can you trust them to do their jobs? Well, it helps that if they do certain things wrong, someone will notice.
      - ris 5y ago
        > and other people can verify that it's doing the right thing by comparing results You're essentially describing the solution I outlined in my original post. What is the proposal in this article actually adding that's not painfully obvious?
        skybrian 5y ago
        Apparently they have an implementation making GitHub Actions one of the parties that's providing results to compare. I don't know how novel that is, but it seems useful?
traceroute66 5y ago
I'm slightly confused. How does relying on multiple third-party moving parts improve your supply chain ? A quick glance through the blog post shows, for example: - A reliance on github runners to compile (rather than in-house) - A reliance on a third-party signing service (rather than own CA) I'm probably missing something obvious, but I just don't get how this helps ? In my mind it seems to add extra places for security issues to be introduced ?
- orthecreedence 5y ago
  > I'm probably missing something obvious No, you're not. The article is actually just braindead enough to suggest that shipping your build and cryptographic signing steps to third-parties is more secure. This isn't just wrong, it's almost criminally devious (and blatantly in service to the ever-growing surveillance state).
  - hulitu 5y ago
    You are right except with using the adverb "braindead". I really think they are doing it on purpose. The aim is that the netizen gets a licence for everything he has: car, house, devices. The "Companies" control every device and the life of the netizen . And this security theater is just there that the netizen does not interfere with their data collection.
- woodruffw 5y ago
  > I'm probably missing something obvious, but I just don't get how this helps ? In my mind it seems to add extra places for security issues to be introduced ? The entire point is integrity and a fully auditable chain of actions. Quoting the post: > Performing these steps guarantees to the consumer that the binary was produced in the trusted builder at a given commit hash attested to in the provenance. They can trust that the information in the provenance was non-forgeable, allowing them to trust the build “recipe” and trace their artifact verifiably back to the source. This provides a broad swath of integrity guarantees that are otherwise extremely difficult to assert for builds that you don't control: you know exactly which machine ran the build, that it wasn't tampered with or silently replaced, and that a particular OpenID identity has signed for the build. You can verify all of these steps publicly without having to keep a trusted keychain, and without having to trust that the signing/publishing parties securely store long-lived signing keys (since everything is ephemeral). The critical thing to note here is that GitHub Actions is just the platform here. The techniques are all open, and sigstore is an open specification with an open implementation. You can do it all in-house if you'd like, or ignore it completely if everything you consume is already in-house.
aftbit 5y ago
>GitHub now supports OIDC in their workflows. Each time a workflow is run, a runner can mint a unique JWT token from GitHub’s OIDC provider. The token contains verifiable information of the workflow identity, including the caller repository, commit hash, trigger, and the current (reuseable) workflow path and reference. Cool, so just emit that token along with your build artifacts. Anyone who trusts Github can then verify that it was signed by them, and there is no need for any of the CA nonsense.
- woodruffw 5y ago
  > Cool, so just emit that token along with your build artifacts. Anyone who trusts Github can then verify that it was signed by them, and there is no need for any of the CA nonsense. This works if you need to verify JWTs from relatively recent packages, but packaging has a long tail. There's no guarantee that GitHub is still publishing their expired JWK sets a decade after the last valid signature for them has gone out.
webmaven 5y ago
Even assuming that the process being described works as advertised, the result is in no way, shape, or form "tamper-proof". At best, it builds software that is "tamper-evident". A tamper-proof solution would produce software that somehow magically foils all attempts to modify or replace it, which isn't really possible (though some malware is somewhat tamper-resistant in that sense).
amluto 5y ago
This is a surprisingly baroque solution for attesting to a computation being done as specified. Imagine how much simpler it would be if GitHub (or Google or Gitlab or, better yet, multiple vendors) actually supported this style of attestation natively. I would love for app store vendors to put their money where their mouth is: allow reproducibly-build open source (or source-available) apps to be verified by the store and to have a clear badge in the store that indicates that you’re getting the app you think you are.

52 points by _-david-_ 5y ago

rmah 5y ago

Am I the only one annoyed that they use the term "supply chain" for a software build process?

firstSpeaker 5y ago
Yes.
bushbaba 5y ago
It's a fair term for supplying software to other organizations. Where there's chance for it to be tampered with between the two orgs.
beams_of_light 5y ago
The verbiage is being extended from hardware supply chain security efforts.

firstSpeaker 5y ago

Supply chain protection and security is making its way into thoughtworks radar as well: https://www.thoughtworks.com/radar/techniques https://www.thoughtworks.com/radar/techniques

mistrial9 5y ago
they say "Software Bill of Materials" in that newsletter, apparently

ris 5y ago

There is so much going on in the world of SBOM that seems so wrong-headed to me, and this is a good example. So much of the effort seems to be going into processes and systems that do little more than build a chain of trust back to someone or something that says "yeah just trust me". Here, from what I can tell (and I may have got this wrong), the "guarantee" of an un-tamperable build seems to come from some behaviour that some product (GitHub Actions) happens to exhibit. Today. That feature subtly changes tomorrow or someone finds a(nother) hole in GitHub Actions security model, it means nothing. I've generally come to believe that supply chain security comes from reproducible builds - though, more importantly, the build transparency that enables those builds to be performed reproducibly. If I can get two disparate, differently-hosted build machines to generate an identical artifact for me, that gives me far more confidence than relying on any behaviours that GitHub Actions is supposed to have. If I can then decide to ignore the pre-built package and choose to very easily perform the full scratch-build on my own systems and inspect every part of that process, that's even more convincing. A lot of the efforts around SBOM give me the impression that they're trying to solve the problem using lightly-digitized bureaucracy (which, as pointed out by Luis Villa in https://blog.tidelift.com/pay-to-play-dont-expect-maintainers-to-solve-your-supply-chain-issues-for-free https://blog.tidelift.com/pay-to-play-dont-expect-maintainer... is a game that many FOSS maintainers may not want to play), whereas, as a Nix person myself (oh hadn't you guessed by now?) it's always struck me that this is really a build system problem and most of the work should be doable automatically. (I guess the approaches I've been seeing are also related to the fixation we seem to have in ops/ci/cd standard practices these days with the "push" model of many individual build processes & steps "pushing" an opaque artifact to the next in a pipeline. It's an idea baked into most components from Docker to .. really every ci pipeline syntax I've seen. But it's an idea that's strewn with subtle complexities - from stale artifacts of previous failed runs to race conditions with dire consequences. In my experience life becomes a lot simpler when your destination instead understands exactly what it needs, can "pull" in exactly those resources and have them built to their specifications or build them itself..)

skybrian 5y ago
Sure, but it seems like the next steps would be (1) make a local build tool that works like GitHub Actions and (2) make your builds reproducible. It seems doable?
- ris 5y ago
  No I think that's a perfect example of why this doesn't work. Once you remove the build tool from the assumed-impenetrable GitHub, where are your guarantees of non-tamperability for it? How is what you have now not "just" another build tool?
  - skybrian 5y ago
    GitHub Actions is somewhat trustworthy, and other people can verify that it's doing the right thing by comparing results. If enough other people vouch for a build (and I don't think you need that many), you don't need to do it yourself. This doesn't seem all that different from how certificate transparency helps to build trust in DNS registries. Can you trust them to do their jobs? Well, it helps that if they do certain things wrong, someone will notice.
    - ris 5y ago
      > and other people can verify that it's doing the right thing by comparing results You're essentially describing the solution I outlined in my original post. What is the proposal in this article actually adding that's not painfully obvious?
      - skybrian 5y ago
        Apparently they have an implementation making GitHub Actions one of the parties that's providing results to compare. I don't know how novel that is, but it seems useful?

traceroute66 5y ago

I'm slightly confused. How does relying on multiple third-party moving parts improve your supply chain ? A quick glance through the blog post shows, for example: - A reliance on github runners to compile (rather than in-house) - A reliance on a third-party signing service (rather than own CA) I'm probably missing something obvious, but I just don't get how this helps ? In my mind it seems to add extra places for security issues to be introduced ?

orthecreedence 5y ago
> I'm probably missing something obvious No, you're not. The article is actually just braindead enough to suggest that shipping your build and cryptographic signing steps to third-parties is more secure. This isn't just wrong, it's almost criminally devious (and blatantly in service to the ever-growing surveillance state).
- hulitu 5y ago
  You are right except with using the adverb "braindead". I really think they are doing it on purpose. The aim is that the netizen gets a licence for everything he has: car, house, devices. The "Companies" control every device and the life of the netizen . And this security theater is just there that the netizen does not interfere with their data collection.
woodruffw 5y ago
> I'm probably missing something obvious, but I just don't get how this helps ? In my mind it seems to add extra places for security issues to be introduced ? The entire point is integrity and a fully auditable chain of actions. Quoting the post: > Performing these steps guarantees to the consumer that the binary was produced in the trusted builder at a given commit hash attested to in the provenance. They can trust that the information in the provenance was non-forgeable, allowing them to trust the build “recipe” and trace their artifact verifiably back to the source. This provides a broad swath of integrity guarantees that are otherwise extremely difficult to assert for builds that you don't control: you know exactly which machine ran the build, that it wasn't tampered with or silently replaced, and that a particular OpenID identity has signed for the build. You can verify all of these steps publicly without having to keep a trusted keychain, and without having to trust that the signing/publishing parties securely store long-lived signing keys (since everything is ephemeral). The critical thing to note here is that GitHub Actions is just the platform here. The techniques are all open, and sigstore is an open specification with an open implementation. You can do it all in-house if you'd like, or ignore it completely if everything you consume is already in-house.

aftbit 5y ago

>GitHub now supports OIDC in their workflows. Each time a workflow is run, a runner can mint a unique JWT token from GitHub’s OIDC provider. The token contains verifiable information of the workflow identity, including the caller repository, commit hash, trigger, and the current (reuseable) workflow path and reference. Cool, so just emit that token along with your build artifacts. Anyone who trusts Github can then verify that it was signed by them, and there is no need for any of the CA nonsense.

woodruffw 5y ago
> Cool, so just emit that token along with your build artifacts. Anyone who trusts Github can then verify that it was signed by them, and there is no need for any of the CA nonsense. This works if you need to verify JWTs from relatively recent packages, but packaging has a long tail. There's no guarantee that GitHub is still publishing their expired JWK sets a decade after the last valid signature for them has gone out.

webmaven 5y ago

Even assuming that the process being described works as advertised, the result is in no way, shape, or form "tamper-proof". At best, it builds software that is "tamper-evident". A tamper-proof solution would produce software that somehow magically foils all attempts to modify or replace it, which isn't really possible (though some malware is somewhat tamper-resistant in that sense).

amluto 5y ago

This is a surprisingly baroque solution for attesting to a computation being done as specified. Imagine how much simpler it would be if GitHub (or Google or Gitlab or, better yet, multiple vendors) actually supported this style of attestation natively. I would love for app store vendors to put their money where their mouth is: allow reproducibly-build open source (or source-available) apps to be verified by the store and to have a clear badge in the store that indicates that you’re getting the app you think you are.