4 ms·
> The researchers identified four fundamental security properties that CI/CD systems need: admittance control, execution control, code control, and access to se
by amluto 11mo ago
> The researchers identified four fundamental security properties that CI/CD systems need: admittance control, execution control, code control, and access to secrets.
Why do CI/CD systems need access to secrets? I would argue need access to APIs and they need privileges to perform specific API calls. But there is absolutely nothing about calling an API that fundamentally requires that the caller know a secret.
I would argue that a good CI/CD system should not support secrets as a first-class object at all. Instead steps may have privileges assigned. At most there should be an adapter, secure enclave style, that may hold a secret and give CI/CD steps the ability to do something with that secret, to be used for APIs that don’t support OIDC or some other mechanism to avoid secrets entirely.
- cyberax 11mo agoYou might want (or _need_) to sign your binary, for example. Or you might want to trigger a deployment. Github actually is doing something right here. You can set it up as a trusted identity provider in AWS, and then use Github to assume a role in your AWS account. And from there, you can get access to credentials stored in Secret Manager or SSM.
- DuncanCoffee 11mo agoYeah I sign my project APKs so people can install them from the action's artefact - name: Retrieve keystore for apk signing env: KEYSTORE: ${{ secrets.KEYSTORE }} run: echo "$KEYSTORE" | base64 --decode > /home/runner/work/keystore.pfk
- amluto 11mo agoExactly. This workflow step takes a rather important secret and sticks it on a VM where any insufficiently sandboxed step before or after it can exfiltrate it. GitHub should instead let you store that key as a different type of secret such that a specific workflow step can sign with it. Then a compromised runner VM could possibly sign something that shouldn’t be signed but could not exfiltrate it. Even better would be to be able to have a policy that the only thing that can be signed is something with a version that matches the immutable release that’s being built.
- jdeastwood 11mo agoYes, their oidc setup was probably their last good feature back when they were actually delivering features back in 2020ish. Everyone else copied it within a few months though.
- sofixa 11mo agoWho is everyone? As with pretty much every GitHub feature, GitLab had them beat by a few years.
- jdeastwood 11mo agoMaybe in some other form, but the current style of injecting an oidc token was definitely in github actions first. Here is the gitlab issue tracking the final bit of it's implementation directly mentioning github: https://gitlab.com/gitlab-org/gitlab/-/issues/356986 https://gitlab.com/gitlab-org/gitlab/-/issues/356986
- sofixa 11mo agoThis is the more flexible and secure (specific aud) replacement for CI_JOB_JWT which has been there since at least 2017, if not before. Functionally it was exactly the same, a JWT token per pipeline allowing you to authenticate to third parties that support OIDC/JWT Auth.
- PunchyHamster 11mo ago> I would argue that a good CI/CD system should not support secrets as a first-class object at all. Instead steps may have privileges assigned. At most there should be an adapter, secure enclave style, that may hold a secret and give CI/CD steps the ability to do something with that secret, to be used for APIs that don’t support OIDC or some other mechanism to avoid secrets entirely. CI/CD does not exist in the vacuum. If you had CI/CD entirely integrated with the rest of the infrastructure it might be possible to do say an app deploy without passing creds to user code (say have the platform APIs that it can call to do the deployment instead of typical "install the client, get the creds, run k8s/ssh/whatever else needed for deploy"). But that's a high level of integration that's very environment specific, and without all that many positives (so what you don't need creds, you still have permission to do a lot of mess if it gets hijacked), and a lot, lot more code to write vs "run a container and pass it some env vars" that had become a standard
- Kinrany 11mo agoCI shouldn't do deployments, deployment pipelines should run separately when a new release passes CI Of course the general purpose task runner that both run on does need to support secrets
- Kinrany 11mo agoHmm, I have long assumed that a perfectly executed CI/CD setup would be based on a generic task runner... But maybe not? Only the CI part needs to build; it needs little else and it's the only part of a coherent setup that needs to build.
- arccy 11mo agoyou conveniently ignored the "CD" part of CI/CD
- Kinrany 11mo agoAs mentioned by sibling comments, these two parts are very different. GHA and alternatives are primarily for CI; many people choose to use it solely for CI. For these reasons I did ignore CD.
- zahlman 11mo ago> But there is absolutely nothing about calling an API that fundamentally requires that the caller know a secret. There is if you pay for API access, surely?
- hinkley 11mo agoCI is arguable, but how do you intend to do deployments with no secrets?
- themafia 11mo agoAWS is great for this. IAM policies can allow IP Addresses or more safely just named EC2 instances. Our deploy server requires nothing.
- nijave 11mo agoCircleCI and I believe GHA support injecting signed JWTs you can use to bootstrap identity be it an IAM role or some other platform where you can trust an OIDC issuer
- gcr 11mo agoThe secret is held by the metadata server that the CI instance has access to Or: the deployment service knows the identity of the instance, so its secret is its private key Or, how PyPI does it: the deployment service coordinates with the trusted CI/CD service to learn the identity of the machine (like its IP address, or a trusted assertion of which repository it’s running on), so the secret is handled in however that out-of-band verification step happens. (PyPI communicates with Github Actions about which pipeline from which repository is doing the deployment, for example) It’s still just secrets all the way down
- barrkel 11mo agoHow do you e.g. validate that a database product works with all the different cloud databases? Every time you change up SQL generation you're going to want to make sure the SQL parses and evaluates as expected on all supported platforms. Those tests will need creds to access third party database endpoints.
- qznc 11mo agoWe use proprietary tools (QNX compiler, Coverity static analysis, ...) and those require access to a license server which requires some secret. I don't really understand what you mean by "secure enclave style"? How would that be different?
- amluto 11mo agoWith a secure enclave or an HSM, there's a secret, but the users do not have access to the secret. So, if you have a workflow that needs to, say, sign with a given private key, you would get an API that signs for you. If you need to open a TLS connection with a client certificate, you get a proxy that authenticates for you. I suppose I would make an exception for license keys. Those have minimal blast radii if they leak.
- gcr 11mo agoAnd how is that exposed to the CI/CD? An environment variable? Some bespoke tool that the CI runs to read the secret from the Secure Enclave? Your approach boils down to “lets give each step its own access to its own hardware-protected secrets, but developers shouldn’t otherwise have access” Which is a great way to “support secrets,” just like the article says.
- gcr 11mo ago“Good CI systems shouldn’t support secrets, at most there should be [the most complicated secret support ever]” Let’s just call it secret support. I agree with your suggestion that capabilities-based APIs are better, but CI/CD needs to meet customers where they’re at currently, not where they should be. Most customers need secrets.
- nijave 11mo agoWhile good in theory, in practice secrets are used to validate those privileges have been assigned. Even in schemes like metadata servers, you still use a secret. Pedantically I'd say maybe it's more fair to say they shouldn't have access to long lived secrets and should only use short lived values. The "I" stands for Integration so it's inevitable CI needs to talk to multiple things--at the very least a git repo which most cases requires a secret to pull.
- LtWorf 11mo agoBecause for some reason they use the same system to do releases and sign them and publish them.
- deleted 11mo ago[deleted]
- woodruffw 11mo ago> I would argue that a good CI/CD system should not support secrets as a first-class object at all. Instead steps may have privileges assigned. At most there should be an adapter, secure enclave style, that may hold a secret and give CI/CD steps the ability to do something with that secret, to be used for APIs that don’t support OIDC or some other mechanism to avoid secrets entirely. This all seems right, but the reality is that people will put secrets into CI/CD, and so the platform should provide an at least passably secure mechanism for them. (A key example being open source: people want to publish from CI, and they’re not going to set up additional infrastructure when the point of using third-party CI is to avoid that setup.)
- amluto 11mo agoFine. Then let people set up a little WASM app that gets access to the secret and has an API callable by the workflow. And make that app be configured as part of the secret’s configuration, not as a file in the repository.
- lionkor 11mo agoYou're missing that the D in CI/CD means deployment; be that packaging on pushing tags and publishing to a registry, or building images, or packaging github releases.
- duped 11mo ago> Why do CI/CD systems need access to secrets? Because you need to be able to sign/notarize with private keys and deploy to cloud environments. Both of these require secrets known to the runner.