4 ms·

> The actor gained access because of an error in Codecov’s Docker image creation process that allowed the actor to extract the credential required to modify our

by oefrha 6y ago

> The actor gained access because of an error in Codecov’s Docker image creation process that allowed the actor to extract the credential required to modify our Bash Uploader script. Docker layers hiding secrets is a big problem. Any solution other than education?

raesene9 6y ago
- If you don't care about the layer cache, flattening the image can get rid of things from intermediate layers - If you're using buildkit, there's secret support there https://docs.docker.com/develop/develop-images/build_enhancements/#new-docker-build-secret-information https://docs.docker.com/develop/develop-images/build_enhance...
- oefrha 6y ago
  I know, but these solutions rely on developers being aware and follow best practices in the first place. The problem is anything that's not default on or at least in the getting started guide is going to be missed by a significant chunk of the userbase, so it's important to make security as automatic as possible. GitHub token scanning is nice for limiting secrets exposure in public git repos, for instance.
marcinzm 6y ago
In general your secrets should expose the minimum access needed for the actions to be performed. For a docker build that'd likely be reading rather than writing data. Short term rotated credentials over long term credentials. The goal is defense in depth so that even if a secret is leaked the damage is minimal.
tmsbrg 6y ago
Basically what you want is a Docker image static analysis tool in your CICD that rejects your pull request if it seems to contain a secret. I'm not really sure what would be the best tool for this, but googling got me to https://sysdig.com/blog/20-docker-security-tools/ https://sysdig.com/blog/20-docker-security-tools/ which has enough scanning tools to research. I don't think any will be foolproof though. Maybe you can also do what this guy did, use conftest with some OPA rules to detect what seems like a secret: https://cloudberry.engineering/article/dockerfile-security-best-practices/ https://cloudberry.engineering/article/dockerfile-security-b... If your secrets all follow RFC 8959 ( https://tools.ietf.org/html/rfc8959 https://tools.ietf.org/html/rfc8959 ) then you could make it fool proof by just searching for secret-token: with the tool from the previous blog post. However I've yet to find a place that uses this RFC for all secrets.
brown9-2 6y ago
do they mean that the Docker image at https://github.com/codecov/codecov-action https://github.com/codecov/codecov-action contained a secret allowing someone to modify the script served by https://codecov.io/bash https://codecov.io/bash ?
throwaway823882 6y ago
Clean room builds. Set up an environment with no access to anything (other than a NAT gateway). Copy only the files needed to do the build into the environment. Run the build. At the end of the build, capture the artifacts. Tear it all down at the end. For a Docker container, that might look like: 1. Create an EC2 instance with no IAM permissions, and a user with ~/.ssh/authorized_keys to allow copying in files. 2. Install Docker. 3. Copy all files for your build to the host. Make sure not to copy any secrets (!). 4. Run the 'docker build'. 5. Set up remote registry authentication. 6. 'docker push' the built image to the remote registry. In that environment, the only "secrets" you can bring along into the container came when you copied files to the EC2 instance (which shouldn't have been there, just like secrets shouldn't be in your source code). So then you also make sure the process that preps your files for copying also contains no secrets.
- remram 6y ago
  This: a whole separate machine might be overkill, but try your hardest to make your build not require secrets, and run it where the secrets aren't. This is usually not too hard, as you can use other methods to restrict access to whatever remote files the build needs to access. Even for my local manual builds, I started doing this from fresh git-worktrees that I do no development in, as .env/.git/.log files tend to accumulate in your development copy and are very easy to put in the Docker image by mistake.

by oefrha 6y ago

raesene9 6y ago

- If you don't care about the layer cache, flattening the image can get rid of things from intermediate layers - If you're using buildkit, there's secret support there https://docs.docker.com/develop/develop-images/build_enhancements/#new-docker-build-secret-information https://docs.docker.com/develop/develop-images/build_enhance...

oefrha 6y ago
I know, but these solutions rely on developers being aware and follow best practices in the first place. The problem is anything that's not default on or at least in the getting started guide is going to be missed by a significant chunk of the userbase, so it's important to make security as automatic as possible. GitHub token scanning is nice for limiting secrets exposure in public git repos, for instance.

marcinzm 6y ago

In general your secrets should expose the minimum access needed for the actions to be performed. For a docker build that'd likely be reading rather than writing data. Short term rotated credentials over long term credentials. The goal is defense in depth so that even if a secret is leaked the damage is minimal.

tmsbrg 6y ago

Basically what you want is a Docker image static analysis tool in your CICD that rejects your pull request if it seems to contain a secret. I'm not really sure what would be the best tool for this, but googling got me to https://sysdig.com/blog/20-docker-security-tools/ https://sysdig.com/blog/20-docker-security-tools/ which has enough scanning tools to research. I don't think any will be foolproof though. Maybe you can also do what this guy did, use conftest with some OPA rules to detect what seems like a secret: https://cloudberry.engineering/article/dockerfile-security-best-practices/ https://cloudberry.engineering/article/dockerfile-security-b... If your secrets all follow RFC 8959 ( https://tools.ietf.org/html/rfc8959 https://tools.ietf.org/html/rfc8959 ) then you could make it fool proof by just searching for secret-token: with the tool from the previous blog post. However I've yet to find a place that uses this RFC for all secrets.

brown9-2 6y ago

do they mean that the Docker image at https://github.com/codecov/codecov-action https://github.com/codecov/codecov-action contained a secret allowing someone to modify the script served by https://codecov.io/bash https://codecov.io/bash ?

throwaway823882 6y ago

Clean room builds. Set up an environment with no access to anything (other than a NAT gateway). Copy only the files needed to do the build into the environment. Run the build. At the end of the build, capture the artifacts. Tear it all down at the end. For a Docker container, that might look like: 1. Create an EC2 instance with no IAM permissions, and a user with ~/.ssh/authorized_keys to allow copying in files. 2. Install Docker. 3. Copy all files for your build to the host. Make sure not to copy any secrets (!). 4. Run the 'docker build'. 5. Set up remote registry authentication. 6. 'docker push' the built image to the remote registry. In that environment, the only "secrets" you can bring along into the container came when you copied files to the EC2 instance (which shouldn't have been there, just like secrets shouldn't be in your source code). So then you also make sure the process that preps your files for copying also contains no secrets.

remram 6y ago
This: a whole separate machine might be overkill, but try your hardest to make your build not require secrets, and run it where the secrets aren't. This is usually not too hard, as you can use other methods to restrict access to whatever remote files the build needs to access. Even for my local manual builds, I started doing this from fresh git-worktrees that I do no development in, as .env/.git/.log files tend to accumulate in your development copy and are very easy to put in the Docker image by mistake.