5 ms·

> makes sense if your threat model includes people pushing malicious code to CI, but aren't you more or less done for at that point anyway? If "legitimate" code

by xmodem 5y ago

> makes sense if your threat model includes people pushing malicious code to CI, but aren't you more or less done for at that point anyway? If "legitimate" code can do a certain thing, then malicious code can do too. The answer is very much, 'it depends'. For oen thing, developers can run whatever code in CI before it's benn reviewed. I could just nab the env vars and post them wherever. If there are no sensitive env vars for me to nab and you have enforced code review, then I need a co-conspirator, and my change is probably going to leave a lot more of a paper trail. Another risk is accidental disclosure - I have on at least two occasions accidentally logged sensitive environment variables in our CI environment. Now your threat model is not just a malicious developer pushing code - it's a developer making a mistake, plus anyone with read access to the CI system. I don't know about your org, but at my job, the set of people who have read access to CI is a lot larger than the set who can push code, which is again a lot larger than the set of people who can merge code without a reviewer signing off. > but drawing these boundaries seems like a nightmare for everyone... As someone currently struggling with how to draw them, yup.

nickjj 5y ago
> For one thing, developers can run whatever code in CI before it's been reviewed. Yeah I don't think this gets talked about enough. If you're talking about private repos in an organization then CI often runs on any pull request. That means a developer is able to make CI run in an unreviewed PR. Of course for it to make its way into a protected branch (main, etc.) it'll likely need a code review but nothing is stopping that developer who opened the unreviewed PR to modify the CI yaml file in a commit to make that PR's pipeline do something different. Requiring a team lead or someone to allow every individual PR's pipeline to run (what GitHub does by default in public repos) would add too much friction and not all major git hosts support the idea of locking down the pipelines file by decoupling it from the code repo. Edit: Depending on which CI provider you use, this situation is mostly preventable -- "mostly" in the sense that you can control how much damage can be done. Check out this comment later in this thread: https://news.ycombinator.com/item?id=29967077 https://news.ycombinator.com/item?id=29967077
- heavenlyblue 5y ago
  The solution to that is simple: do not do any tests with production secrets :)
- reflectiv 5y ago
  Each environment should have its own keys for the services it is talking to so ideally this would restrict the scope of damage.
  - nickjj 5y ago
    > Each environment should have its own keys for the services it is talking to so ideally this would restrict the scope of damage. If a developer changes the CI pipeline file to make their PR's code run in `deployment: "production"` instead of `deployment: "test"` doesn't that bypass this? Edit: I'll leave my original question here because I think it's an important one but I answered this myself. It depends on which CI provider you're using but some of them do let you restrict specific deployments from being run only on specific branches or by specific folks (such as repo admins). In the above case if the production deployment was only allowed to run on the main branch and the only way code makes its way into the main branch is after at least 1 person reviewed + merged it (or whatever policy your company wants) then a rogue developer can't edit the pipeline in an unreviewed PR to make something run in production. Also with deployment specific environment variables then a rogue developer is also not able to edit a pipeline file to try and run commands that may affect production such as doing a `terraform apply` or pushing an unreviewed Docker image to your production registry.
    - Nextgrid 5y ago
      CI should only have environment variables needed for testing. For building/deploying to production, it just has to push the code/package/container image, not run it, meaning it has no need for production-level credentials. CI should never ever have access to anything related to production; not just for security but also to prevent potentially bad code being run in tests from trashing production data.
    - 5y ago
marcosdumay 5y ago
Wait. Isn't any CI step done before review expected to be configured for a test environment? I'm failing to understand how that procedure even works. How do you run the tests?
majkinetor 5y ago
Some CI systems like GitLab can protected against this. It allows that only maintainers can run highly privileged pipelines that expose secretes as env vars. Many systems mask or remove passwords (Gitlab, Rundeck etc.)
asimpletune 5y ago
If there's malicious code already excepting with that level of privilege then, yes, you're pwn'd. However, I don't think that's what the main threat is. Probably the main threat is totally legitimate code, that has access to those variables, and now it may do something bad on accident. It's sort of like how I tell my mom, "Even you don't want to know your passwords", when explaining that she should use a password manager. The reason in this case is because she may get fished and redirected to a fake website. If there's a password manager, and she doesn't even know her own password, then it won't recognize the certificate and she's safe. However, if she does know it, then she has the ability to leak it. This is the same with environment variables, and a reason why secret storage is preferred over them. Think of environment variable as your mom knowing her password, and secret storage to her using a password manager. More or less haha.

by xmodem 5y ago

nickjj 5y ago

> For one thing, developers can run whatever code in CI before it's been reviewed. Yeah I don't think this gets talked about enough. If you're talking about private repos in an organization then CI often runs on any pull request. That means a developer is able to make CI run in an unreviewed PR. Of course for it to make its way into a protected branch (main, etc.) it'll likely need a code review but nothing is stopping that developer who opened the unreviewed PR to modify the CI yaml file in a commit to make that PR's pipeline do something different. Requiring a team lead or someone to allow every individual PR's pipeline to run (what GitHub does by default in public repos) would add too much friction and not all major git hosts support the idea of locking down the pipelines file by decoupling it from the code repo. Edit: Depending on which CI provider you use, this situation is mostly preventable -- "mostly" in the sense that you can control how much damage can be done. Check out this comment later in this thread: https://news.ycombinator.com/item?id=29967077 https://news.ycombinator.com/item?id=29967077

heavenlyblue 5y ago
The solution to that is simple: do not do any tests with production secrets :)
reflectiv 5y ago
Each environment should have its own keys for the services it is talking to so ideally this would restrict the scope of damage.
- nickjj 5y ago
  > Each environment should have its own keys for the services it is talking to so ideally this would restrict the scope of damage. If a developer changes the CI pipeline file to make their PR's code run in `deployment: "production"` instead of `deployment: "test"` doesn't that bypass this? Edit: I'll leave my original question here because I think it's an important one but I answered this myself. It depends on which CI provider you're using but some of them do let you restrict specific deployments from being run only on specific branches or by specific folks (such as repo admins). In the above case if the production deployment was only allowed to run on the main branch and the only way code makes its way into the main branch is after at least 1 person reviewed + merged it (or whatever policy your company wants) then a rogue developer can't edit the pipeline in an unreviewed PR to make something run in production. Also with deployment specific environment variables then a rogue developer is also not able to edit a pipeline file to try and run commands that may affect production such as doing a `terraform apply` or pushing an unreviewed Docker image to your production registry.
  - Nextgrid 5y ago
    CI should only have environment variables needed for testing. For building/deploying to production, it just has to push the code/package/container image, not run it, meaning it has no need for production-level credentials. CI should never ever have access to anything related to production; not just for security but also to prevent potentially bad code being run in tests from trashing production data.
  - 5y ago

marcosdumay 5y ago

Wait. Isn't any CI step done before review expected to be configured for a test environment? I'm failing to understand how that procedure even works. How do you run the tests?

majkinetor 5y ago

Some CI systems like GitLab can protected against this. It allows that only maintainers can run highly privileged pipelines that expose secretes as env vars. Many systems mask or remove passwords (Gitlab, Rundeck etc.)

asimpletune 5y ago

If there's malicious code already excepting with that level of privilege then, yes, you're pwn'd. However, I don't think that's what the main threat is. Probably the main threat is totally legitimate code, that has access to those variables, and now it may do something bad on accident. It's sort of like how I tell my mom, "Even you don't want to know your passwords", when explaining that she should use a password manager. The reason in this case is because she may get fished and redirected to a fake website. If there's a password manager, and she doesn't even know her own password, then it won't recognize the certificate and she's safe. However, if she does know it, then she has the ability to leak it. This is the same with environment variables, and a reason why secret storage is preferred over them. Think of environment variable as your mom knowing her password, and secret storage to her using a password manager. More or less haha.