13 ms·

Real-world stories of how we’ve compromised CI/CD pipelines

265 points by usrme 5y ago

cerved 5y ago
> a hardcoded git command with a credential was revealed cries in security
- i_like_waiting 5y ago
  reminds me of tons docker tutorials, where all of them are doing default password in plaintext in docker-compose file
  - rietta 5y ago
    I put devonly: as part of every placeholder secret in docker-compose.yml or similar config that is committed to Git. The goal is a developer who has just cloned the repo should be able to run the setup script and have the whole system running with random seed data without futzing with copying secrets from coworkers.
    - nickjj 5y ago
      > I put devonly: as part of every placeholder secret in docker-compose.yml or similar config that is committed to Git. The goal is a developer who has just cloned the repo should be able to run the setup script and have the whole system running with random seed data without futzing with copying secrets from coworkers. This problem is solvable without hard coding env variables into your docker-compose.yml file. You can commit an .env.example file to version control which has non-secret defaults set so that all a developer has to do is run `cp .env.example .env` before `docker-compose up --build` and they're good to go. There's examples of this in all of my Docker example apps for Flask, Rails, Django, Phoenix, Node and Play at: https://github.com/nickjj?tab=repositories&q=docker-*-example&type=&language=&sort= https://github.com/nickjj?tab=repositories&q=docker-*-exampl... It's nice because it also means the same docker-compose.yml file can be used in dev vs prod. The only thing that changes are a few environment variables.
    - inetknght 5y ago
      > *I put devonly: as part of every placeholder secret in docker-compose.yml or similar config that is committed to Git.& I put it `insecure`. I think it makes it clear that the password, and file, aren't secure by default and should be treated as such.
    - staticassertion 5y ago
      With buildkit Docker now has support for secrets natively with `--secret`. This mounts a file that will only be exposed during build.
dlor 5y ago
This is a great resource. I'd love to see more reports like it published. CI/CD pipelines often run with highly elevated permissions (access to source code, artifact repositories, and production environments), but they are traditionally neglected.
- kevin_nisbet 5y ago
  I suspect this is also an under considered area even in organizations with lots of attention to security. So would be good to also get more mindshare, as after we discovered some of our own CI/CD related vulnerabilities[1], it feels like most approaches we looked at had similar problems, and it took alot of research to find the rare solution that we could be confident in. [1] - https://goteleport.com/blog/hack-via-pull-request/ https://goteleport.com/blog/hack-via-pull-request/
  - 0xbadcafebee 5y ago
    Been there, done that, bought the t-shirt.... We had this "deploy" Jenkins box set up with limited access for devs, because it had assume-role privs to an IAM role to manage AWS infra with Terraform. The devs run their tests on a different Jenkins box, and when they pass, they upload artifacts to a repo and trigger this "deploy" Jenkins box to promote the new build to prod. The devs can do their own CI, but CD is on a box they don't have access to, hence less chance for accidental credential leakage. Me being Mr. Devops-play-nice-with-the-devs, I let them issue PRs against the CD box's repo. Commits to PRs get run on the deploy Jenkins in a stage environment to validate the changes. This one dev wanted to change something in AWS. But for whatever reason, they didn't ask me (maybe because they knew I'd say no, or at least ask them about it?). So instead the dev opens a PR against the CD jobs, proposing some syntax change. Then the dev modifies a script which was being included as part of the CD jobs, and makes the script download some binaries and make AWS API calls (I found out via CloudTrail). Once they've made the calls, they rewrite Git history to remove the AWS API commits and force-push to the PR branch, erasing evidence that the code was ever issued. Then close the PR with "need to refactor". In the morning I'm looking through my e-mail, and see all these GitHub commits with code that looks like it's doing something in AWS... and I go look at the PR, and the code in my e-mails isn't anyware in any of the commits. He actually tried to cover it up. And I would never have known about any of this if I hadn't enabled 'watching' on all commits to the repo. Who'd have thought e-mail would be the best append-only security log?
lox 5y ago
We’ve been using Sysbox (https://github.com/nestybox/sysbox https://github.com/nestybox/sysbox) for our Buildkite based CI/CD setup, allows docker-in-docker without privileged containers. Paired with careful IAM/STS design we’ve ended up with isolated job containers with their own IAM roles limited to least-privilege.
- xmodem 5y ago
  Could you elaborate a bit more how you get the containers into their own IAM roles?
  - lox 5y ago
    Yup, we have a sidecar process/container that runs for each job and assumes an AWS IAM Role for that specific pipeline (with constraints like whether it’s an approved PR as well). The credentials are provided to the job container via a volume mount. This allows us to have shared agents with very granular roles per-pipeline and job.
  - nijave 5y ago
    Not sure if this applies to the parent, but one way this Buildkite Queues map pipelines to agents. Agents can be assigned IAM roles. If you want a certain build to run as an IAM role, you give it a queue where the agents have that role. For AWS, Buildkite has as a Cloud Formation stack that sets up auto scaling groups and some other resources for your agents to run.
    - xmodem 5y ago
      Most CI systems will have some way of assigning builds to groups of agents. But it would in some cases be useful to grant different privileges to different containers running on the same agent, which is what I understood OP to have.
  - orf 5y ago
    AWS has IAM service accounts for containers. Comes for free with EKS, not sure how you’d do it without EKS. Basically it adds a signed web identity file into the container which can be used to assume roles.
mvdwoord 5y ago
The company I currently do contract work for, decided it would be best to have one large team in Azure DevOps and subdivide all teams in repositories etc with prefixes and homegrown "Governer" scripts, which are enforced in all pipelines. Global find on some terms like "key", "password" etc were great fun. It really showed most people, our team included, struggled with getting the pipeline to work at all. Let alone doing it in a secure manner. This is a 50k+ employee financial institute. I am honestly surprised these kind of attacks are not much more widespread.
MauranKilom 5y ago
Interesting to learn that credentials in environment variables are frowned upon. I mean, makes sense if your threat model includes people pushing malicious code to CI, but aren't you more or less done for at that point anyway? If "legitimate" code can do a certain thing, then malicious code can do too. I guess you'll want to limit the blast radius, but drawing these boundaries seems like a nightmare for everyone...
- imachine1980_ 5y ago
  say from everbody to all sec teams
- xmodem 5y ago
  > makes sense if your threat model includes people pushing malicious code to CI, but aren't you more or less done for at that point anyway? If "legitimate" code can do a certain thing, then malicious code can do too. The answer is very much, 'it depends'. For oen thing, developers can run whatever code in CI before it's benn reviewed. I could just nab the env vars and post them wherever. If there are no sensitive env vars for me to nab and you have enforced code review, then I need a co-conspirator, and my change is probably going to leave a lot more of a paper trail. Another risk is accidental disclosure - I have on at least two occasions accidentally logged sensitive environment variables in our CI environment. Now your threat model is not just a malicious developer pushing code - it's a developer making a mistake, plus anyone with read access to the CI system. I don't know about your org, but at my job, the set of people who have read access to CI is a lot larger than the set who can push code, which is again a lot larger than the set of people who can merge code without a reviewer signing off. > but drawing these boundaries seems like a nightmare for everyone... As someone currently struggling with how to draw them, yup.
  - nickjj 5y ago
    > For one thing, developers can run whatever code in CI before it's been reviewed. Yeah I don't think this gets talked about enough. If you're talking about private repos in an organization then CI often runs on any pull request. That means a developer is able to make CI run in an unreviewed PR. Of course for it to make its way into a protected branch (main, etc.) it'll likely need a code review but nothing is stopping that developer who opened the unreviewed PR to modify the CI yaml file in a commit to make that PR's pipeline do something different. Requiring a team lead or someone to allow every individual PR's pipeline to run (what GitHub does by default in public repos) would add too much friction and not all major git hosts support the idea of locking down the pipelines file by decoupling it from the code repo. Edit: Depending on which CI provider you use, this situation is mostly preventable -- "mostly" in the sense that you can control how much damage can be done. Check out this comment later in this thread: https://news.ycombinator.com/item?id=29967077 https://news.ycombinator.com/item?id=29967077
Lucasoato 5y ago
Is it just my impression or security in Jenkins seems much more challenging and more time-consuming than in GitLab? This post gives many examples where GitLab was attacked, so of course bad practices like privileged containers can lead to the compromise of a server independently by the technology used, but from my experience with Jenkins, I've seen using passwords in plaintext so many times, even in big companies.
- ramoz 5y ago
  I don’t really like either. Both have traditionally been bad & related to on-prem legacy workloads. Building for SVN apps or teams new to git. It’s usually a mess.
  - nathanlied 5y ago
    As someone adjacently interested in the field: care to elaborate on what systems you do like? It's always interesting to get new perspectives.
    - staticassertion 5y ago
      We've been happy with buildkite and hashicorp vault. One nice feature we've leveraged in our CI is that vault lets us revoke tokens after use, so we have very short lived tokens and they're made that much shorter by having the jobs clean up after themselves.
    - ramoz 5y ago
      going cloud native (AWS/GCP/Asure) & using their build tools makes things simple for things like container management and integrated development. GitHub because it’s better UX. It’s even quite simple to setup good automation around a codebase. Platform teams are using Argo, dev teams not really doing too much ci/cd which I like. & to be honest CI/CD requires continuous investment as things continuously change. Not that it isn’t necessary… but in an enterprise environment you I’ve seen teams become more successful on their own rather than trying to fulfill any “reciprocity” bs.
- 5y ago
INTPenis 5y ago
Many of these points are about running pipelines in privileged containers. Something I actually took extra time to resolve for my team. That's when I discovered kaniko first, and shortly after podman/buildah. After that podman and buildah have gotten a lot of great reviews from people so I think they're awesome. For an old time Unix sysadmin it just doesn't make sense to run something as root unless you absolutely have to. Which also makes the client excuse in the article so strange, they had to run the container privileged to run static code analysis. wtf. Doesn't that just mean they run a tool against a binary artefact from a previous job? I fail to see how that requires privileges.
- wdb 5y ago
  If you start fresh would you use Kaniko or Podman? Trying to find out the best way to avoid docker —privileged on custom Gitlab runner
  - INTPenis 5y ago
    Podman all the way. I abandoned kaniko as soon as I found podman. That also eliminates the risk of accessing the host docker.
    - wdb 5y ago
      Cool, going to try it out. Wondering if the cache layering still works with it
rawgabbit 5y ago
You would think by now we would have better credential methods. I still see username and passwords for system credentials. I see tokens created by three legged auths. I don’t get how that is an improvement. The problem is that most deployed code doesn’t have just one credential but a dozen. Multiply that with several environments and you get security fatigue and apathy.
mdoms 5y ago
> The credentials gave the NCC Group consultant access as a limited user to the Jenkins Master web login UI which was only accessible internally and not from the Internet. After a couple of clicks and looking around in the cluster they were able to switch to an administrator account. These kinds of statements are giving major "draw the rest of the owl" vibes. https://i.kym-cdn.com/photos/images/newsfeed/000/572/078/d6d.jpg https://i.kym-cdn.com/photos/images/newsfeed/000/572/078/d6d...
staticassertion 5y ago
Thank you for writing this up. Some thoughts: 1. Hardcoded credentials are a plague. You should consider tagging all of your secrets so that they're easier to scan for. Github automatically scans for secrets, which is great. 2. Jenkins is particularly bad for security. I've seen it owned a million and one times. 3. Containers are overused as a security boundary and footguns like `--privileged` completely eliminate any boundary. 4. Environment variables are a dangerous place to store secrets - they're global to the process and therefor easy to leak. I've thought about this a lot lately, especially after log4j. I think one pattern that may help is clearing the variables after you've loaded them into memory. Another I've considered is encrypting the variables. A lot of the time what you have is something like this: Secret Store -> Control Plane Agent -> Container -> Process Where secrets flow from left to right. The control plane agent and container have full access to the credentials and they're "plaintext" in the Process's environment. In theory you should be able to pin the secrets to that process with a key. During your CD phase you would embed a private key into the process's binary (or a file on the container) and then tell your Secret Manager to use the associated public key to transmit the secrets. The process could decrypt those secrets with its private key but they're E2E encrypted across any hops between the Secret Store and Process and they can't be leaked without explicitly decrypting them first.
- teddyh 5y ago
  > Environment variables are a dangerous place to store secrets - they're global to the process and therefor easy to leak. The two real problems with environment variables are: 1. Environment variables are traditionally readable by any other process in the system. There are settings you can do on modern kernels to turn this off, but how do you know that you will always run on such a system? 2. Environment variables are inherited to all subprocesses by default, unless you either unset them after you fork() (but before you exec()), or if you take special care to use execve() (or similar) function to provide your own custom-made environment for the new process.
  - otterley 5y ago
    A foreign process’s environment variables are only readable if the current UID is root or is the same as the foreign process’s ID. As user joe I can’t see user andrea’s process’s envvars.
thomasmarcelis 5y ago
>>In our final scenario, the NCC Group consultant got booked on a scenario-based assessment: >>“Pretend you have compromised a developer’s laptop.” Most companies will fail right here. Especially outside of the tech world security hygiene with developer's laptops is very bad from what I have seen.
movedx 5y ago
This is exactly why I both love and hate CI/CD. Ultimately most CI/CD setups are basically systems administrators with privileged access to everything, network connected and running 24/7. It's pretty dangerous stuff. I don't have an answer though, expect maybe to keep the CI and CD in separate, isolated instances that require manual intervention to bridge the gap on a case by case basis. That doesn't scale very well though.
- hinkley 5y ago
  I think in general we put too much logic into our CI/CD configurations. There is an argument to be made for a minimalist CI/CD implementation that can handle task scheduling and dependencies, understands how to fetch and tag version control, count version numbers and not much else. Even extracting test result summaries, while handy, maybe should be handled another way. For many of us, if CI is down you can't deploy anything to production, not even roll back to a previous build. Everything but the credentials should be under version control, and the right people should be able to fire off a one-liner from a runbook that has two to four sanity checked arguments in order to trigger a deployment.
  - jeffjeff2 5y ago
    A place I was at recently used makefiles in each project containing entries for "build", "deploy", "test" etc. All the CI did was use shared generic jobs to call the relevant makefile command. These could just as easily be run from a developers machine with the right credentials.
    - john-tells-all 5y ago
      I strongly recommend this. Put all the CICD machinery inside the dev repos, letting all the devs see, understand, and modify the pipeline. With carefully chosen creds, Devs can run the CICD things directly, thus making feedback loops much faster in certain circumstances. Maybe don't let them delete databases, or run expensive VMs, but other than that go for it. In my experience if the pipeline is even slightly different Devs will treat it as a foreign object and will whine about it not working the way they want... they don't feel like they can change the pipeline, which is odd. Dude, it's code, just change it, I'm happy to approve your PR and/or help you make a change.
jiggawatts 5y ago
A weakness of modern secret management is that it isn’t. A secret value ought to be very carefully guarded even from the host machine itself. .NET for example has SecureString, which is a good start — it can’t be accidentally printed or serialised insecurely. If it is serialised, then it is automatically encrypted by the host OS data protection API. Windows even has TPM-hosted certificates! They’re essentially a smart card plugged into the motherboard. A running app can use a TPM credential to sign requests but it can’t read or copy it. These advancements are just completely ignored in the UNIX world, where everything is blindly copied into easily accessible locations in plain text…
- pxx 5y ago
  Except afaict SecureString doesn't reliably do that and shouldn't be used. https://github.com/dotnet/platform-compat/blob/master/docs/DE0001.md https://github.com/dotnet/platform-compat/blob/master/docs/D...
  - jiggawatts 5y ago
    “It’s not perfectly secure so use a totally insecure alternative instead” seems like terrible advice.
    - pxx 5y ago
      No it's "don't use this thing which doesn't do what it says on the tin and is therefore a foot gun." Something that is obviously insecure will be treated with more caution / put on the correct side of the authorization boundary compared to something that claims to be.
      - jiggawatts 5y ago
        > "obviously insecure will be treated with more caution" What planet are you from and can I go there? SecureString provides one layer in the "defence in depth". If someone accidentally logs it, then it won't leak. If it is used as a script parameter, then the prompt will use password input characters automatically. Seriously, I want you to try this PowerShell snippet right now. (You can even run it on Linux too with pwsh v7, so no excuses!): PARAM( [parameter(Mandatory=$true)] [securestring]$Secret ) Write-Warning "Oops I didn't mean to log this: $Secret" $Secret | ConvertTo-Json -Compress $Secret | Export-Clixml 'accidental serialisation.xml' Get-Content 'accidental serialisation.xml' Run the the script and see what happens. There's even low-level protection built-in, such zeroing out the memory when it is garbage collected (unlike normal strings). Why is this a bad thing!? Do you like secrets leaking everywhere unless everyone is always hyper vigilant? Or do you prefer to roll your own half-baked secret storage type that nothing else is compatible with? PS: The page with that advice is based on an "archived" read-only repo with a bunch of open issues of people befuddled as to why this bad, BAD, BAD advice is being published there.
tialaramex 5y ago
A recurring theme is that they obtain secret credentials from a service which needs to verify credentials, and then turn around and use those to impersonate the entity providing those credentials. For example getting Jenkins to run some Groovy discovers credentials Jenkins uses to verify who is accessing it, and then you can just use those credentials yourself. To fix this - almost anywhere - stop using shared secrets. Every time you visit a (HTTPS) web site, you are provided with the credentials to verify its identity. But, you don't gain the ability to impersonate the site because they're not secret credentials, they're public. You can and should use this in a few places in typical CI / CD type infrastructure today, and we should be encouraging other services to enable it too ASAP. In a few places they mention MFA. Again, most MFA involves secrets, for example TOTP Relying Parties need to know what code you should be typing in, so, they need the seed from which to generate that code, and attackers can steal that seed. WebAuthn doesn't involve secrets, so, attackers who steal WebAuthn credentials don't achieve anything. Unfortunately chances are you enabled one or more vulnerable credential types "just in case"...

265 points by usrme 5y ago

cerved 5y ago

> a hardcoded git command with a credential was revealed cries in security

i_like_waiting 5y ago
reminds me of tons docker tutorials, where all of them are doing default password in plaintext in docker-compose file
- rietta 5y ago
  I put devonly: as part of every placeholder secret in docker-compose.yml or similar config that is committed to Git. The goal is a developer who has just cloned the repo should be able to run the setup script and have the whole system running with random seed data without futzing with copying secrets from coworkers.
  - nickjj 5y ago
    > I put devonly: as part of every placeholder secret in docker-compose.yml or similar config that is committed to Git. The goal is a developer who has just cloned the repo should be able to run the setup script and have the whole system running with random seed data without futzing with copying secrets from coworkers. This problem is solvable without hard coding env variables into your docker-compose.yml file. You can commit an .env.example file to version control which has non-secret defaults set so that all a developer has to do is run `cp .env.example .env` before `docker-compose up --build` and they're good to go. There's examples of this in all of my Docker example apps for Flask, Rails, Django, Phoenix, Node and Play at: https://github.com/nickjj?tab=repositories&q=docker-*-example&type=&language=&sort= https://github.com/nickjj?tab=repositories&q=docker-*-exampl... It's nice because it also means the same docker-compose.yml file can be used in dev vs prod. The only thing that changes are a few environment variables.
  - inetknght 5y ago
    > *I put devonly: as part of every placeholder secret in docker-compose.yml or similar config that is committed to Git.& I put it `insecure`. I think it makes it clear that the password, and file, aren't secure by default and should be treated as such.
  - staticassertion 5y ago
    With buildkit Docker now has support for secrets natively with `--secret`. This mounts a file that will only be exposed during build.

dlor 5y ago

This is a great resource. I'd love to see more reports like it published. CI/CD pipelines often run with highly elevated permissions (access to source code, artifact repositories, and production environments), but they are traditionally neglected.

kevin_nisbet 5y ago
I suspect this is also an under considered area even in organizations with lots of attention to security. So would be good to also get more mindshare, as after we discovered some of our own CI/CD related vulnerabilities[1], it feels like most approaches we looked at had similar problems, and it took alot of research to find the rare solution that we could be confident in. [1] - https://goteleport.com/blog/hack-via-pull-request/ https://goteleport.com/blog/hack-via-pull-request/
- 0xbadcafebee 5y ago
  Been there, done that, bought the t-shirt.... We had this "deploy" Jenkins box set up with limited access for devs, because it had assume-role privs to an IAM role to manage AWS infra with Terraform. The devs run their tests on a different Jenkins box, and when they pass, they upload artifacts to a repo and trigger this "deploy" Jenkins box to promote the new build to prod. The devs can do their own CI, but CD is on a box they don't have access to, hence less chance for accidental credential leakage. Me being Mr. Devops-play-nice-with-the-devs, I let them issue PRs against the CD box's repo. Commits to PRs get run on the deploy Jenkins in a stage environment to validate the changes. This one dev wanted to change something in AWS. But for whatever reason, they didn't ask me (maybe because they knew I'd say no, or at least ask them about it?). So instead the dev opens a PR against the CD jobs, proposing some syntax change. Then the dev modifies a script which was being included as part of the CD jobs, and makes the script download some binaries and make AWS API calls (I found out via CloudTrail). Once they've made the calls, they rewrite Git history to remove the AWS API commits and force-push to the PR branch, erasing evidence that the code was ever issued. Then close the PR with "need to refactor". In the morning I'm looking through my e-mail, and see all these GitHub commits with code that looks like it's doing something in AWS... and I go look at the PR, and the code in my e-mails isn't anyware in any of the commits. He actually tried to cover it up. And I would never have known about any of this if I hadn't enabled 'watching' on all commits to the repo. Who'd have thought e-mail would be the best append-only security log?

lox 5y ago

We’ve been using Sysbox (https://github.com/nestybox/sysbox https://github.com/nestybox/sysbox) for our Buildkite based CI/CD setup, allows docker-in-docker without privileged containers. Paired with careful IAM/STS design we’ve ended up with isolated job containers with their own IAM roles limited to least-privilege.

xmodem 5y ago
Could you elaborate a bit more how you get the containers into their own IAM roles?
- lox 5y ago
  Yup, we have a sidecar process/container that runs for each job and assumes an AWS IAM Role for that specific pipeline (with constraints like whether it’s an approved PR as well). The credentials are provided to the job container via a volume mount. This allows us to have shared agents with very granular roles per-pipeline and job.
- nijave 5y ago
  Not sure if this applies to the parent, but one way this Buildkite Queues map pipelines to agents. Agents can be assigned IAM roles. If you want a certain build to run as an IAM role, you give it a queue where the agents have that role. For AWS, Buildkite has as a Cloud Formation stack that sets up auto scaling groups and some other resources for your agents to run.
  - xmodem 5y ago
    Most CI systems will have some way of assigning builds to groups of agents. But it would in some cases be useful to grant different privileges to different containers running on the same agent, which is what I understood OP to have.
- orf 5y ago
  AWS has IAM service accounts for containers. Comes for free with EKS, not sure how you’d do it without EKS. Basically it adds a signed web identity file into the container which can be used to assume roles.

mvdwoord 5y ago

The company I currently do contract work for, decided it would be best to have one large team in Azure DevOps and subdivide all teams in repositories etc with prefixes and homegrown "Governer" scripts, which are enforced in all pipelines. Global find on some terms like "key", "password" etc were great fun. It really showed most people, our team included, struggled with getting the pipeline to work at all. Let alone doing it in a secure manner. This is a 50k+ employee financial institute. I am honestly surprised these kind of attacks are not much more widespread.

MauranKilom 5y ago

Interesting to learn that credentials in environment variables are frowned upon. I mean, makes sense if your threat model includes people pushing malicious code to CI, but aren't you more or less done for at that point anyway? If "legitimate" code can do a certain thing, then malicious code can do too. I guess you'll want to limit the blast radius, but drawing these boundaries seems like a nightmare for everyone...

imachine1980_ 5y ago
say from everbody to all sec teams
xmodem 5y ago
> makes sense if your threat model includes people pushing malicious code to CI, but aren't you more or less done for at that point anyway? If "legitimate" code can do a certain thing, then malicious code can do too. The answer is very much, 'it depends'. For oen thing, developers can run whatever code in CI before it's benn reviewed. I could just nab the env vars and post them wherever. If there are no sensitive env vars for me to nab and you have enforced code review, then I need a co-conspirator, and my change is probably going to leave a lot more of a paper trail. Another risk is accidental disclosure - I have on at least two occasions accidentally logged sensitive environment variables in our CI environment. Now your threat model is not just a malicious developer pushing code - it's a developer making a mistake, plus anyone with read access to the CI system. I don't know about your org, but at my job, the set of people who have read access to CI is a lot larger than the set who can push code, which is again a lot larger than the set of people who can merge code without a reviewer signing off. > but drawing these boundaries seems like a nightmare for everyone... As someone currently struggling with how to draw them, yup.
- nickjj 5y ago
  > For one thing, developers can run whatever code in CI before it's been reviewed. Yeah I don't think this gets talked about enough. If you're talking about private repos in an organization then CI often runs on any pull request. That means a developer is able to make CI run in an unreviewed PR. Of course for it to make its way into a protected branch (main, etc.) it'll likely need a code review but nothing is stopping that developer who opened the unreviewed PR to modify the CI yaml file in a commit to make that PR's pipeline do something different. Requiring a team lead or someone to allow every individual PR's pipeline to run (what GitHub does by default in public repos) would add too much friction and not all major git hosts support the idea of locking down the pipelines file by decoupling it from the code repo. Edit: Depending on which CI provider you use, this situation is mostly preventable -- "mostly" in the sense that you can control how much damage can be done. Check out this comment later in this thread: https://news.ycombinator.com/item?id=29967077 https://news.ycombinator.com/item?id=29967077

Lucasoato 5y ago

Is it just my impression or security in Jenkins seems much more challenging and more time-consuming than in GitLab? This post gives many examples where GitLab was attacked, so of course bad practices like privileged containers can lead to the compromise of a server independently by the technology used, but from my experience with Jenkins, I've seen using passwords in plaintext so many times, even in big companies.

ramoz 5y ago
I don’t really like either. Both have traditionally been bad & related to on-prem legacy workloads. Building for SVN apps or teams new to git. It’s usually a mess.
- nathanlied 5y ago
  As someone adjacently interested in the field: care to elaborate on what systems you do like? It's always interesting to get new perspectives.
  - staticassertion 5y ago
    We've been happy with buildkite and hashicorp vault. One nice feature we've leveraged in our CI is that vault lets us revoke tokens after use, so we have very short lived tokens and they're made that much shorter by having the jobs clean up after themselves.
  - ramoz 5y ago
    going cloud native (AWS/GCP/Asure) & using their build tools makes things simple for things like container management and integrated development. GitHub because it’s better UX. It’s even quite simple to setup good automation around a codebase. Platform teams are using Argo, dev teams not really doing too much ci/cd which I like. & to be honest CI/CD requires continuous investment as things continuously change. Not that it isn’t necessary… but in an enterprise environment you I’ve seen teams become more successful on their own rather than trying to fulfill any “reciprocity” bs.
5y ago

INTPenis 5y ago

Many of these points are about running pipelines in privileged containers. Something I actually took extra time to resolve for my team. That's when I discovered kaniko first, and shortly after podman/buildah. After that podman and buildah have gotten a lot of great reviews from people so I think they're awesome. For an old time Unix sysadmin it just doesn't make sense to run something as root unless you absolutely have to. Which also makes the client excuse in the article so strange, they had to run the container privileged to run static code analysis. wtf. Doesn't that just mean they run a tool against a binary artefact from a previous job? I fail to see how that requires privileges.

wdb 5y ago
If you start fresh would you use Kaniko or Podman? Trying to find out the best way to avoid docker —privileged on custom Gitlab runner
- INTPenis 5y ago
  Podman all the way. I abandoned kaniko as soon as I found podman. That also eliminates the risk of accessing the host docker.
  - wdb 5y ago
    Cool, going to try it out. Wondering if the cache layering still works with it

rawgabbit 5y ago

You would think by now we would have better credential methods. I still see username and passwords for system credentials. I see tokens created by three legged auths. I don’t get how that is an improvement. The problem is that most deployed code doesn’t have just one credential but a dozen. Multiply that with several environments and you get security fatigue and apathy.

mdoms 5y ago

> The credentials gave the NCC Group consultant access as a limited user to the Jenkins Master web login UI which was only accessible internally and not from the Internet. After a couple of clicks and looking around in the cluster they were able to switch to an administrator account. These kinds of statements are giving major "draw the rest of the owl" vibes. https://i.kym-cdn.com/photos/images/newsfeed/000/572/078/d6d.jpg https://i.kym-cdn.com/photos/images/newsfeed/000/572/078/d6d...

staticassertion 5y ago

Thank you for writing this up. Some thoughts: 1. Hardcoded credentials are a plague. You should consider tagging all of your secrets so that they're easier to scan for. Github automatically scans for secrets, which is great. 2. Jenkins is particularly bad for security. I've seen it owned a million and one times. 3. Containers are overused as a security boundary and footguns like `--privileged` completely eliminate any boundary. 4. Environment variables are a dangerous place to store secrets - they're global to the process and therefor easy to leak. I've thought about this a lot lately, especially after log4j. I think one pattern that may help is clearing the variables after you've loaded them into memory. Another I've considered is encrypting the variables. A lot of the time what you have is something like this: Secret Store -> Control Plane Agent -> Container -> Process Where secrets flow from left to right. The control plane agent and container have full access to the credentials and they're "plaintext" in the Process's environment. In theory you should be able to pin the secrets to that process with a key. During your CD phase you would embed a private key into the process's binary (or a file on the container) and then tell your Secret Manager to use the associated public key to transmit the secrets. The process could decrypt those secrets with its private key but they're E2E encrypted across any hops between the Secret Store and Process and they can't be leaked without explicitly decrypting them first.

teddyh 5y ago
> Environment variables are a dangerous place to store secrets - they're global to the process and therefor easy to leak. The two real problems with environment variables are: 1. Environment variables are traditionally readable by any other process in the system. There are settings you can do on modern kernels to turn this off, but how do you know that you will always run on such a system? 2. Environment variables are inherited to all subprocesses by default, unless you either unset them after you fork() (but before you exec()), or if you take special care to use execve() (or similar) function to provide your own custom-made environment for the new process.
- otterley 5y ago
  A foreign process’s environment variables are only readable if the current UID is root or is the same as the foreign process’s ID. As user joe I can’t see user andrea’s process’s envvars.

thomasmarcelis 5y ago

>>In our final scenario, the NCC Group consultant got booked on a scenario-based assessment: >>“Pretend you have compromised a developer’s laptop.” Most companies will fail right here. Especially outside of the tech world security hygiene with developer's laptops is very bad from what I have seen.

movedx 5y ago

This is exactly why I both love and hate CI/CD. Ultimately most CI/CD setups are basically systems administrators with privileged access to everything, network connected and running 24/7. It's pretty dangerous stuff. I don't have an answer though, expect maybe to keep the CI and CD in separate, isolated instances that require manual intervention to bridge the gap on a case by case basis. That doesn't scale very well though.

hinkley 5y ago
I think in general we put too much logic into our CI/CD configurations. There is an argument to be made for a minimalist CI/CD implementation that can handle task scheduling and dependencies, understands how to fetch and tag version control, count version numbers and not much else. Even extracting test result summaries, while handy, maybe should be handled another way. For many of us, if CI is down you can't deploy anything to production, not even roll back to a previous build. Everything but the credentials should be under version control, and the right people should be able to fire off a one-liner from a runbook that has two to four sanity checked arguments in order to trigger a deployment.
- jeffjeff2 5y ago
  A place I was at recently used makefiles in each project containing entries for "build", "deploy", "test" etc. All the CI did was use shared generic jobs to call the relevant makefile command. These could just as easily be run from a developers machine with the right credentials.
  - john-tells-all 5y ago
    I strongly recommend this. Put all the CICD machinery inside the dev repos, letting all the devs see, understand, and modify the pipeline. With carefully chosen creds, Devs can run the CICD things directly, thus making feedback loops much faster in certain circumstances. Maybe don't let them delete databases, or run expensive VMs, but other than that go for it. In my experience if the pipeline is even slightly different Devs will treat it as a foreign object and will whine about it not working the way they want... they don't feel like they can change the pipeline, which is odd. Dude, it's code, just change it, I'm happy to approve your PR and/or help you make a change.

jiggawatts 5y ago

A weakness of modern secret management is that it isn’t. A secret value ought to be very carefully guarded even from the host machine itself. .NET for example has SecureString, which is a good start — it can’t be accidentally printed or serialised insecurely. If it is serialised, then it is automatically encrypted by the host OS data protection API. Windows even has TPM-hosted certificates! They’re essentially a smart card plugged into the motherboard. A running app can use a TPM credential to sign requests but it can’t read or copy it. These advancements are just completely ignored in the UNIX world, where everything is blindly copied into easily accessible locations in plain text…

pxx 5y ago
Except afaict SecureString doesn't reliably do that and shouldn't be used. https://github.com/dotnet/platform-compat/blob/master/docs/DE0001.md https://github.com/dotnet/platform-compat/blob/master/docs/D...
- jiggawatts 5y ago
  “It’s not perfectly secure so use a totally insecure alternative instead” seems like terrible advice.
  - pxx 5y ago
    No it's "don't use this thing which doesn't do what it says on the tin and is therefore a foot gun." Something that is obviously insecure will be treated with more caution / put on the correct side of the authorization boundary compared to something that claims to be.
    - jiggawatts 5y ago
      > "obviously insecure will be treated with more caution" What planet are you from and can I go there? SecureString provides one layer in the "defence in depth". If someone accidentally logs it, then it won't leak. If it is used as a script parameter, then the prompt will use password input characters automatically. Seriously, I want you to try this PowerShell snippet right now. (You can even run it on Linux too with pwsh v7, so no excuses!): PARAM( [parameter(Mandatory=$true)] [securestring]$Secret ) Write-Warning "Oops I didn't mean to log this: $Secret" $Secret | ConvertTo-Json -Compress $Secret | Export-Clixml 'accidental serialisation.xml' Get-Content 'accidental serialisation.xml' Run the the script and see what happens. There's even low-level protection built-in, such zeroing out the memory when it is garbage collected (unlike normal strings). Why is this a bad thing!? Do you like secrets leaking everywhere unless everyone is always hyper vigilant? Or do you prefer to roll your own half-baked secret storage type that nothing else is compatible with? PS: The page with that advice is based on an "archived" read-only repo with a bunch of open issues of people befuddled as to why this bad, BAD, BAD advice is being published there.

tialaramex 5y ago

A recurring theme is that they obtain secret credentials from a service which needs to verify credentials, and then turn around and use those to impersonate the entity providing those credentials. For example getting Jenkins to run some Groovy discovers credentials Jenkins uses to verify who is accessing it, and then you can just use those credentials yourself. To fix this - almost anywhere - stop using shared secrets. Every time you visit a (HTTPS) web site, you are provided with the credentials to verify its identity. But, you don't gain the ability to impersonate the site because they're not secret credentials, they're public. You can and should use this in a few places in typical CI / CD type infrastructure today, and we should be encouraging other services to enable it too ASAP. In a few places they mention MFA. Again, most MFA involves secrets, for example TOTP Relying Parties need to know what code you should be typing in, so, they need the seed from which to generate that code, and attackers can steal that seed. WebAuthn doesn't involve secrets, so, attackers who steal WebAuthn credentials don't achieve anything. Unfortunately chances are you enabled one or more vulnerable credential types "just in case"...