6 ms·

That’s a really smart breach target, since reading their security notice it sounds like it works by you downloading and executing their code in your CI environm

by mcintyre1994 6y ago

That’s a really smart breach target, since reading their security notice it sounds like it works by you downloading and executing their code in your CI environment. So their code has access to your code and all the secrets you inject into your CI environment. There’s probably a lot of pipelines with deploy credentials available to other jobs. As an example I’m most familiar with, Gitlab provides all environment variables to all jobs so if you’re using them for deploy tokens then any job like code coverage can access them. They do document this and offer an alternative in something like Hashicorp Vault, but it’s a big target regardless.

raffraffraff 6y ago
Well, a minor clarification... If you have multiple projects you can set up different per-project environment variables. But it's fair to say that all jobs within a project have access to all environment variables in that project.
jrockway 6y ago
The other thing that is interesting about Codecov is how they distribute their analysis/uploader. You download a fresh uploader for every CI run, even if you're using your CI platform's native integration (i.e. the CircleCI "orb"). I get why they do this -- if they instructed people to download a binary and store it, customers would do that exactly once and never update it, causing them to have to support old versions of their API indefinitely (or annoy their customers greatly). So basically, we took a shortcut that triggers everyone's security spidey sense, and exactly as predicted, we all got burned. (I will point out that their uploader is literally a curl | bash, but that isn't really the problem here. Installing a Debian package or downloading a statically linked Go binary would be just as risky. Actually, it would have been harder to find the underlying problem.) All in all I'm pretty disappointed. We trusted Codecov with a shortcut that makes it easy for them to deliver a better product. They didn't have any security in place to ensure that what was checked into source control was what customers downloaded, and now we're all hacked.
- captn3m0 6y ago
  Debian packages can be signed, so depending on how the signing keys are managed - it might have been much easier to find the underlying problem.
  - jrockway 6y ago
    True! I think since in the CI case you're bootstrapping a fresh machine for every build, you'd have to apt-key add their key as part of installing the package, so an attacker would just make a new one of those. I bet the compromise duration would have been shorter, though; there's got to be one engineer that installs the Debian package on their workstation and would be confused when their release system's signature stopped working. (And honestly, an attacker that can build a Debian package and update the public key for apt-key is probably harder to find than one that can edit a cleartext bash script. Security through hoping your adversaries can't perform tedious devops operations is no security at all, as they say, but in the real world it probably helps a tiny bit.)
    - CameronNemo 6y ago
      I'm not above throwing public keys in the git repo for use in CI/CD jobs.
- hakre 6y ago
  They offer, but you don't have to. Only add the uploader that passed your review and use that, not just anything from the interwebs.
- WhyNotHugo 6y ago
  You can use their bash uploaded, but you can also use language specific ones. The Python one, for example, is downloaded via PyPI, and you’ll usually have a pinned version with the hash of it. Any tampering would make pipelines fail. If a new release were pushed by rogue actors, someone would notice an unplanned version bump.
  - 6y ago
PaywallBuster 6y ago
you can also scope down variables by branch
qeternity 6y ago
Gitlab has protected variables which are only made available to protected branches (i.e. the ones that would have deploy stages). But yeah in general secrets management is a trainwreck.
- WhyNotHugo 6y ago
  Those branches still run tests and coverage, so it’s likely that code of would run on them anyway.
  - deleted 6y ago
    [deleted]
  - lloeki 6y ago
    The design is that such tests would be run in PR non-protected branches and thus reviewed before being merged into protected branches for later deployment; that, we agree as much. I guess the protected branch segregation should go -both ways- and more effort should be made into auditing and vetting which CI step has run access to secret-authorised branches.
    - CameronNemo 6y ago
      Protected secrets has always seemed like such a rough barrier. No fine tuning at all (e.g. if you have stage and prod creds that you want to be kept separate, but both "protected"... No good way to do that from my POV).
alfiedotwtf 6y ago
CI has always been a Yellow Smiley Face Post-it, and the only thing I can true see to combat this is reproducible builds - just like if your repo was compromised, a dev is going to notice pretty fast if their local build differs from a compromised CI build
- watermelon0 6y ago
  Would they really notice? You can have reproducible builds, but unless someone explicitly checks checksums of local builds, it won't be noticed.
moondev 6y ago
I'm not that familiar with codecov but it seems to me way worse then that. The "bash uploader" that was compromised is designed to curl and run a script on codecov's server. This script was compromised to in turn silently run a script from the attacker's server. This could have done anything but I doubt they just dumped ENV, probably opened a reverse shell to literally do anything they want as the user that originally invoked the script. This could be anything from stealing ~/.kube/config and ~/.ssh/id_rsa... If one of the 29,000 customers invoked as root it could be even worse, such as adding users and compromising entire platforms.
bchapuis 6y ago
A search for their bash script on github gives more than 400K results... The scale of this breach is staggering. https://github.com/search?q=https%3A%2F%2Fcodecov.io%2Fbash&type=code https://github.com/search?q=https%3A%2F%2Fcodecov.io%2Fbash&...

raffraffraff 6y ago

Well, a minor clarification... If you have multiple projects you can set up different per-project environment variables. But it's fair to say that all jobs within a project have access to all environment variables in that project.

jrockway 6y ago

The other thing that is interesting about Codecov is how they distribute their analysis/uploader. You download a fresh uploader for every CI run, even if you're using your CI platform's native integration (i.e. the CircleCI "orb"). I get why they do this -- if they instructed people to download a binary and store it, customers would do that exactly once and never update it, causing them to have to support old versions of their API indefinitely (or annoy their customers greatly). So basically, we took a shortcut that triggers everyone's security spidey sense, and exactly as predicted, we all got burned. (I will point out that their uploader is literally a curl | bash, but that isn't really the problem here. Installing a Debian package or downloading a statically linked Go binary would be just as risky. Actually, it would have been harder to find the underlying problem.) All in all I'm pretty disappointed. We trusted Codecov with a shortcut that makes it easy for them to deliver a better product. They didn't have any security in place to ensure that what was checked into source control was what customers downloaded, and now we're all hacked.

captn3m0 6y ago
Debian packages can be signed, so depending on how the signing keys are managed - it might have been much easier to find the underlying problem.
- jrockway 6y ago
  True! I think since in the CI case you're bootstrapping a fresh machine for every build, you'd have to apt-key add their key as part of installing the package, so an attacker would just make a new one of those. I bet the compromise duration would have been shorter, though; there's got to be one engineer that installs the Debian package on their workstation and would be confused when their release system's signature stopped working. (And honestly, an attacker that can build a Debian package and update the public key for apt-key is probably harder to find than one that can edit a cleartext bash script. Security through hoping your adversaries can't perform tedious devops operations is no security at all, as they say, but in the real world it probably helps a tiny bit.)
  - CameronNemo 6y ago
    I'm not above throwing public keys in the git repo for use in CI/CD jobs.
hakre 6y ago
They offer, but you don't have to. Only add the uploader that passed your review and use that, not just anything from the interwebs.
WhyNotHugo 6y ago
You can use their bash uploaded, but you can also use language specific ones. The Python one, for example, is downloaded via PyPI, and you’ll usually have a pinned version with the hash of it. Any tampering would make pipelines fail. If a new release were pushed by rogue actors, someone would notice an unplanned version bump.
- 6y ago

PaywallBuster 6y ago

you can also scope down variables by branch

qeternity 6y ago

Gitlab has protected variables which are only made available to protected branches (i.e. the ones that would have deploy stages). But yeah in general secrets management is a trainwreck.

WhyNotHugo 6y ago
Those branches still run tests and coverage, so it’s likely that code of would run on them anyway.
- deleted 6y ago
  [deleted]
- lloeki 6y ago
  The design is that such tests would be run in PR non-protected branches and thus reviewed before being merged into protected branches for later deployment; that, we agree as much. I guess the protected branch segregation should go -both ways- and more effort should be made into auditing and vetting which CI step has run access to secret-authorised branches.
  - CameronNemo 6y ago
    Protected secrets has always seemed like such a rough barrier. No fine tuning at all (e.g. if you have stage and prod creds that you want to be kept separate, but both "protected"... No good way to do that from my POV).

alfiedotwtf 6y ago

CI has always been a Yellow Smiley Face Post-it, and the only thing I can true see to combat this is reproducible builds - just like if your repo was compromised, a dev is going to notice pretty fast if their local build differs from a compromised CI build

watermelon0 6y ago
Would they really notice? You can have reproducible builds, but unless someone explicitly checks checksums of local builds, it won't be noticed.

moondev 6y ago

I'm not that familiar with codecov but it seems to me way worse then that. The "bash uploader" that was compromised is designed to curl and run a script on codecov's server. This script was compromised to in turn silently run a script from the attacker's server. This could have done anything but I doubt they just dumped ENV, probably opened a reverse shell to literally do anything they want as the user that originally invoked the script. This could be anything from stealing ~/.kube/config and ~/.ssh/id_rsa... If one of the 29,000 customers invoked as root it could be even worse, such as adding users and compromising entire platforms.

bchapuis 6y ago

A search for their bash script on github gives more than 400K results... The scale of this breach is staggering. https://github.com/search?q=https%3A%2F%2Fcodecov.io%2Fbash&type=code https://github.com/search?q=https%3A%2F%2Fcodecov.io%2Fbash&...