8 ms·

CircleCI Security Incident

162 points by sciurus 7y ago

aftbit 7y ago
This is how you handle a security notification. Well done CircleCI, looking forward to the full postmortem.
- rvz 7y ago
  I absolutely agree. This is a very professional example of handling a serious security incident. CircleCI immediately took action and notified all affected accounts afterwards and as with this post, they did a detailed FAQ-style breakdown of what happened and how they will stop another attack. > Security is taken very seriously here at CircleCI. Compared with the other security handlings i've seen, this statement sounds genuine.
- ilogik 7y ago
  I love the fact that it took them 10 minutes to first respond to the incident, and cut-off the account in under 30 minutes from the first sign something was wrong
EamonnMR 7y ago
Initially freaked out when I got the email, but it looks like actual repo contents weren't compromised.
cgijoe 7y ago
Dupe? https://news.ycombinator.com/item?id=20882624 https://news.ycombinator.com/item?id=20882624
- paulddraper 7y ago
  Yes, that was later. Comments have been moved.
traskjd 7y ago
I didn't see it, but "who is the third party vendor?" is my key question, since this could be impacting anyone working with that vendor.
- antoncohen 7y ago
  The vendor might not have been compromised. > we will review our policies for enforcing 2FA on third-party accounts to the extent possible, and continue our transition to single sign-on (SSO) for all of our integrations I think it is more likely the CircleCI didn't enforce 2FA or SSO[1] with the vendor, and a CircleCI employee reused an already compromised password. [1] Nearly all SSO identity providers support 2FA, so using SSO is often the easiest way to get 2FA with third-party SaaS providers.
  - traskjd 7y ago
    Valid point, thanks :)
  - gingerlime 7y ago
    Naming the vendor in this case should be fine. Shouldn’t it? In general what I’m missing is why they share my email and github account with a 3rd party. GDPR should prevent them from doing so (at least for European users) without a legitimate reason, or explicit consent.
    - erikpukinskis 7y ago
      Is "they make graphs for us of what our users do" a legitimate reason? Or, "they store our error logs in a way we can easily find stuff"?
      - gingerlime 7y ago
        I'm not sure "make graphs for us" is a legitimate reason to share personal information (and email is considered personal information under GDPR). Error logs might be a bit different I suppose, if they are used to help resolve specific issues for your users, but even then I suppose one could argue that pseudo-anonymised data would be better than sharing an email address. I'm no GDPR expert, but that's my understanding of it.
telecuda 7y ago
Curious to hear from CircleCI how their FedRAMP authorization played into what was (or wasn’t) compromised and if it improved their response. https://circleci.com/blog/modernizing-federal-devops-circleci-becomes-first-continuous-integration-tool-with-fedramp-authorization/ https://circleci.com/blog/modernizing-federal-devops-circlec...
rficcaglia 7y ago
The initial reaction is to TL;DR as weak passwords and no MFA but I am equally/more concerned that the policy granted to vendors allows them to access more data than they really need, or the data they must have access could be better de-identified. For example they could hash/uuid the email and the git repo and branch info. Hacks are going to happen; no password or MFA will prevent it. Zero trust in your policy/schema design minimizes blast radius in these inevitable cases.
- pushpop 7y ago
  The idea of treating email addresses as secrets is a nice idea in theory but given the plethora of databases where it’s already stored as a user name, including those that have already been compromised, and how email addresses are published in the git commits anyway, I don’t think it makes much sense hashing email addresses in practice. Repository details are another interesting point though. They too are published openly in any git clones but I accept that some people’s repositories, including any local clones, would be private and kept secured from prying eyes. That said, what matters more there is access credentials, encryption at rest (for local clones), etc. So keeping the branch names and repository URIs secret feels a little redundant compared to the actual security benefit it brings. One might even say it’s security through obscurity, though I don’t personally think it offers up even enough benefit to fall under that category. That said, I’m happy to be convinced otherwise if you feel I’ve overlooked some important detail.
147 7y ago
Relevant post on HN 2 years ago: https://news.ycombinator.com/item?id=15442636 https://news.ycombinator.com/item?id=15442636
- robocat 7y ago
  "CircleCI trusts 8 analytics companies with your source code and API tokens, October 9, 2017. When you navigate to your project in CircleCI's UI, Javascript from eight different analytics companies gets loaded and executed in your browser." So, did they do anything about that? Even if using iframes and a secure api over postMessage/onmessage: safe, and security can be controlled by the external provider i.e. safer for both parties (ideally the servers pass the security context between each other - we even use an iframe internally for all the same reasons). https://kevin.burke.dev/kevin/circleci-is-hopelessly-insecure/ https://kevin.burke.dev/kevin/circleci-is-hopelessly-insecur...
  - deleted 7y ago
    [deleted]
  - aeonflux 7y ago
    > for e.g CircleCI’s account for Google Analytics Google allows custom JS injection via Tag manager, so there are possible vectors, even if only access to 3rd party site was stolen. Of course I have no idea what was the deal here. It's worth noting, that CircleCI did nothing to address issues raised in the linked post. Their main app still loads tons of third party analytics garbage for e.g. Google, Hotjar, Amplitude and even Facebook of all people. I do block all those, but as someone pointed out it's not a solution and cant even be reliable at all times. Please bear in mind, that CircleCI not only has access to private repositories. More often then not they do store private SSH keys to your production servers.
- kevinburke 7y ago
  Hi, I wrote that post. I don’t think that this is quite the same issue. The 2019 post is vague but I think what happened is an attacker stole the credentials for e.g CircleCI’s account for Google Analytics. So an attacker could see URL’s that were visited, IP addresses, but they couldn’t run JavaScript on CircleCI’s domain. The difference is between an attacker compromising the Google Analytics JavaScript snippet CDN and an attacker compromising a Google Analytics user account. The latter only lets you read the metadata not make any of the juicy AJAX calls. (Also, it’s probably not actually Google Analytics, just listing a tool most people are familiar with). It’s obviously not good but it’s not the same vulnerability I wrote about nor as bad as the one I described two years ago.
Proven 7y ago
No suspicion of data tampering and that’s great, but the attacker got email addresses and possibly identities of CircleCI users. If you own an anti-gov repo and have been active in the past 2 months, maybe you’ve been ID’d. Not a pleasant thought. Do we really have to use throwaway email accounts for everything? That is depressing.
jacques_chester 7y ago
> On August 31st at 2:32 p.m. UTC, a CircleCI team member saw an email notification from one of our third-party analytics vendors and suspected that unusual activity was taking place in this particular vendor account. A convention I like is that team members creating, updating or removing accounts should reply to any automated emails sent to mailing lists saying "this is me". Ideally with a link to the relevant issue or story. It really cuts down on unnecessary nerves.
nisten 7y ago
What's sad about this is that CircleCI actually has a great product and is one of the nicest ways to do end to end automation for mobile development/releases. Having their pipeline in place actually feels quite liberating. The sad part is that they take this for granted and liberate all your data and security weaknesses too to unknown third parties for either a weird ideological reason about interoperability or a small marginal profit. I don't understand why they can't do both(interoperability and security), their main profit is not data, it's customers paying for infrasctructure. They don't have to share any data at all and are fully capable of providing a competent level of security. If anything I think if they cut out the profit line from sharing metrics etc with third parties it would actually bring more customers in because of increased confidence in the product.
- jjeaff 7y ago
  I doubt they are getting paid to share any data at all. Usually, the point of 3rd party services is for improving the service or keeping it bug free and functional. I would imagine they have some services that are logging JavaScript errors and probably some that are tracking how people use the service. If you have a Freemium or tiered service especially, you have to understand how people use your service in order to convince them to pay or upgrade.
  - aeonflux 7y ago
    > you have to understand how people use your service Thats why they need Facebook's tracking inside the admin part of the APP (on na paid account)?
- PaulRobinson 7y ago
  Ex-Circler here, I was there for the second half of 2018. The product is the source of profit, not the data. I was never asked to work on anything relating to sharing metrics for money, and didn't hear of any such product in my time there. That's not the business model. In fact, some users the only thing they know is the VCS username, and only then because that user has pushed a commit that has triggered a build: the user data they have for most users is literally worthless to any other party.
mr_justin 7y ago
Nobody deserves this, but lordy I have lost days of dev time to Circle in the last several years.
- paulddraper 7y ago
  For what?
  - JshWright 7y ago
    Well, for starters their service availability is... not stellar.
- SkyPuncher 7y ago
  If I have the choice, I will not be starting any project with them in the future after the crap they pulled with CircleCI 2.0. "We're introducing this new, 2.0, version. It has some benefits, but is also missing a lot of features and bug ridden. It's easier for us though, so you're going to have to waste a couple days converting from 1.0 to 2.0."
  - aeonflux 7y ago
    My feelings as well. What we really need is a ansible script or a docker file, which would build a Jenkins instance which everyone could use. You would of course need different tools for various build setup and technologies, but CircleCI is far from point and click anyway. I really which I would spend all the time I wasted on their config to build the above. This way everyone could run 5000$/mo of CircleCI on a 50$/mo bare metal.
    - nathanaldensr 7y ago
      You might want to try Azure Pipelines. It uses a dual model of Microsoft-hosted build agents running several OSes including macOS, and self-hosted build agents running on infrastructure of your choosing. The team I lead uses for Windows-, Linux-, and macOS-based builds, Microsoft-hosted and self-hosted build agents.
symlinkk 7y ago
I have a feeling this is the nail in the coffin for CircleCI. Github Actions are the way forward.
guytv 7y ago
> Security is taken very seriously here at CircleCI ... > In the meantime, we will review our policies for enforcing 2FA on third-party accounts to the extent possible, and continue our transition to single sign-on (SSO) for all of our integrations Well, if you had taken security seriously, the first thing you'd do is ensure 2fa on all accounts.
mindfulplay 7y ago
I would like one company to post an "incident" reveal in a more honest way: " Due to our carelessness and relatively insecure practices, we had improperly disclosed user accounts to a moderately savvy hacker. We realize this is our fault. If you'd like to help and given that we have your attention now, it would be valuable if you can help pentest our servers: the attacker used a simple SQL attack based on an unpatched server via CVE-3245. Are we missing anything else? Please let us know. Thank you."
agustif 7y ago
I recieved the email from them a few hours ago, because of the wording of that, and because I spend last night integrating segment's analytics on my project. I might suspect bc of their wording, segment and not google analytics would be such said 3rd party, and it seems it was one of the team's accounts in such service which got compromised and created a new destination, and no one noticed until 2 months later.. This is just pure speculation, but I would like to know!
- jbattle 7y ago
  I just got an email from Segment reporting something very similar. I think you are right
  - agustif 7y ago
    Yeah I just saw it on HN homepage, By the timing of it an alert about -segment- or just a segment employee watching this comment thread might have forced them to come clean sooner than they were planning to?
paulcarroty 7y ago
CircleCI also use wordpress, not ideal in security aspect (as Google Analytics, too), be honest. Also found these services inside my profile page: * cloudfront * githubusercontent * launchdarkly * zendesk * zopim * gravatar * optimizely * pusher * pusherapp * segment * statuspage * wp * zdassets (3 domains) I use uMatrix to cut half of this crap, cause the page is too heavy to load on my laptop. Started looking for alternatives.
minimaul 7y ago
CircleCI has been very frustrating for us recently. We have a lot of problems with waiting for builds to start - it seems CircleCI don’t have even remotely enough capacity at peak times. We regularly get 10min plus delays to builds starting, sometimes it’s over 30 mins! Mix that with this security issue... it’s not a rosy outlook for us. Edit: oh, and the UI has terrible problems with caching and not updating state. Constantly have to hard refresh pages without cache for build status to update.

162 points by sciurus 7y ago

aftbit 7y ago

This is how you handle a security notification. Well done CircleCI, looking forward to the full postmortem.

rvz 7y ago
I absolutely agree. This is a very professional example of handling a serious security incident. CircleCI immediately took action and notified all affected accounts afterwards and as with this post, they did a detailed FAQ-style breakdown of what happened and how they will stop another attack. > Security is taken very seriously here at CircleCI. Compared with the other security handlings i've seen, this statement sounds genuine.
ilogik 7y ago
I love the fact that it took them 10 minutes to first respond to the incident, and cut-off the account in under 30 minutes from the first sign something was wrong

EamonnMR 7y ago

Initially freaked out when I got the email, but it looks like actual repo contents weren't compromised.

cgijoe 7y ago

Dupe? https://news.ycombinator.com/item?id=20882624 https://news.ycombinator.com/item?id=20882624

paulddraper 7y ago
Yes, that was later. Comments have been moved.

traskjd 7y ago

I didn't see it, but "who is the third party vendor?" is my key question, since this could be impacting anyone working with that vendor.

antoncohen 7y ago
The vendor might not have been compromised. > we will review our policies for enforcing 2FA on third-party accounts to the extent possible, and continue our transition to single sign-on (SSO) for all of our integrations I think it is more likely the CircleCI didn't enforce 2FA or SSO[1] with the vendor, and a CircleCI employee reused an already compromised password. [1] Nearly all SSO identity providers support 2FA, so using SSO is often the easiest way to get 2FA with third-party SaaS providers.
- traskjd 7y ago
  Valid point, thanks :)
- gingerlime 7y ago
  Naming the vendor in this case should be fine. Shouldn’t it? In general what I’m missing is why they share my email and github account with a 3rd party. GDPR should prevent them from doing so (at least for European users) without a legitimate reason, or explicit consent.
  - erikpukinskis 7y ago
    Is "they make graphs for us of what our users do" a legitimate reason? Or, "they store our error logs in a way we can easily find stuff"?
    - gingerlime 7y ago
      I'm not sure "make graphs for us" is a legitimate reason to share personal information (and email is considered personal information under GDPR). Error logs might be a bit different I suppose, if they are used to help resolve specific issues for your users, but even then I suppose one could argue that pseudo-anonymised data would be better than sharing an email address. I'm no GDPR expert, but that's my understanding of it.

telecuda 7y ago

Curious to hear from CircleCI how their FedRAMP authorization played into what was (or wasn’t) compromised and if it improved their response. https://circleci.com/blog/modernizing-federal-devops-circleci-becomes-first-continuous-integration-tool-with-fedramp-authorization/ https://circleci.com/blog/modernizing-federal-devops-circlec...

rficcaglia 7y ago

The initial reaction is to TL;DR as weak passwords and no MFA but I am equally/more concerned that the policy granted to vendors allows them to access more data than they really need, or the data they must have access could be better de-identified. For example they could hash/uuid the email and the git repo and branch info. Hacks are going to happen; no password or MFA will prevent it. Zero trust in your policy/schema design minimizes blast radius in these inevitable cases.

pushpop 7y ago
The idea of treating email addresses as secrets is a nice idea in theory but given the plethora of databases where it’s already stored as a user name, including those that have already been compromised, and how email addresses are published in the git commits anyway, I don’t think it makes much sense hashing email addresses in practice. Repository details are another interesting point though. They too are published openly in any git clones but I accept that some people’s repositories, including any local clones, would be private and kept secured from prying eyes. That said, what matters more there is access credentials, encryption at rest (for local clones), etc. So keeping the branch names and repository URIs secret feels a little redundant compared to the actual security benefit it brings. One might even say it’s security through obscurity, though I don’t personally think it offers up even enough benefit to fall under that category. That said, I’m happy to be convinced otherwise if you feel I’ve overlooked some important detail.

147 7y ago

Relevant post on HN 2 years ago: https://news.ycombinator.com/item?id=15442636 https://news.ycombinator.com/item?id=15442636

robocat 7y ago
"CircleCI trusts 8 analytics companies with your source code and API tokens, October 9, 2017. When you navigate to your project in CircleCI's UI, Javascript from eight different analytics companies gets loaded and executed in your browser." So, did they do anything about that? Even if using iframes and a secure api over postMessage/onmessage: safe, and security can be controlled by the external provider i.e. safer for both parties (ideally the servers pass the security context between each other - we even use an iframe internally for all the same reasons). https://kevin.burke.dev/kevin/circleci-is-hopelessly-insecure/ https://kevin.burke.dev/kevin/circleci-is-hopelessly-insecur...
- deleted 7y ago
  [deleted]
- aeonflux 7y ago
  > for e.g CircleCI’s account for Google Analytics Google allows custom JS injection via Tag manager, so there are possible vectors, even if only access to 3rd party site was stolen. Of course I have no idea what was the deal here. It's worth noting, that CircleCI did nothing to address issues raised in the linked post. Their main app still loads tons of third party analytics garbage for e.g. Google, Hotjar, Amplitude and even Facebook of all people. I do block all those, but as someone pointed out it's not a solution and cant even be reliable at all times. Please bear in mind, that CircleCI not only has access to private repositories. More often then not they do store private SSH keys to your production servers.
kevinburke 7y ago
Hi, I wrote that post. I don’t think that this is quite the same issue. The 2019 post is vague but I think what happened is an attacker stole the credentials for e.g CircleCI’s account for Google Analytics. So an attacker could see URL’s that were visited, IP addresses, but they couldn’t run JavaScript on CircleCI’s domain. The difference is between an attacker compromising the Google Analytics JavaScript snippet CDN and an attacker compromising a Google Analytics user account. The latter only lets you read the metadata not make any of the juicy AJAX calls. (Also, it’s probably not actually Google Analytics, just listing a tool most people are familiar with). It’s obviously not good but it’s not the same vulnerability I wrote about nor as bad as the one I described two years ago.

Proven 7y ago

No suspicion of data tampering and that’s great, but the attacker got email addresses and possibly identities of CircleCI users. If you own an anti-gov repo and have been active in the past 2 months, maybe you’ve been ID’d. Not a pleasant thought. Do we really have to use throwaway email accounts for everything? That is depressing.

jacques_chester 7y ago

> On August 31st at 2:32 p.m. UTC, a CircleCI team member saw an email notification from one of our third-party analytics vendors and suspected that unusual activity was taking place in this particular vendor account. A convention I like is that team members creating, updating or removing accounts should reply to any automated emails sent to mailing lists saying "this is me". Ideally with a link to the relevant issue or story. It really cuts down on unnecessary nerves.

nisten 7y ago

What's sad about this is that CircleCI actually has a great product and is one of the nicest ways to do end to end automation for mobile development/releases. Having their pipeline in place actually feels quite liberating. The sad part is that they take this for granted and liberate all your data and security weaknesses too to unknown third parties for either a weird ideological reason about interoperability or a small marginal profit. I don't understand why they can't do both(interoperability and security), their main profit is not data, it's customers paying for infrasctructure. They don't have to share any data at all and are fully capable of providing a competent level of security. If anything I think if they cut out the profit line from sharing metrics etc with third parties it would actually bring more customers in because of increased confidence in the product.

jjeaff 7y ago
I doubt they are getting paid to share any data at all. Usually, the point of 3rd party services is for improving the service or keeping it bug free and functional. I would imagine they have some services that are logging JavaScript errors and probably some that are tracking how people use the service. If you have a Freemium or tiered service especially, you have to understand how people use your service in order to convince them to pay or upgrade.
- aeonflux 7y ago
  > you have to understand how people use your service Thats why they need Facebook's tracking inside the admin part of the APP (on na paid account)?
PaulRobinson 7y ago
Ex-Circler here, I was there for the second half of 2018. The product is the source of profit, not the data. I was never asked to work on anything relating to sharing metrics for money, and didn't hear of any such product in my time there. That's not the business model. In fact, some users the only thing they know is the VCS username, and only then because that user has pushed a commit that has triggered a build: the user data they have for most users is literally worthless to any other party.

mr_justin 7y ago

Nobody deserves this, but lordy I have lost days of dev time to Circle in the last several years.

paulddraper 7y ago
For what?
- JshWright 7y ago
  Well, for starters their service availability is... not stellar.
SkyPuncher 7y ago
If I have the choice, I will not be starting any project with them in the future after the crap they pulled with CircleCI 2.0. "We're introducing this new, 2.0, version. It has some benefits, but is also missing a lot of features and bug ridden. It's easier for us though, so you're going to have to waste a couple days converting from 1.0 to 2.0."
- aeonflux 7y ago
  My feelings as well. What we really need is a ansible script or a docker file, which would build a Jenkins instance which everyone could use. You would of course need different tools for various build setup and technologies, but CircleCI is far from point and click anyway. I really which I would spend all the time I wasted on their config to build the above. This way everyone could run 5000$/mo of CircleCI on a 50$/mo bare metal.
  - nathanaldensr 7y ago
    You might want to try Azure Pipelines. It uses a dual model of Microsoft-hosted build agents running several OSes including macOS, and self-hosted build agents running on infrastructure of your choosing. The team I lead uses for Windows-, Linux-, and macOS-based builds, Microsoft-hosted and self-hosted build agents.

symlinkk 7y ago

I have a feeling this is the nail in the coffin for CircleCI. Github Actions are the way forward.

guytv 7y ago

> Security is taken very seriously here at CircleCI ... > In the meantime, we will review our policies for enforcing 2FA on third-party accounts to the extent possible, and continue our transition to single sign-on (SSO) for all of our integrations Well, if you had taken security seriously, the first thing you'd do is ensure 2fa on all accounts.

mindfulplay 7y ago

I would like one company to post an "incident" reveal in a more honest way: " Due to our carelessness and relatively insecure practices, we had improperly disclosed user accounts to a moderately savvy hacker. We realize this is our fault. If you'd like to help and given that we have your attention now, it would be valuable if you can help pentest our servers: the attacker used a simple SQL attack based on an unpatched server via CVE-3245. Are we missing anything else? Please let us know. Thank you."

agustif 7y ago

I recieved the email from them a few hours ago, because of the wording of that, and because I spend last night integrating segment's analytics on my project. I might suspect bc of their wording, segment and not google analytics would be such said 3rd party, and it seems it was one of the team's accounts in such service which got compromised and created a new destination, and no one noticed until 2 months later.. This is just pure speculation, but I would like to know!

jbattle 7y ago
I just got an email from Segment reporting something very similar. I think you are right
- agustif 7y ago
  Yeah I just saw it on HN homepage, By the timing of it an alert about -segment- or just a segment employee watching this comment thread might have forced them to come clean sooner than they were planning to?

paulcarroty 7y ago

CircleCI also use wordpress, not ideal in security aspect (as Google Analytics, too), be honest. Also found these services inside my profile page: * cloudfront * githubusercontent * launchdarkly * zendesk * zopim * gravatar * optimizely * pusher * pusherapp * segment * statuspage * wp * zdassets (3 domains) I use uMatrix to cut half of this crap, cause the page is too heavy to load on my laptop. Started looking for alternatives.

minimaul 7y ago

CircleCI has been very frustrating for us recently. We have a lot of problems with waiting for builds to start - it seems CircleCI don’t have even remotely enough capacity at peak times. We regularly get 10min plus delays to builds starting, sometimes it’s over 30 mins! Mix that with this security issue... it’s not a rosy outlook for us. Edit: oh, and the UI has terrible problems with caching and not updating state. Constantly have to hard refresh pages without cache for build status to update.