7 ms·

CircleCI's incident response

59 points by cr4zy 13y ago

johnwards 13y ago
So, to be clear. I need to contact Github directly to see if my source has been downloaded using compromised deploy keys?
- cr4zy 13y ago
  I would. Although you can see some of that info here: https://github.com/settings/security https://github.com/settings/security Edit: There are also logs for your organization in https://github.com/organizations/<your_organization>/settings/security https://github.com/organizations/<your_organization>/setting...
- pegler 13y ago
  I emailed GitHub this morning around 8am EST. I received this response this afternoon. It sounds like they are doing their own audit and will contact you if they find anything unusual: > I wanted to check in with you about this incident - access to repositories on GitHub is logged and we're currently investigating the potential of unauthorized repository access allowed by the MongoHQ and CircleCI breaches. Although we don’t currently have any evidence of unauthorized access, or specific news regarding your repositories. > Last night, in coordination with CircleCI, we revoked all SSH keys and OAuth tokens added to GitHub accounts and repositories by their service. We'll be in contact again when we have a more substantial update or news about your organization's repositories.
jonny_eh 13y ago
If I had a heroku API key stored with CircleCI, should I assume all of my app's production config variables have been compromised?
- 1qaz2wsx3edc 13y ago
  Err on the side of caution and rotate your keys.
WestCoastJustin 13y ago
Ouch -- Amazing how there is this ripple effect happening now that everone is using the cloud and there are so many middle men or service APIs. > In order to protect your data and users, we strongly urge you to secure each of these systems: > SSH keys uploaded to CircleCI > API tokens stored in CircleCI env vars > API/SSH key stored in a GitHub repo accessible from your CircleCI Interesting to note that Stripe and Kickstarter are customers. This is a little scary to think their source code could have been exposed, given the $$$ flowing through their systems. So, someone could have used these keys to have much wider access!
- sehrope 13y ago
  > Interesting to note that Stripe and Kickstarter are customers. This is a little scary to think their source code could have been exposed, given the $$$ flowing through their systems. If either of them have any sense (which I think they do!) then having there source code exposed shouldn't in and of itself pose a security risk. If it did then that would be security via obscurity. Now I'm not saying that having the source code for a site is useless or that having it exposed is a good thing (it's not!). If there is a bug/hole you could find it, but if there isn't then the source being available should not make anything less secure. > So, someone could have used these keys to have much wider access! SSH keys and API tokens are another story. These need to be revoked immediately (CircleCI's response looks great by the way) as there would be nothing besides firewall rules (if they exist!) preventing an attacker from using them. This whole incident is a friendly reminder to follow the principle of least privilege[1] with your XaaS providers. There's not much you can do for API tokens. If you're relying on someone else to provide a service on your behalf then they'll need their own token. For data though it's possible to encrypt sensitive data on your end and use the DBaaS provider as an opaque blob store. Doing it across the board gets a bit unwieldly as it's hard to query encrypted data but it's fine sensitive details that you don't want to necessarily expose (such as ENV vars, payment info or equivalent tokens). If the provider doesn't need access to X, and X is sensitive, then encrypt it before you save it with them. That way you don't have to worry about them leaking it. One of the best examples of this approach is tarsnap[2] which considers everything sensitive and encrypts everything client side (on your machine) before uploading to its central server (and then S3). EDIT: One more quick point. If you explicitly encrypt sensitive data as part of storing it (regardless of whether you're using an XaaS provider or running a DB yourself) there's one more big advantage: Backups! Your backups themselves should be encrypted (ex: we use gpg prior to uploading them to S3) but in addition to that by following the above, the sensitive fields in the backup are also encrypted. This means that they're encrypted in any scratch/temp files that are part of the backup generation process. Otherwise they would be stored in plaintext on your server and possibly be on your server's hard disk (or your XaaS providers disks) without you even knowing it. Oh and remember that disk blocks don't actually get overwritten when you "rm" a file so this is more important than you think. [1]: http://en.wikipedia.org/wiki/Principle_of_least_privilege http://en.wikipedia.org/wiki/Principle_of_least_privilege [2]: http://www.tarsnap.com/ http://www.tarsnap.com/
tkiley 13y ago
Now might be a good time for circleci customers to pick up a code analysis tool such as brakeman and look for obvious security holes in their apps. If the attackers have hundreds or thousands of web apps' source code, I'd expect them to start trawling for vulnerabilities with automated tools.
damncabbage 13y ago
What happened between the 19th and the 28th? https://news.ycombinator.com/item?id=6638004 https://news.ycombinator.com/item?id=6638004 Hi Justin. To clarify, from what I understand, October 28 is the date MongoHQ detected this. They've provided us with the logs of database access, and unfortunately the queries leading to our spam attack on Saturday started as early as October 19.

59 points by cr4zy 13y ago

So, to be clear. I need to contact Github directly to see if my source has been downloaded using compromised deploy keys?

cr4zy 13y ago
I would. Although you can see some of that info here: https://github.com/settings/security https://github.com/settings/security Edit: There are also logs for your organization in https://github.com/organizations/<your_organization>/settings/security https://github.com/organizations/<your_organization>/setting...
pegler 13y ago
I emailed GitHub this morning around 8am EST. I received this response this afternoon. It sounds like they are doing their own audit and will contact you if they find anything unusual: > I wanted to check in with you about this incident - access to repositories on GitHub is logged and we're currently investigating the potential of unauthorized repository access allowed by the MongoHQ and CircleCI breaches. Although we don’t currently have any evidence of unauthorized access, or specific news regarding your repositories. > Last night, in coordination with CircleCI, we revoked all SSH keys and OAuth tokens added to GitHub accounts and repositories by their service. We'll be in contact again when we have a more substantial update or news about your organization's repositories.

jonny_eh 13y ago

If I had a heroku API key stored with CircleCI, should I assume all of my app's production config variables have been compromised?

1qaz2wsx3edc 13y ago
Err on the side of caution and rotate your keys.

WestCoastJustin 13y ago

Ouch -- Amazing how there is this ripple effect happening now that everone is using the cloud and there are so many middle men or service APIs. > In order to protect your data and users, we strongly urge you to secure each of these systems: > SSH keys uploaded to CircleCI > API tokens stored in CircleCI env vars > API/SSH key stored in a GitHub repo accessible from your CircleCI Interesting to note that Stripe and Kickstarter are customers. This is a little scary to think their source code could have been exposed, given the $$$ flowing through their systems. So, someone could have used these keys to have much wider access!

sehrope 13y ago
> Interesting to note that Stripe and Kickstarter are customers. This is a little scary to think their source code could have been exposed, given the $$$ flowing through their systems. If either of them have any sense (which I think they do!) then having there source code exposed shouldn't in and of itself pose a security risk. If it did then that would be security via obscurity. Now I'm not saying that having the source code for a site is useless or that having it exposed is a good thing (it's not!). If there is a bug/hole you could find it, but if there isn't then the source being available should not make anything less secure. > So, someone could have used these keys to have much wider access! SSH keys and API tokens are another story. These need to be revoked immediately (CircleCI's response looks great by the way) as there would be nothing besides firewall rules (if they exist!) preventing an attacker from using them. This whole incident is a friendly reminder to follow the principle of least privilege[1] with your XaaS providers. There's not much you can do for API tokens. If you're relying on someone else to provide a service on your behalf then they'll need their own token. For data though it's possible to encrypt sensitive data on your end and use the DBaaS provider as an opaque blob store. Doing it across the board gets a bit unwieldly as it's hard to query encrypted data but it's fine sensitive details that you don't want to necessarily expose (such as ENV vars, payment info or equivalent tokens). If the provider doesn't need access to X, and X is sensitive, then encrypt it before you save it with them. That way you don't have to worry about them leaking it. One of the best examples of this approach is tarsnap[2] which considers everything sensitive and encrypts everything client side (on your machine) before uploading to its central server (and then S3). EDIT: One more quick point. If you explicitly encrypt sensitive data as part of storing it (regardless of whether you're using an XaaS provider or running a DB yourself) there's one more big advantage: Backups! Your backups themselves should be encrypted (ex: we use gpg prior to uploading them to S3) but in addition to that by following the above, the sensitive fields in the backup are also encrypted. This means that they're encrypted in any scratch/temp files that are part of the backup generation process. Otherwise they would be stored in plaintext on your server and possibly be on your server's hard disk (or your XaaS providers disks) without you even knowing it. Oh and remember that disk blocks don't actually get overwritten when you "rm" a file so this is more important than you think. [1]: http://en.wikipedia.org/wiki/Principle_of_least_privilege http://en.wikipedia.org/wiki/Principle_of_least_privilege [2]: http://www.tarsnap.com/ http://www.tarsnap.com/

tkiley 13y ago

Now might be a good time for circleci customers to pick up a code analysis tool such as brakeman and look for obvious security holes in their apps. If the attackers have hundreds or thousands of web apps' source code, I'd expect them to start trawling for vulnerabilities with automated tools.

damncabbage 13y ago

What happened between the 19th and the 28th? https://news.ycombinator.com/item?id=6638004 https://news.ycombinator.com/item?id=6638004 Hi Justin. To clarify, from what I understand, October 28 is the date MongoHQ detected this. They've provided us with the logs of database access, and unfortunately the queries leading to our spam attack on Saturday started as early as October 19.