5 ms·

I once pushed my Amazon S3 key to GitHub accidentally. Realized instantly what I'd done, and while in the process of feverishly regenerating a new key, my cell

I once pushed my Amazon S3 key to GitHub accidentally. Realized instantly what I'd done, and while in the process of feverishly regenerating a new key, my cell phone rings. It's Amazon telling me I pushed my S3 key to GH.

libeclipse 10y ago
Whoa, that's actually amazing. Wonder how they got alerted and reacted so fast.
- ellisv 10y ago
  Alexa probably overhead the developer swearing…
- jsjohnst 10y ago
  I heard AWS has a crawler for that specifically. Not sure if it's true, but makes sense based on the anecdata.
- ecnahc515 10y ago
  It's cheaper for them to give a few engineers a web crawler project that's this specific than it is to refund people. Im just surprised they don't have an "auto revoke access key if found on interwebz" setting in the AWS account settings actually.
  - ygjb-dupe 10y ago
    It's not surprising, consider the failure modes: - a key is made public, and we have to call a user or refund them (for retention purposes) - a key is made public, and we revoked the key, potentially breaking the customers builds/deploys and potentially knocking a customers stuff out (if, for example, a key is disabled during a push to production).
- dsp1234 10y ago
  Github provides a public firehose for events[0]. So it's possible to hook a process to read from the firehose, and look for commit events and then match file contents against the list of API keys. [0] - https://developer.github.com/v3/activity/events/ https://developer.github.com/v3/activity/events/
  - i336_ 10y ago
    Yikes. So this is where the evil crawlers are sitting. Reminds me of the water pipeline in Finding Nemo with the crabs above it.
stephenr 10y ago
You pushed your secret key, and they recognised it? Does that imply that they are not hashing secret keys, or did you also push the account key (allowing for a single auth test on their side)?
- hughes 10y ago
  The secret key can probably be used to generate the account key.
- pyre 10y ago
  It's also possible that they just scrape the Github firehose for common patterns like AWS_SECRET_KEY="FOOBAR" and send a message to the committer's email (since you presumably used a correct/valid email in the git commit).
jfroma 10y ago
It happens to us as well. The interesting thing is that there is also an evil crawler that will automatically launch thousands of windows vms to mine bitcoins (that's all they do). Amazon told us that we have leaked our account id and secret but also they notice the other crawler has launched a lot of VMs and they did a refund to us. yes, we love amazon. Lesson learned: you never put the account id and secret in your code, not only that you should not hardcode it, but there is no need to even read that from the environment etc. Don't do something like this `new S3({accountKey: ..., accountSecret: ..}` instead you do `new S3()` and that's it. Every AWS SDKS is smart enough to find the keys in the environment following a series of steps: - environment variables - ~/.aws/credentials - and when your code is run on ec2, lambda, etc. you should use IAM Roles. So, in addition to not hardcoding an AWS secret, your code should not even pass the secret to the SDK. Consider also enabling CloudTrail and have alerts on that. There is also a way to not have ~/.aws/credentials in your machine and have another thing that requires MFA. I am not familiar how this work yet but we started to use it.

libeclipse 10y ago

Whoa, that's actually amazing. Wonder how they got alerted and reacted so fast.

ellisv 10y ago
Alexa probably overhead the developer swearing…
jsjohnst 10y ago
I heard AWS has a crawler for that specifically. Not sure if it's true, but makes sense based on the anecdata.
ecnahc515 10y ago
It's cheaper for them to give a few engineers a web crawler project that's this specific than it is to refund people. Im just surprised they don't have an "auto revoke access key if found on interwebz" setting in the AWS account settings actually.
- ygjb-dupe 10y ago
  It's not surprising, consider the failure modes: - a key is made public, and we have to call a user or refund them (for retention purposes) - a key is made public, and we revoked the key, potentially breaking the customers builds/deploys and potentially knocking a customers stuff out (if, for example, a key is disabled during a push to production).
dsp1234 10y ago
Github provides a public firehose for events[0]. So it's possible to hook a process to read from the firehose, and look for commit events and then match file contents against the list of API keys. [0] - https://developer.github.com/v3/activity/events/ https://developer.github.com/v3/activity/events/
- i336_ 10y ago
  Yikes. So this is where the evil crawlers are sitting. Reminds me of the water pipeline in Finding Nemo with the crabs above it.

stephenr 10y ago

You pushed your secret key, and they recognised it? Does that imply that they are not hashing secret keys, or did you also push the account key (allowing for a single auth test on their side)?

hughes 10y ago
The secret key can probably be used to generate the account key.
pyre 10y ago
It's also possible that they just scrape the Github firehose for common patterns like AWS_SECRET_KEY="FOOBAR" and send a message to the committer's email (since you presumably used a correct/valid email in the git commit).

jfroma 10y ago

It happens to us as well. The interesting thing is that there is also an evil crawler that will automatically launch thousands of windows vms to mine bitcoins (that's all they do). Amazon told us that we have leaked our account id and secret but also they notice the other crawler has launched a lot of VMs and they did a refund to us. yes, we love amazon. Lesson learned: you never put the account id and secret in your code, not only that you should not hardcode it, but there is no need to even read that from the environment etc. Don't do something like this `new S3({accountKey: ..., accountSecret: ..}` instead you do `new S3()` and that's it. Every AWS SDKS is smart enough to find the keys in the environment following a series of steps: - environment variables - ~/.aws/credentials - and when your code is run on ec2, lambda, etc. you should use IAM Roles. So, in addition to not hardcoding an AWS secret, your code should not even pass the secret to the SDK. Consider also enabling CloudTrail and have alerts on that. There is also a way to not have ~/.aws/credentials in your machine and have another thing that requires MFA. I am not familiar how this work yet but we started to use it.