7 ms·

About three weeks ago I started working on a new open source project meant to interact with AWS. Coding fast and dumb, I cut and pasted my personal AWS credenti

About three weeks ago I started working on a new open source project meant to interact with AWS. Coding fast and dumb, I cut and pasted my personal AWS credentials into my source code, committed it, and pushed it to Github. The next day I got an email from Amazon, alerting me to the problem. Apparently, they scrape github looking for just that kind of stupidity. I instantly deleted the project, but it was too late. Amazon ended up waving the nearly $3k in EC2 charges I incurred, thankfully. I'm now a zealous advocate for making sure a person never even HAS AWS credentials. Instead, make a new user without a password for each use case, and manually select the privileges that account has. If you have a password to AWS, you shouldn't have credentials.

e40 11y ago
I posted to the AWS forum and accidentally copy/pasted my secret key. Within 24 hours $11k of charges. I called them, they wiped them. It's amazing how quickly people find and use these things. What kills me is there is no easy way to stop all the instances for an account, in a region. It took me hours to kill all the instances. They had maxed out the number of instances in every single region. Very, very annoying.
- pests 11y ago
  You can automate the stopping via the CLI interface. While I don't think there is a single command to stop all instances, you should be able to whip up a script to get all id's then call the stop-instances command on each of them.
- agumonkey 11y ago
  You must lose a fair amount of water when you're greeted with a surprise bill of $11k.
  - e40 11y ago
    Yeah, it's a rare moment in life. Especially fun since it happened on a Saturday morning (the notification, that is).
- copperx 11y ago
  When people find the accounts, what do they use them for? Mining bitcoin?
- mrestko 11y ago
  Any idea what they were doing with them? Mining bitcoins? Hosting CnC for a botnet?
  - fpgaminer 11y ago
    It's always Bitcoin mining. The attacker spins up a bunch of GPU EC2 instances and mines as long as they can. I don't think the profit ends up being very large (those GPU instances put out a puny amount of hashing power compared to modern mining ASICs)...
    - icelancer 11y ago
      ROI for BTC mining on standard AWS pricing is about -90%. So attackers get 10% free BTC on spend.
      - tajen 11y ago
        So, out of $11k, they make $1k. That's the kind of money they need to make between once and three times a month minimum depending on where they live. Doesn't Amazon notice patterns in terms of source, scripts which are uploaded and scaling profile: Who uses 2 medium instances for a year then spins off 2,000 in 20 minutes?
        icelancer 11y ago
        Certainly there's a pattern there, but it's not THAT far away from people who intermittantly scale stuff for short bursts of huge processing. False positives in those cases for people that really intended to spend $10-50k in a short time might mean a HUGE loss of revenue and/or incurred customer support and service costs.
    - Taek 11y ago
      It's not that bad. The smart attackers don't do Bitcoin mining, they do altcoin mining, and they pick altcoins that don't have ASICs out. Then they are competing with other GPUs. I'm not sure what the profit is, but I'd guess it's between 30% and 70% of the bill incurred.
aikah 11y ago
> Coding fast and dumb, And that's why you should use Bibucket first and foremost. And then when you vet your stuff and you know it's clean, you can publish it on publicly Github. If you rush pushing code on github public repos that's exactly what will happen. If you start with a private one then move to a public one you have more time to think things through. Now one could imagine a third party service warning users of potential issues with files pushed,based on their name/extension/folder name. But privacy first , then open the code to the public.
- bosdev 11y ago
  You could just pay for GitHub to get private repos. If it's something this important to you, it's probably worth a few bucks a month. This solution is also strange because if you ever committed anything private it will be in the history. So to make this work you either have to rebase over some history, or lose all the history.
  - zippergz 11y ago
    I have a lot of private repos. One for just about every little project I've done. Some larger projects have more than one. It would be pretty expensive to keep all of this in Github that way. The alternative is to combine a bunch of unrelated stuff into fewer repos (or not keep all of my projects in source control), neither of which is especially appealing. I'd be happy to pay Github something for my usage and to support development. I believe in paying people for work that I find useful. But the pricing model makes the cost disproportionate with the value for me.
  - eropple 11y ago
    > You could just pay for GitHub to get private repos. If it's something this important to you, it's probably worth a few bucks a month. Github doesn't have a personal tier big enough for my private repos. (And I'd host my own before paying $50/month.) I do pay Bitbucket, though, because I have a consistent and small-enough group of collaborators that it makes sense.
  - belovedeagle 11y ago
0x0 11y ago
Github provides a public near-realtime stream of events, I'm sure blackhats are constantly monitoring it looking for private keys all the time. Even if you undo a commit within seconds, it may be too late. https://api.github.com/events https://api.github.com/events
- meddlepal 11y ago
  I did not know that existed and that is pretty awesome!
- unethical_ban 11y ago
  If Amazon (and my coworker in a 3 person dev shop) can automate the scanning of common API key patterns as a pre-commit filter, I wonder why Github itself doesn't flag for this sort of thing before publishing.
  - giaour 11y ago
    IIRC, a regex that matches a generic AWS access key will also match a git hash.
    - giaour 11y ago
      Not sure what I said that warrants multiple downvotes. Git commits are identified by SHA1 hashes, so they would be caught by the same regex that would catch AWS keys. According to the AWS security blog, this is the regex you should use for secret keys: `(?<![A-Za-z0-9/+=])[A-Za-z0-9/+=]{40}(?![A-Za-z0-9/+=])`. To match git hashes, you can use this one: `[0-9a-f]{40}`. See the likeness?
  - IanCal 11y ago
    What should they do? Reject a "git push"?
    - geofft 11y ago
      That's fairly reasonable. At least they can prompt you "Did you really mean to do this?" -- they have enough infrastructure to update your project page with a "Make pull request" button after you push, they can repurpose that for a "Approve next push without asking questions." Alternatively, if they supported custom pre-receive hooks via e.g. webhooks, someone could publish a script or stand up a web server to check for such things.
scott_karana 11y ago
> Amazon ended up waving the nearly $3k in EC2 charges I incurred, thankfully For what it's worth, the word is "waived". :-) I'm glad Amazon dealt with you well!
nadams 11y ago
> I cut and pasted my personal AWS credentials into my source code, committed it, and pushed it to Github. This paradigm is very puzzling to me. Why do people feel the need to publish every small project to the public? Is it because github is so easy to create a new repo? Why don't more people use private repos like self hosted or bitbucket? The whole reason why I spawned my own source code hosting service is so that I can work on projects in private without worrying about random people looking at what I'm working on (some of my C++ projects would give even Stroustrup a heart attack...). Note - this isn't saying bitbucket or gitlab or any of the others aren't good (oh I have my own opinions and comments about them...) - I've become rather bitter/paranoid/resentment of offers of free hosting.
- akerl_ 11y ago
  I'm surprised that of the quoted comment you focus on the "paradigm" of open sourcing code, and not the "paradigm" of mixing credentials with code. Trying to protect against leaking creds by not open sourcing is a bit of misdirected effort.
  - nadams 11y ago
    I pointed this out because I've seen this over and over again with the single theme of "I was working on some silly project using AWS and I pushed my credentials to github". I don't think I've ever seen a professional team push AWS credentials (or any other credentials for that matter) - and if they did it's very rare to the point where I don't remember. As far as mixing credentials with code - that happens all the time. I'm not saying it's right but I have much bigger concerns - such as why people keep silly projects to github with their credentials. The first time I saw an article about it - I found out that there are bad people who are monitoring github for exactly that - and they will use it even before you realize it what you did. And not just AWS credentials mind you - MySQL, SSH, anything they can get their hands on... I think my comments are getting lost in translation - I'm not saying I'm against people pushing their silly projects. Push your 1 line GPL code projects all day long - I don't care. That's why github is there. What I find puzzling is not that people want to create new projects - but they feel like they must create a git repo for EVERYTHING they do and push it up to a public github like it's going to change the face of computer science as we know it or something. I've seen people keep documents and even bash profiles as public repos. I'm not saying it isn't a good idea to use git for those purposes - but I wouldn't want my bash profile common knowledge (especially if it contains commands or functions that I use at my day job).
voltagex_ 11y ago
I'm as close to a "casual" user of AWS as you can get. I find IAM incredibly difficult to use. Any pointers?
joshuafalken_ 11y ago
I created a small tool[1] to help continuously audit public github commits for secrets, like aws keys. It uses the AWS provided regexes[2] to do to. [1] - https://github.com/jfalken/github_commit_crawler https://github.com/jfalken/github_commit_crawler [2] - http://blogs.aws.amazon.com/security/blog/tag/key+rotation http://blogs.aws.amazon.com/security/blog/tag/key+rotation
snorkel 11y ago
The real mistake here is using your account owner API key in your code, when you shouldn't use that key for anything ... In fact you should delete the account owner access key!!! Otherwise you may have used the key for a Power User or Administrator level IAM user in your code instead of a custom user with least privileges. Always use least privileged IAM keys in code if you must, preferably use EC2 roles instead of access keys, and never give your IAM users permission to run instances ... That way if the key is leaked then evil hacker can't do much with it.
- jebblue 11y ago
  I agree. I haven't used AWS for a while but IIRC for a small side project I was looking into with a couple of friends I set up a low privilege IAM access for them but I logged directly into the VPS using a password if I needed to update something.

e40 11y ago

I posted to the AWS forum and accidentally copy/pasted my secret key. Within 24 hours $11k of charges. I called them, they wiped them. It's amazing how quickly people find and use these things. What kills me is there is no easy way to stop all the instances for an account, in a region. It took me hours to kill all the instances. They had maxed out the number of instances in every single region. Very, very annoying.

pests 11y ago
You can automate the stopping via the CLI interface. While I don't think there is a single command to stop all instances, you should be able to whip up a script to get all id's then call the stop-instances command on each of them.
agumonkey 11y ago
You must lose a fair amount of water when you're greeted with a surprise bill of $11k.
- e40 11y ago
  Yeah, it's a rare moment in life. Especially fun since it happened on a Saturday morning (the notification, that is).
copperx 11y ago
When people find the accounts, what do they use them for? Mining bitcoin?
mrestko 11y ago
Any idea what they were doing with them? Mining bitcoins? Hosting CnC for a botnet?
- fpgaminer 11y ago
  It's always Bitcoin mining. The attacker spins up a bunch of GPU EC2 instances and mines as long as they can. I don't think the profit ends up being very large (those GPU instances put out a puny amount of hashing power compared to modern mining ASICs)...
  - icelancer 11y ago
    ROI for BTC mining on standard AWS pricing is about -90%. So attackers get 10% free BTC on spend.
    - tajen 11y ago
      So, out of $11k, they make $1k. That's the kind of money they need to make between once and three times a month minimum depending on where they live. Doesn't Amazon notice patterns in terms of source, scripts which are uploaded and scaling profile: Who uses 2 medium instances for a year then spins off 2,000 in 20 minutes?
      - icelancer 11y ago
        Certainly there's a pattern there, but it's not THAT far away from people who intermittantly scale stuff for short bursts of huge processing. False positives in those cases for people that really intended to spend $10-50k in a short time might mean a HUGE loss of revenue and/or incurred customer support and service costs.
  - Taek 11y ago
    It's not that bad. The smart attackers don't do Bitcoin mining, they do altcoin mining, and they pick altcoins that don't have ASICs out. Then they are competing with other GPUs. I'm not sure what the profit is, but I'd guess it's between 30% and 70% of the bill incurred.

aikah 11y ago

> Coding fast and dumb, And that's why you should use Bibucket first and foremost. And then when you vet your stuff and you know it's clean, you can publish it on publicly Github. If you rush pushing code on github public repos that's exactly what will happen. If you start with a private one then move to a public one you have more time to think things through. Now one could imagine a third party service warning users of potential issues with files pushed,based on their name/extension/folder name. But privacy first , then open the code to the public.

bosdev 11y ago
You could just pay for GitHub to get private repos. If it's something this important to you, it's probably worth a few bucks a month. This solution is also strange because if you ever committed anything private it will be in the history. So to make this work you either have to rebase over some history, or lose all the history.
- zippergz 11y ago
  I have a lot of private repos. One for just about every little project I've done. Some larger projects have more than one. It would be pretty expensive to keep all of this in Github that way. The alternative is to combine a bunch of unrelated stuff into fewer repos (or not keep all of my projects in source control), neither of which is especially appealing. I'd be happy to pay Github something for my usage and to support development. I believe in paying people for work that I find useful. But the pricing model makes the cost disproportionate with the value for me.
- eropple 11y ago
  > You could just pay for GitHub to get private repos. If it's something this important to you, it's probably worth a few bucks a month. Github doesn't have a personal tier big enough for my private repos. (And I'd host my own before paying $50/month.) I do pay Bitbucket, though, because I have a consistent and small-enough group of collaborators that it makes sense.
- belovedeagle 11y ago

0x0 11y ago

Github provides a public near-realtime stream of events, I'm sure blackhats are constantly monitoring it looking for private keys all the time. Even if you undo a commit within seconds, it may be too late. https://api.github.com/events https://api.github.com/events

meddlepal 11y ago
I did not know that existed and that is pretty awesome!
unethical_ban 11y ago
If Amazon (and my coworker in a 3 person dev shop) can automate the scanning of common API key patterns as a pre-commit filter, I wonder why Github itself doesn't flag for this sort of thing before publishing.
- giaour 11y ago
  IIRC, a regex that matches a generic AWS access key will also match a git hash.
  - giaour 11y ago
    Not sure what I said that warrants multiple downvotes. Git commits are identified by SHA1 hashes, so they would be caught by the same regex that would catch AWS keys. According to the AWS security blog, this is the regex you should use for secret keys: `(?<![A-Za-z0-9/+=])[A-Za-z0-9/+=]{40}(?![A-Za-z0-9/+=])`. To match git hashes, you can use this one: `[0-9a-f]{40}`. See the likeness?
- IanCal 11y ago
  What should they do? Reject a "git push"?
  - geofft 11y ago
    That's fairly reasonable. At least they can prompt you "Did you really mean to do this?" -- they have enough infrastructure to update your project page with a "Make pull request" button after you push, they can repurpose that for a "Approve next push without asking questions." Alternatively, if they supported custom pre-receive hooks via e.g. webhooks, someone could publish a script or stand up a web server to check for such things.

scott_karana 11y ago

> Amazon ended up waving the nearly $3k in EC2 charges I incurred, thankfully For what it's worth, the word is "waived". :-) I'm glad Amazon dealt with you well!

nadams 11y ago

> I cut and pasted my personal AWS credentials into my source code, committed it, and pushed it to Github. This paradigm is very puzzling to me. Why do people feel the need to publish every small project to the public? Is it because github is so easy to create a new repo? Why don't more people use private repos like self hosted or bitbucket? The whole reason why I spawned my own source code hosting service is so that I can work on projects in private without worrying about random people looking at what I'm working on (some of my C++ projects would give even Stroustrup a heart attack...). Note - this isn't saying bitbucket or gitlab or any of the others aren't good (oh I have my own opinions and comments about them...) - I've become rather bitter/paranoid/resentment of offers of free hosting.

akerl_ 11y ago
I'm surprised that of the quoted comment you focus on the "paradigm" of open sourcing code, and not the "paradigm" of mixing credentials with code. Trying to protect against leaking creds by not open sourcing is a bit of misdirected effort.
- nadams 11y ago
  I pointed this out because I've seen this over and over again with the single theme of "I was working on some silly project using AWS and I pushed my credentials to github". I don't think I've ever seen a professional team push AWS credentials (or any other credentials for that matter) - and if they did it's very rare to the point where I don't remember. As far as mixing credentials with code - that happens all the time. I'm not saying it's right but I have much bigger concerns - such as why people keep silly projects to github with their credentials. The first time I saw an article about it - I found out that there are bad people who are monitoring github for exactly that - and they will use it even before you realize it what you did. And not just AWS credentials mind you - MySQL, SSH, anything they can get their hands on... I think my comments are getting lost in translation - I'm not saying I'm against people pushing their silly projects. Push your 1 line GPL code projects all day long - I don't care. That's why github is there. What I find puzzling is not that people want to create new projects - but they feel like they must create a git repo for EVERYTHING they do and push it up to a public github like it's going to change the face of computer science as we know it or something. I've seen people keep documents and even bash profiles as public repos. I'm not saying it isn't a good idea to use git for those purposes - but I wouldn't want my bash profile common knowledge (especially if it contains commands or functions that I use at my day job).

voltagex_ 11y ago

I'm as close to a "casual" user of AWS as you can get. I find IAM incredibly difficult to use. Any pointers?

joshuafalken_ 11y ago

I created a small tool[1] to help continuously audit public github commits for secrets, like aws keys. It uses the AWS provided regexes[2] to do to. [1] - https://github.com/jfalken/github_commit_crawler https://github.com/jfalken/github_commit_crawler [2] - http://blogs.aws.amazon.com/security/blog/tag/key+rotation http://blogs.aws.amazon.com/security/blog/tag/key+rotation

snorkel 11y ago

The real mistake here is using your account owner API key in your code, when you shouldn't use that key for anything ... In fact you should delete the account owner access key!!! Otherwise you may have used the key for a Power User or Administrator level IAM user in your code instead of a custom user with least privileges. Always use least privileged IAM keys in code if you must, preferably use EC2 roles instead of access keys, and never give your IAM users permission to run instances ... That way if the key is leaked then evil hacker can't do much with it.

jebblue 11y ago
I agree. I haven't used AWS for a while but IIRC for a small side project I was looking into with a couple of friends I set up a low privilege IAM access for them but I logged directly into the VPS using a password if I needed to update something.