5 ms·
This is becoming such a common story that I'm starting to think it's a little irresponsible of both Amazon and GitHub to not react better to it. Clearly Amazon
by Analemma_ 11y ago
This is becoming such a common story that I'm starting to think it's a little irresponsible of both Amazon and GitHub to not react better to it. Clearly Amazon is capable of detecting when this happens as evidenced by their rapid response; when they see AWS keys published to GitHub, they should block the account, or at least prevent it from spinning up any new instances. They can then contact the owner over e-mail to follow up.
Same thing with GitHub. When they see an AWS key, they need to block that push and send back a message saying "If you're really, really, REALLY sure you want to do this, add .allowawskey at the root of your commit and push again". There's no excuse for letting people shoot themselves in the foot like this over and over.
(Edited for typo)
- vacri 11y agoAWS do do things. They have service limits and routinely waive charges from abuse events like this. I do wonder if GitHub's events firehose has benefits that outweigh all the compromised credentials, though. Not just limited to AWS keys.
- Analemma_ 11y ago> They have service limits and routinely waive charges from abuse events like this. That's almost worse though. Maybe I'm being unfair here, but the fact that they will waive charges immediately if you ask, but don't just block them pre-emptively, tells me that they're hoping at least some customers (with deep pockets and automated billing that no one pays attention to) will just pay the bill and never notice. Maybe they're making more money from that than they're losing by waiving charges on compromised accounts for customers who ask.
- slg 11y agoDevils advocate, wouldn't you rather have illegitimate charges waived after the fact that legitimate charges declined because their algorithm threw up a red flag? Something like being posted to HN could cause a huge spike in charges for your account in a similar way as leaking your keys and it would be beyond frustrating to see that opportunity disappear because Amazon saw your bill spike.
- inopinatus 11y agoDon't be ridiculous. AWS want legitimate revenue, not brand damage.
- kylec 11y agoThis is a really good point. There was another post recently linking to a search for id_rsa files (SSH private keys) on GitHub. Instead of allowing people to publish sensitive things like that accidentally, GitHub really should require an extra confirmation.
- tomschlick 11y ago> or at least prevent it from spinning up any new instances This is the answer I would like to see implemented. If they can detect the key they can put a freeze on the account so that only existing resources are available. They should also have a timeline of all created instances in every zone so you can see what has happened in the last x hours and kill all instances at once.
- voltagex_ 11y agoTwo things would make controlling these situations easier: * Allow me to see all instances in all regions from a single screen * When accounts are first created for AWS, set up some sane default IAM policies and/or hire some UI designers to fix that godawful mess.
- Alupis 11y agoIs it really on the web service companies to make sure you (the developer) don't do anything stupid? The article author could have avoided a lot of headache if he simply checked the repo before walking away. Red flags should have gone up as soon as he received the first weird Amazon messages, but he still didn't think to check his own work. The author calls this a "Data Breach" but in reality it was a "Data Leak".
- RussianCow 11y agoAmazon would save some money by doing this, since they usually reimburse "victims" of public AWS keys (at least on the first instance) and eat the cost.
- spdionis 11y agoThey don't "reimburse" anything. They only cancel the due amount. The only thing they lose is their wasted computing power (and some support costs) and I think the RAW computing power is very cheap to them, especially considering their scale. The cost in terms of support for the victims is probably bigger but still not significant.
- RussianCow 11y agoIt's still a cost that could be mitigated with a couple hours of someone's time.
- mhuffman 11y agoMaybe not, but it think it is the web service company's responsibility to alert and contact you on suspicious activity ... like a sudden $6,000 jump in service usage of spot instances in regions you have never used before. I mean a quick automated email would be enough for you to take action. The way it is set up now, you pay for it and humbly beg Amazon to deduct those charges if they see fit.
- Alupis 11y ago
- basch 11y agoobviously someone is good at detecting it considering 6.5k in usage got racked up with data scraped within 10 minutes of commit
- dsp1234 11y agoThe quick bitcoin mining spin-ups, and the initial email from Amazon about the key going public, are (probably) thanks to the github events firehose. https://github.com/hootware/github-firehose-node https://github.com/hootware/github-firehose-node http://blog.scalyr.com/2013/10/exploring-the-github-firehose/ http://blog.scalyr.com/2013/10/exploring-the-github-firehose...
- deleted 11y ago[deleted]
- chrismcb 11y agoIt sounds like it happened the next day, after he changed the keys.
- tomphoolery 11y ago"letting people shoot themselves in the foot like this over and over" Stop doing it in the first place, problem solved. I don't see why GitHub or Amazon need to hold my hand through a serious process. It would diminish the importance of what you're doing. Also, this guy didn't just use AWS keys...he clearly used AWS master keys...which has not been the recommended course of action for AT LEAST THREE YEARS...if he had used a proper IAM user with only the access he needed, this wouldn't have happened. Sorry, but that was just irresponsible.
- s73v3r 11y ago"I don't see why GitHub or Amazon need to hold my hand through a serious process. It would diminish the importance of what you're doing." No it wouldn't, not at all. Not unless you're one of those, "REAL MEN us {$editor}" people, in which case no one cares about your opinion.
- einrealist 11y agoIt would probably be easy for Amazon to offer an automatic key blacklisting feature....
- teuna 11y agoNatural selection at work, do not disturb.
- inopinatus 11y agoI wonder if you actually read the story. The author displays incompetence in technology administration at multiple steps. From committing the basic design error of using hard-coded credentials at all; to committing them to any kind of repository; to failing to check on the visibility of that repository; to giving those credentials privileges they did not need; to failing to act effectively on warning messages. And all this, apparently, is laid at the feet of Visual Studio. I have no love of the Microsoft ecosystem, but the author does not acquit themselves well in this incident. The arrogant claims of seniority in the opening paragraph seem ironic in this light.
- LeonB 11y ago> all this, apparently, is laid at the feet of Visual Studio I wonder if you actually read the story. He admits several faults including most if not all of those you mention.
- inopinatus 11y agoSo what? Admitting fault isn't absolution. He really does go out of his way to blame the tools.
- schrodinger 11y agoThe tool has a huge bug! Even if he wasn't putting credentials in there, it could still have led to sensitive source code getting out or something!
- x0n 11y agoArticle updated - VS wasn't the issue. He had installed a 3rd party github extension (from github themselves) that overrides the built-in Git support in visual studio... bug was found, confirmed and fixed yesterday though, so a quick turnaround.
- hamami 11y agoExposing an AWS key through a public repo happened to me a few years ago and AWS support called me to confirm whether I was aware of the suspicious activity, when I stated I do not know what's going on they immediately blocked the account and told me I wouldn't be responsible for the cost. Mine was actually much worse, the account was billing a thousand dollars every hour. They told me I'd have to sign up again with a different account.. I'm surprised this process was not followed in author's case.
- cwisecarver 11y agoYou really want GitHub analyzing your code? REALLY? If the internet found out they were analyzing every push the backlash would be 10x what it is now. Even if it's public... That's still a Gmail-esque situation that would immediately raise privacy concerns. Developers should pay attention to what they're doing. Your AWS keys shouldn't even been in your private repo. You should find another way or you shouldn't be using a third-party for source control.
- bhandziuk 11y agoLooks like someone is analyzing it anyway. May as well be to your benefit.