7 ms·
How a bug in VS2015 exposed my source code on GitHub and cost me $6,500
- ocdtrekkie 11y agoI still find it amazing that AWS doesn't offer any way to hardcap your billing and cut off service.
- mbrutsch 11y agoThere's really no financial incentive to do so.
- tomschlick 11y agoThere is when they keep refunding people for having their account hijacked and on the other end are giving away CPU resources to the bitcoin miners.
- ohashi 11y agoHere's a hypothetical, what is the real cost, he said they were using spot instances too. A huge spike in demand might have an effect on price, pushing the cost up for all legitimate users. As far as the reserved instances, they were there, not being used presumably. They don't really lose anything except possibly some electricity and bandwidth costs?
- travoc 11y agoWhen people run up a huge bill and don't pay it, that's an incentive.
- toomuchtodo 11y agoThe profit margin on AWS is stupid high. It costs Amazon nothing but power and opportunity cost to refund unintended use.
- choward 11y agoWhat about the potential customers they lose?
- JorgeGT 11y agoSide question: is there any similar cloud service with hard billing caps?
- ocdtrekkie 11y agoI'd really love to know. Because if I could get AWS-like service with the ability to have it cut off if my bill gets to say... three times my usual monthly... That'd be worth switching for the peace of mind.
- mahouse 11y agoAre we really going to run this story every month? "I posted my private AWS keys by mistake on github, a spider got them, I got a huge bill" Yeah, you don't say
- TimothyFitz 11y agoHe didn't post it by mistake. He asked Visual Studio to create a private repo and it created a public one. A bug which he reproduced.
- williamcotton 11y agoSure, but he shouldn't have checked keys in to any repo, public or private!
- morder 11y agoIt wasn't even visual studio but an extension created by github that caused the issue. Visual studio does not come with github integration by default
- tonywebster 11y agoDon't be so quick to dismiss this one. This is notable for several reasons, including (1) it wasn't his mistake, but rather that of a Visual Studio bug not following his instructions to make a private repo on GitHub; (2) the speed in which this happened (minutes), and (3) it has useful analysis into some sorely lacking functionality in AWS that lets this continue to happen.
- nathancahill 11y agoI don't know. Even for private repos, it's bad practice to commit private keys to source control.
- zeveb 11y ago> Even for private repos, it's bad practice to commit private keys to source control. Is it, though? Committing them to the same repository as one's code lives in, sure, but committing them to a separate production-deploy repo seems okay to me (although I'd much prefer that private repo never to hit a centralised service like GitHub).
- Analemma_ 11y agoThis is becoming such a common story that I'm starting to think it's a little irresponsible of both Amazon and GitHub to not react better to it. Clearly Amazon is capable of detecting when this happens as evidenced by their rapid response; when they see AWS keys published to GitHub, they should block the account, or at least prevent it from spinning up any new instances. They can then contact the owner over e-mail to follow up. Same thing with GitHub. When they see an AWS key, they need to block that push and send back a message saying "If you're really, really, REALLY sure you want to do this, add .allowawskey at the root of your commit and push again". There's no excuse for letting people shoot themselves in the foot like this over and over. (Edited for typo)
- vacri 11y agoAWS do do things. They have service limits and routinely waive charges from abuse events like this. I do wonder if GitHub's events firehose has benefits that outweigh all the compromised credentials, though. Not just limited to AWS keys.
- Analemma_ 11y ago> They have service limits and routinely waive charges from abuse events like this. That's almost worse though. Maybe I'm being unfair here, but the fact that they will waive charges immediately if you ask, but don't just block them pre-emptively, tells me that they're hoping at least some customers (with deep pockets and automated billing that no one pays attention to) will just pay the bill and never notice. Maybe they're making more money from that than they're losing by waiving charges on compromised accounts for customers who ask.
- slg 11y agoDevils advocate, wouldn't you rather have illegitimate charges waived after the fact that legitimate charges declined because their algorithm threw up a red flag? Something like being posted to HN could cause a huge spike in charges for your account in a similar way as leaking your keys and it would be beyond frustrating to see that opportunity disappear because Amazon saw your bill spike.
- famousactress 11y ago"As a senior developer with over 12 years of experience in the financial services industry, I didn't think it was possible that I could be the victim of a data breach." We're only one sentence in and I'm already very, very nervous about the author's hubris.
- forrestthewoods 11y agoI took it less as hubris and more of a way to imply to readers that no matter their experience they are not immune to the mistake he is going to tell you about.
- merlincorey 11y agoI think it's definitely some hubris, because the mistake he made wasn't publishing to github publicly by accident. It was thinking a git repository was a good place to store secrets.
- merlincorey 11y agoThankfully, the author realizes later what it they really did wrong here: > What could be done to prevent this? > Always test new version control GUIs before using them in the wild. There could be a bug that could expose your data. > Encrypt access keys in config files instead of just leaving them exposed in the config file. > Better yet, move access keys to a seperate config file, and exclude this from Git deploys with a .gitignore. The fact this supposedly security experienced and conscious developer thought it was okay to have any kind of access credentials stored in a git repository is glaring. You should consider all data in any repository public for security reasons. Publishing to github publicly made it quite obvious - but he could have been just as screwed over by people he gave access to the private repositories and people who have broken into github and can directly access the repositories. Sure the numbers on both those groups are lower than the numbers of people apparently scanning public github for access keys - but the severity of the risk of keeping access keys and other secrets in your revision control are the same.
- 11y ago
- AdamGibbins 11y agoAmazon are well known for refunding erroneous charges like this, they've done so many times in the past - get in contact and ask for some leeway.
- blablabla123 11y agoMost UIs for programmers still seem to lag behind the command line tools a lot. I wonder if it will be possible again some day to create apps like in Delphi without some black magic happening in the background. P.S.: never ever put keys into repositories. Except those repos are deployment config only.
- slg 11y agoDoes Amazon actually end up charging you when this type of thing happens or do they eventually wipe the bill after you talk to support? If Amazon tried to charge me, I would definitely be in touch with the credit card company I have attached to the account. This is rather straightforward fraud/identity theft so you should be covered by the standard consumer protections most credit card's provide.
- e40 11y agoI accidentally posted AWS credentials to the AWS EC2 forum, and got > $10k (US) of charges very quickly. They removed all the charges immediately, without me pushing at all. They offered it.
- slg 11y agoThat is good to hear, but it also means the headline is a little on the click-baity side. This is a bug that caused a lot of headaches and cost time to fix, not one that cost $6,500.
- d2xdy2 11y ago> As a senior developer with over 12 years of experience in the financial services industry, I didn't think it was possible that I could be the victim of a data breach. Ok... > Better yet, move access keys to a seperate config file, and exclude this from Git deploys with a .gitignore. No shit?
- curun1r 11y agoEven that's a terrible solution. AWS specifically changed their tooling to look for access keys in ~/.aws and make it more difficult to load them from inside a project directory. It's really not that hard to setup MFA and require it when using STS to assume whatever role you need to make changes. For automated changes, you can give instances the necessary an IAM role and only run those from inside EC2. He didn't get a $6.5k bill because of a bug in VisualStudio, he got one because he didn't finish setting up his AWS account. Regardless of the public/private mixup, checking in AWS credentials to any repository is just bad practice and creates a huge risk of the exact situation that happened in this story.
- d2xdy2 11y agoAlbeit not on AWS, I typically just set environment vars to reflect configs I would have set in a .env (or .local.env, or .testing.env), or have some sort of config manager daemon type thing running that keeps that sort of stuff locked away. > He didn't get a $6.5k bill because of a bug in VisualStudio, he got one because he didn't finish setting up his AWS account. Regardless of the public/private mixup, checking in AWS credentials to any repository is just bad practice and creates a huge risk of the exact situation that happened in this story. I agree
- jerrac 11y agoHopefully he posts updates about his unanswered questions. Since it sounds like the compromised key should not have been able to create ec2 instances. And I don't blame him for making that mistake. That kind of bug should have been caught in QA. On the other hand, I'm paranoid enough about things that I wouldn't push unsecured access keys to any server I didn't control. Even if I was paying for a private repo.
- medina 11y agohttp://ghcc.io http://ghcc.io - Tool used to continuously monitor a Github org for mistaken public commits (& lets you monitor repos you care about but might not be able to enforce commit-hooks for).
- hoare 11y agoim impressed how fast a bot fetched the key from github, set up a trillion instances and did what exactly on them? edit: my guess, some kind of crypto currency mining?
- merb 11y agoWhy does somebody pushes AWS Keys into a Git Repository? Why should somebody ever push some Authentications? Currently when I code an app I create a database with name of the app and have new credentials with just name:name so that the only thing that I check into my config if the app needs a application.conf or resource.conf are default username:passwords for development, so that bootstrapping is easier. However EVERYTHING else gets read from the Environment Variables that even Works on Windows since Visual Studio has a Feature for that, but be aware the sln file we keep the values. So however never ever put anything into git / source code repo which shouldn't be read.
- zeveb 11y ago> Why does somebody pushes AWS Keys into a Git Repository? If one wants to use version control to track credentials, then this makes sense. Why would one want to do that? In order to track versions of one's production system. This enables things like one-shot rebuilds. Now, it's obvious that such a production repository should be separate from one's development repo(s). And one could make a decent argument that it should never be pushed someplace like GitHub anyway, even as a private repository.
- balls187 11y ago> Why does somebody pushes AWS Keys into a Git Repository? Wild speculation, but this is a side effect of "developer" centric operations. Everything is done via code, and developers are instinctively trained to check in code.
- apetresc 11y agoThe most important part of this story seems to be glossed over – how did an IAM key that the author insists only had access to the Alexa service get used to spin up EC2 instances? Either the author is just mistaken/lying about that point, or there's another leak he doesn't know about, or there's a serious privilege escalation vulnerability in AWS.
- merlincorey 11y agoSomeone else in a thread above supposes that they actually published amazon master keys, rather than an AWS particular IAM role. The author of the post states they didn't even use AWS prior, so they are most likely not aware of such things.
- frozenport 11y agoMaybe secure keys are a lot secure than passwords?
- DenisM 11y agoIf you're keeping your keys in source control, you're only one mistake away from a disaster. It could be your own mistake, a bug in a VS, an error in GitHub, a mistake by a teammate, anything, really. It's similar to driving your car too close to the car in front - you will get away with it most of the time and get lulled into a false sense of security, but eventually it will catch up with you. Don't be one mistake away from a disaster. Make room for at least two.
- pbreit 11y ago1) Seems like an inherent risk of closed source tools. 2) Someone other than the OP should pickup/waive the tab.
- JamesBarney 11y agoIf he is using the GitHub Extension then it's open source see https://visualstudio.github.com/ https://visualstudio.github.com/
- pbreit 11y agoI think you might be right: https://github.com/github/VisualStudio/commit/eb3f7f254de51af3a1f3d36fc164d6fd7eb2383a https://github.com/github/VisualStudio/commit/eb3f7f254de51a...
- morder 11y agoVisual studio does not integrate with Github. This was a 3rd party extension created by Github that this user installed and used.
- jongalloway2 11y agoI'm fairly certain this is a bug in the GitHub extension for Visual Studio (https://visualstudio.github.com/ https://visualstudio.github.com/), not in Visual Studio itself. Here's how the sync dialog appears in Visual Studio 2015 without the GitHub extension: http://i.imgur.com/EGjYziR.png http://i.imgur.com/EGjYziR.png After installing the GitHub extension: http://i.imgur.com/shRBwXM.png http://i.imgur.com/shRBwXM.png Try it yourself, then complete these fun, but completely optional activities: 1. Re-read all comments on this thread using your best Andy Rooney voice. 2. Wish that the author had comments on his blog.
- jongalloway2 11y agoI logged an issue on the GitHub extension in case that's the problem: https://github.com/github/VisualStudio/issues/62 https://github.com/github/VisualStudio/issues/62
- lightlyused 11y agopre commit hooks could have prevented this: https://gist.github.com/stevendanna/4db970716cabb8f5def4 https://gist.github.com/stevendanna/4db970716cabb8f5def4