20 ms·
Tell HN: Rise in AWS accounts getting hacked and owner being stuck with the bill
I have been seeing a lot of posts on Reddit and other forums of mostly students setting up an AWS account only for them to be hacked and account owner being stuck with a significant bill.
Most likely scenario is hackers are trying leaked username/password pairs from other breaches against AWS and gaining access to those accounts.
They then spin up EC2 instances in all sorts of regions on the compromised accounts
PSA set up MFA on your account if you haven't already.
Some examples:
https://www.reddit.com/r/aws/comments/rv3lm5/i_lost_55k_from_hackers/ https://www.reddit.com/r/aws/comments/rv3lm5/i_lost_55k_from...
https://www.reddit.com/r/aws/comments/rvbncu/account_hacked_unable_to_sign_in_4000_in/ https://www.reddit.com/r/aws/comments/rvbncu/account_hacked_...
https://www.reddit.com/r/aws/comments/qx8i02/got_hacked_and_found_a_30k_bill_please_turn_on/ https://www.reddit.com/r/aws/comments/qx8i02/got_hacked_and_...
https://www.reddit.com/r/aws/comments/rv4mnq/my_account_was_hacked_and_now_my_bill_is_over/ https://www.reddit.com/r/aws/comments/rv4mnq/my_account_was_...
- smackeyacky 5y agoI know it's easy to get lazy about checking your AWS billing dashboard but I do it once a week - you can set up alerts and whatnot but I find it easier just to go look at the current usage to make sure nothing has gone awry.
- digitalsushi 5y agoI think the point is that someone could bankrupt the average hobbyist in an hour or two, if they were expected to pay it.
- danial 5y agoI think that rate of usage would trigger EC2 abuse alerts. Usually it's something that looks plausible but accumulated over 30 days.
- danial 5y agoI agree that you have to make it a daily or weekly habit to check because 30 days is too long a time to incur costs you're not aware of. That's why I built Billgist.com, same idea but it sends a daily email with usage amount right in the subject, like: $16.56 XYZAccount – Daily AWS billing alert.
- andrewguenther 5y agoMFA needs to be forced on by default for all new accounts created with the UI. It is utterly irresponsible for AWS to not do this. People always counter with "you should know better!" But AWS markets itself to college kids. People were all up in arms and Robinhood allowing 18 year olds to create accounts which could result in unbounded losses, but AWS somehow gets a pass?
- vmception 5y agoThis happened to me, the bill was so ridiculous that I wasnt even bothered by it. It got voided by aws support as predicted. MFA does not prevent this. Its IAM keys.
- heavyset_go 5y agoHow big was the bill?
- vmception 5y agoLike $120,000 after paying like $.30 for cloudfront for 10 months It was pretty obviously uncharacteristic
- technion 5y agoThis is an underrated point. Every "best security practices" guide you read has you setup MFA for console access, then create IAM keys with no such protection. The credentials I'm using with Terraform require MFA in order to call 'AssumeAdmin'. Everyone I've ever shown this configuration has complained about it being overkill and tried to argue Terraform should just have IAM keys sitting on disk, one desktop compromise away from taking everything. And since it's used to provision basically everything it's a highly privileged account.
- notemaker 5y agoaws-vault works too
- Aeolun 5y agoI find it hard to believe AWS would try to suck blood from a stone. I’m willing to believe they get a 55k bill, but do they actually end up paying those?
- stevespang 5y ago
- NikolaeVarius 5y agoNo.
- anduru_h 5y agoI think you can get refunded for fraudulent charges, like if you get hacked. But if you leave an instance running or screw up auto-scaling somehow, that's normally when the charges may tend to stick around. I believe they have a one strike policy, though that could have changed.
- jackson1442 5y ago@aws, why not mandate MFA for a root user? in child org accounts where this is less feasible, you could allow access to the root user only from the parent account, no direct login at all.
- smackeyacky 5y agoYou just reminded me to set up MFA on my root account so thanks!
- staticassertion 5y agoOne option is to set your root user's password to some random 64 character string and forget it. Any time you want root access (rare) you go through a reset flow, which means your root auth is tied to your email. Something like GMail has pretty strict controls so this is actually imo the safest option available.
- joombaga 5y agoYou can keep the password unset as well.
- jackson1442 5y agoWhile this may be safest, it doesn’t make sense why Amazon doesn’t save themselves a couple (hundred) grand in refunds by locking down root accounts.
- oneplane 5y agoYou can create rootless organisation-managed child accounts. This is a solved problem.
- andrewguenther 5y agoOrganization child accounts are not rootless, they're just seeded with a random password
- 5y ago
- ujetin 5y agoI deleted my AWS account yesterday. It is obviously catered towards large organisations - very complicated tools and pricing that I couldn't really fit into my use case. I tried to just shut down the services that were using money but wasn't even sure I had found them all so I just closed the whole account. I don't even like the idea of any of this stuff. I want to run my own little raspberry pi server or whatever, it seems much more fun and startupish than aws, which appears to be all of the corporate stuff I left (AIMs etc). This is funny because I remember AWS being thought of as great for "just experimenting with stuff".
- karanbhangui 5y agolove digitalocean for this
- Sebb767 5y agoAWS is great if you have a very large infrastructure budget. Playing around on a test AWS account where thousand dollars plus or minus is no big deal is a lot of fun. Playing around while trying to keep the bill under 10$ is just torture.
- 300bps 5y agoAWS has a free tier that lasts for a year. I actually used it to get hands-on experience with enough services that I was able to get my AWS Certified Solutions Architect, Cloud Practitioner and Certified Developer certifications. I easily spent 60 hours creating and deleting services and it didn't cost me a cent. I don't even know how you'd accidentally get to thousands of dollars accidentally unless you were doing something more complicated like setting up an autoscaling group. Spin up a t3a.micro EC2 instance and run it as much as you like - it's free. If you go into it (as I did many years ago) thinking, "Let me just spin up a 16 core server with 128 GB of RAM" because that's what you'd set up in your on-prem data center then of course you're going to have a big bill. But that's not what cloud is about. It's about resources popping in and out of existence for the minimal time and resources needed to accomplish a task.
- 5y ago
- anothernewdude 5y agoIf only they had some kind of charge limiting on accounts like their customers have been asking for years now.
- spekcular 5y agoThe fact that AWS has no way to limit billing seems insane to me. Your only recourse for an accidental (or malicious) overcharge is beg customer support. It's an incredible liability.
- thejosh 5y agoIf you were "hacked" (ie password reuse), the attacker can just relimit this.
- notreallyserio 5y agoAWS could simply require the user verify they have access to the card or other payment method on file.
- oneplane 5y agoThat is super impractical when you run 200 AWS accounts in production.
- notreallyserio 5y agoDoes AWS require distinct billing accounts for each project? I use GCP, I can attach the same account to multiple projects. (It also lacks this basic billing limit feature, unfortunately.)
- oneplane 5y agoNo, that's why it would be silly to have to change it for every account. (and also problematic for a team with a bunch of child accounts to then have to gain access to the org's card or something like that to change settings for their childs)
- notreallyserio 5y agoI don't follow. But they could make the quota feature a toggle (that has the same mechanic to unlock). Easy, easy enough for a junior dev. And I'm sure folks spending more than a few minutes chatting about it could come up with an even better solution.
- blibble 5y agotheir hardware MFA functionality is worse than useless it only permits a single hardware token to be registered to an account so good luck if you misplace or break your hardware token
- timf 5y agoI did not like that limitation either and get around it by using the Yubico Authenticator app with three separate hardware tokens configured with the same shared TOTP seed. There's also path to reset your root account access, btw.
- halotrope 5y agoWhen setting up the token you can scan the QR with multiple devices. E.g YubiKey and Authenticator App. This at least allows for a backup in case one goes missing. I agree it is kind of incredible that multiple tokens are not supported.
- Sebb767 5y agoYou actually can't mix hardware tokens and OTP apps. You're only option is to scan the code twice and skip hardware tokens entirely (which is quite reasonable, as the recovery for an app would be easier than for a failed/lost hardware token). Note, though, that the new SSO login actually supports MFA in a normal way.
- nefitty 5y agoDon't token limits reduce attack surface?
- aborsy 5y agoYou create several users with admin privileges each with separate MFA.
- jeppesen-io 5y agoThe point is to block unauthorized access. It is not less than useless. It absolutely does that. You really think if your lost your token Amazon would lock you out forever? Wrong assumption Besides, just use AUTHY if that's your concern
- BackBlast 5y agoI migrated off AWS in December and closed my account. With the outages, they are likely looking at revenue shortfalls. Outages don't scare me, but their billing practices are already uncomfortable. This may make them prone to be more stingy about bill forgiveness. Bill forgiveness is inherent in the unsafe billing model which makes even having the account open even more of a huge liability. I'm done.
- deleted 5y ago[deleted]
- staticassertion 5y agoWell, yeah, of course they're stuck with the bill. I feel like people think AWS is supposed to have infinite guard rails regardless of what the engineers using it do, like when people write code that infinite loops and it blows up their bill. AWS gives money back in a lot of cases that I think they legitimately aren't responsible for. I don't know that other cloud providers are going to do any better - an attacker who has your credentials and spins up 10's of thousands of dollars of infra will cost you thousands of dollars. I'll certainly echo the advice for 2FA but, more importantly, use a strong, unique password.
- technion 5y agoI'm surprised MFA isn't mandated by default. It is in Azure: https://docs.microsoft.com/en-us/azure/active-directory/fundamentals/concept-fundamentals-security-defaults https://docs.microsoft.com/en-us/azure/active-directory/fund...
- zinekeller 5y agoThat link looks like MFA can be disabled but this chart shows otherwise (baseline versus opt-in enhanced): https://docs.microsoft.com/en-us/azure/active-directory/fundamentals/concept-fundamentals-security-defaults#authentication-methods https://docs.microsoft.com/en-us/azure/active-directory/fund...
- Olreich 5y agoYep, buying goodwill is very effective.
- Gigachad 5y agoI don't feel that some kind of quota management or predictable pricing is a significant ask. Most people or orgs do not need instant and infinite scaling. And would rather a 1 hour outage while they sort things out rather than a $100,000 bill for a minor bug.
- caslon 5y agoDon't use a "cloud" vendor then? Just use a VPS vendor.
- jeppesen-io 5y agoMFA is good advice Also, I'd create a IAM user and severely limit your usage of the root account, and over time stop using the root completely.
- javagram 5y agoGood reminder to completely close my AWS account. I have TOTP MFA on it but having a AWS account that had the same root login as my Amazon retail account was risky and a mistake from the beginning. Luckily I haven't used it for anything in years so it was as simple as following the "Close account" procedure. I'll use Digital Ocean for anything small if I need to spin up a server in the future.
- edoceo 5y agoCan you connect CloudWatch to your billing so you'd get an alert on some kind of spikes?
- quickthrower2 5y agoThanks! You've reminded me to close my hobby AWS account. I will do that now. Edit: Done! Was quite easy. Luckily I had no services running or needed.
- sgarg26 5y agoThis happened to me. I had a 9k bill and I got hacked and stuck with it on my personal account. I gave AWS $100k business and that did not matter through being a decision maker at a startup I work at. AWS does not care about your business unless you are going to IPO. True story. Feel free to message me at Sgargconsulting --> GMAIL
- halilduygulu 5y agoplease, everyone enable MFA and billing alerts in your accounts. do not commit access&secret keys to github.
- aborsy 5y agoServices like Lightsail have caps and flat fees. AWS needs more of this.