13 ms·

Hacker Accessed AWS for $50k+ – AWS Ignoring Me

57 points by csakon 5y ago

I'm trying to get help anywhere I can and a friend recommended I post this here. My business has used AWS for around 3 years and our normal usage is $1k per month in EC2 and S3. In early March a hacker accessed our AWS account through my login via an IP address in Austria (I'm in Austin, TX). They spun up 3 large instances of EC2 which began charging us $1k-$2k per day. In mid-April, while reviewing our books for the month of March, I saw a $26k charge from AWS. I thought it was a typo as $2.6k and asked the accountant. She stated that was the correct amount. I immediately got my dev team involved and we discovered the 3 instances to which we did not have any access to and stopped them immediately. I opened a support case immediately which somehow got posted twice. Because the case was posted twice, the support team marked both cases as duplicates. I reopened one of the cases, it was resolved again as a duplicate. This has now happened several times. I Googled around looking for a way to escalate this matter and found the following emails and cc'ed them on May 5th with an urgent plea via the original support case thread with another summary of the issue and links to my cases with my phone number to no avail. ams-csdm@amazon.com ams-opsmanager@amazon.com, ams-director@amazon.com, ams-vp@amazon.com That email was ignored and I'm not sure where I can turn to next. I've tweeted about this and tagged AWS here - https://twitter.com/csakon/status/1391873413107617799?s=20 I'm not sure where to go next, can anyone give me any advice?

plasma 5y ago
Ensure you’re paying for business support or the non-free one, and make a new case with a different title (don’t reopen existing ones) to try and get through.
- csakon 5y ago
  Great advice, thank you.
- tonfreed 5y ago
  Million times this. Free and developer tier support go straight to entry level drones in India, who will deal with it within Indian business hours. Business and enterprise tier are 24 hours, and will be dealt with by more experienced technicians within an hour.
scrollaway 5y ago
AWS support doesn't generally suck or behave the way you're describing without good reason, so I feel we're missing part of the story here. What are you leaving out? Anyway, it's important to frame what happened correctly: the security of someone on your team was sloppy, and most likely a bot was able to get an access key or access to one of your accounts, spin up crypto miners on EC2s and now you're responsible for the bill. If it hadn't been that, it'd have been ransomware, you probably got lucky. Now, to see if your situation can be improved: Put up some dollars and get business support. Make a clear and polite case, from the beginning. Ask for a refund but you don't have grounds to demand it; if they issue one, it's a gesture of good will. They probably will issue one if you haven't had to ask for that before, but it reflects badly on everybody that cryptominers weren't caught for two months. And before you create that ticket, make some billing alerts so you can show AWS support that this won't happen again.
- csakon 5y ago
  I don't think i'm leaving anything out. It was my account (which now has had password changes and MFA set up), but I don't understand how there weren't red flags on the Austrian IP address login and the sudden spike in usage. I realize (now) that CloudWatch exists, but not sure why this isn't standard. I was at fault for the double post of the support case, but that was a simple error on my part due to not thinking the first went through. Once access was made, we were completely unaware of their existence until I saw the charges and asked our devs about it. They said they didn't have any knowledge or access to these new instances. I appreciate the advice, will upgrade the support and try again.
  - orf 5y ago
    I mean, it’s hard to have sympathy. Your setup sounds like a complete mess especially if you didn’t notice it. You didn’t even have MFA and you’re using the console to create resources? It’s not on AWS to flag anything. They give you the tools to comprehensively monitor your account, but you choose not to use them. Cloudwatch is also “standard”. Datadog has a free tier, I’d suggest checking that out because you don’t seem to have any infrastructure monitoring? I’d honestly hire some people that have experience in this area, because your “dev team” sounds clueless.
seyeong_aws 5y ago
AMS is not a support org. Can you share your support case numbers? I may be able to escalate this to the right people.
poxrud 5y ago
For the future create a CloudWatch billing alert so that this doesn't happen again.
w0mbat 5y ago
Forgive me for stating the obvious, but if a problem gets reported more than once, you don't close ALL reports as duplicates and ignore the problem. Whether the original security breach was the user's fault or not, Amazon Support dropped the ball here.
- tonfreed 5y ago
  Yep, they have a habit of doing that. I had a Lambda timing out because of dropped network packets, they told me to up the timeout and closed the ticket without reading my description.
ies7 5y ago
You are aws user for 3 years but don’t have aws rep? In my case we’ve got a miner on our jenkin for a day. I just call my aws sales rep and he get me a free lunch and a few credit to pay the business support for 1 month, then open the ticket through that business support. At the end of the week aws gave us extra credits around 10% of our yearly usage. I don’t think they will waived all yours 26k. Thats your dev team fault and also your finance team or whoever that don’t watch the billing. But they can give you a lot of aws credit for many reason (promising startup, loyal customer, big company, etc)
- atatatat 5y ago
  > watch the billing As a busy founder, I'll just go with the providers that don't chain me to their dashboard/email instead of providing meaningful caps.
  - i_have_an_idea 5y ago
    you can setup whatever caps and alerts you want, it's not hard
caseymarquis 5y ago
So, what are best practices to avoid this situation in the first place? MFA. Billing alerts when estimated charges are over expected spending amounts. Anything else? Seems like a small mistake here could really harm a small business. Are there good ways to detect access that hasn't yet been exploited? Someone mentioned monitoring API calls, but what I'd googled on that seems fairly broad.
- ryan29 5y ago
  I've been looking into it and I think it would be possible to do if the documentation were better. They have `Budget Actions`, but it's fairly new so the documentation and examples are lacking. Based on what I've learned for myself, I would say the starter rules are: - Create a root account that's only used for consolidated billing and account recovery. Secure it with 2FA. I use TOTP saved on my Yubikey and my backup Yubikey with the setting that requires a physical touch to generate a code. - Create organizational accounts for every day use. The exact way to structure them can get complicated, but there's a fair bit of documentation on it. - Set up budget alerts and budget actions before you start using a resource type. - Only create users with permissions to access resource types with budget actions set up. It's easy to say, but very hard to do in practice based on my experience. The biggest problem is that if you want to use a couple of services (ex: EC2, S3), you have the complexity of 1000 services, IAM policies, etc. jammed in your face right at the start and it's almost impossible to figure out what permissions you need to do something. It reminds me of SELinux where the permissions are difficult enough to deal with that you can write an audit log while performing an action and simply enable all the permissions that were logged. The second biggest problem is that runaway billing is far worse for small users than for large users and big tech only cares about other big users because that's where the money is. Everything revolves around catering to huge users who don't care if they need to hire a consultant to tame their AWS billing, so the smaller users and startups are left with systems that are far too complex to meet their needs. I prefer the way Digital Ocean works, but there are some things you just can't do with them. For example, Lambdas and SES don't have good alternatives at DO. I also like Cloudflare Workers since I find it significantly easier to reason about price in the context of cost per execution instead of the complex formula used for Lambdas, etc.. I think Cloudflare is in a very good position to claw market share from the big clouds, but their Workers Unbound is pretty much a copy of Lambdas, Functions, etc. in terms of pricing structure, so it looks like they might be starting to go after those fat egress charges that everyone else makes their money from.
icedchai 5y ago
Did you set up billing alerts? If you had an alert at say, $1500/month, you would've noticed almost immediately.
- Dah00n 5y ago
  How does this help?
  - icedchai 5y ago
    If it happens again, you can stop a small problem before it becomes a very big one. Also, it is a good practice in general.
    - TrueGeek 5y ago
      It's still crap though. If they spin up an instance billing $1k a day you'll find out after they've already billed several hundred, if not at least $1k. There needs to be a way to set actually limits, not alerts. You should be able to say "I'm not using this service, the limit should be $0." Of course, if OP didn't have MFA / let their keys leak this may not have helped anyway if the hacker was able to just remove the limits.
      - icedchai 5y ago
        If they had actual limits, you'd have people complaining about their sites getting shut down because someone broke into their account and spun up a bunch of instances. Or a developer did it accidentally. Alerts are much safer.
      - isbvhodnvemrwvn 5y ago
        > You should be able to say "I'm not using this service, the limit should be $0." In your organization, add a service control policy which denies access to services you don't use. This will prevent all member accounts from executing actions you don't want, including root users. You can also deny any action on any resource in region other than whatever you expect to use (with some exceptions due to legacy stuff).
readonthegoapp 5y ago
not OP i had set up some billing alerts on my AWS and it was just all terrible and sh*tty, clicking around in a million places to get not what i actually wanted. i thought about building my own service to just let me know a daily total of my expected bill at the end of the month then i'd add stuff to tell me about fast-increasing charges, as quickly as was necessary depending on the steepness of the charge rise curve. then i found Billgist, which i tried for a bit -- it worked great, looks great, etc., so i'm not building my own. no connection to them. https://www.billgist.com/ https://www.billgist.com/ i used it at first just to see if it worked at all, then to see if it sucked (in which case I would prob try to build my own), and then ultimately to try to help me get comfortable with the idea that i probably wasn't going to wake up one beautiful Saturday morning to a $50k AWS bill. that never worked -- that is, i never got comfortable with the idea that I would _not_ wake up to a $50k AWS bill one beautiful Saturday morning -- it just seems completely plausible, even likely. so i shut down most of my aws stuff (i was always particularly worried about my Lambda stuff), moved some things to Digital Ocean, and i'm guessing i'll revisit AWS at some point when i reach some critical mass of: * "i actually need AWS", and * "i actually have something to implement that has the possibility of making money", and * "i'm comfortable, thru my own alerts/billing limits/cutoffs/aws-expertise, that i probably won't wake up to that $50k hacker AWS bill". one thing i learned is that AWS charges you for _everything_ -- including your single daily API call to figure out how much you're going to owe at the end of the month -- the fee for that call is 3 cents per call, or at least for the first call. tho you can log in thru the console and check this estimate for free. one of the things you get charged for is 'Configs' -- pretty much any config setting you've customized in any way -- permissions, roles, tags (?), etc. i understand the logic, but damn -- i'm trying to use AWS so i can get things done, not so i can worry about the costs of every. single. not-completely-optimized. miniscule. design decision. down to the penny. nickle and diming would be a luxury. i can imagine my hypothetical company's AWS Cost Saving Specialist coming to me and saying, "I'm glad you've set up this incredibly fast and secure and resilient system, but....we need to save a few bucks, so....yeah, i'm gonna need you to come in tomorrow...." i may have a former co-worker that works there - if you run out of options i'll try to ping someone in my chain, see if i can contact them.
BelenusMordred 5y ago
> I'm not sure where to go next, can anyone give me any advice? Have you contacted the FBI and Europol? A police report is the first thing you need before any company starts taking you seriously about crime being committed on your billing accounts. https://www.fbi.gov/investigate/cyber https://www.fbi.gov/investigate/cyber https://www.europol.europa.eu/report-a-crime/report-cybercrime-online https://www.europol.europa.eu/report-a-crime/report-cybercri...
deleted 5y ago
[deleted]
usr1106 5y ago
AWS is a bad service because they don't let the customer set billing limits. It seems earning money from users' mistakes is part of their business model. Yes, resource limits are not a silver bullet. Users will complain when their important service goes down because it would have gone slightly over budget during "normal" use. Implementing it in a reasonable way is not perfectly simple. Probably you want separate limits for network, storage, and processing as well as different ways to enforce the limit. Deleting all S3 data might not be what many users would want. They might still be willing to pay for keeping the existing data. And obviously changing the limit must not be possible with the same credentials that allow you to use resources. Another fundamental challenge. But with the size of AWS's business there is no excuse not to implement anything. They just value profit over customers. (Sorry, not a reply to the original poster's question. I don't have anything significantly different from what has been mentioned by others.)

57 points by csakon 5y ago

plasma 5y ago

Ensure you’re paying for business support or the non-free one, and make a new case with a different title (don’t reopen existing ones) to try and get through.

csakon 5y ago
Great advice, thank you.
tonfreed 5y ago
Million times this. Free and developer tier support go straight to entry level drones in India, who will deal with it within Indian business hours. Business and enterprise tier are 24 hours, and will be dealt with by more experienced technicians within an hour.

scrollaway 5y ago

AWS support doesn't generally suck or behave the way you're describing without good reason, so I feel we're missing part of the story here. What are you leaving out? Anyway, it's important to frame what happened correctly: the security of someone on your team was sloppy, and most likely a bot was able to get an access key or access to one of your accounts, spin up crypto miners on EC2s and now you're responsible for the bill. If it hadn't been that, it'd have been ransomware, you probably got lucky. Now, to see if your situation can be improved: Put up some dollars and get business support. Make a clear and polite case, from the beginning. Ask for a refund but you don't have grounds to demand it; if they issue one, it's a gesture of good will. They probably will issue one if you haven't had to ask for that before, but it reflects badly on everybody that cryptominers weren't caught for two months. And before you create that ticket, make some billing alerts so you can show AWS support that this won't happen again.

csakon 5y ago
I don't think i'm leaving anything out. It was my account (which now has had password changes and MFA set up), but I don't understand how there weren't red flags on the Austrian IP address login and the sudden spike in usage. I realize (now) that CloudWatch exists, but not sure why this isn't standard. I was at fault for the double post of the support case, but that was a simple error on my part due to not thinking the first went through. Once access was made, we were completely unaware of their existence until I saw the charges and asked our devs about it. They said they didn't have any knowledge or access to these new instances. I appreciate the advice, will upgrade the support and try again.
- orf 5y ago
  I mean, it’s hard to have sympathy. Your setup sounds like a complete mess especially if you didn’t notice it. You didn’t even have MFA and you’re using the console to create resources? It’s not on AWS to flag anything. They give you the tools to comprehensively monitor your account, but you choose not to use them. Cloudwatch is also “standard”. Datadog has a free tier, I’d suggest checking that out because you don’t seem to have any infrastructure monitoring? I’d honestly hire some people that have experience in this area, because your “dev team” sounds clueless.

seyeong_aws 5y ago

AMS is not a support org. Can you share your support case numbers? I may be able to escalate this to the right people.

poxrud 5y ago

For the future create a CloudWatch billing alert so that this doesn't happen again.

w0mbat 5y ago

Forgive me for stating the obvious, but if a problem gets reported more than once, you don't close ALL reports as duplicates and ignore the problem. Whether the original security breach was the user's fault or not, Amazon Support dropped the ball here.

tonfreed 5y ago
Yep, they have a habit of doing that. I had a Lambda timing out because of dropped network packets, they told me to up the timeout and closed the ticket without reading my description.

ies7 5y ago

You are aws user for 3 years but don’t have aws rep? In my case we’ve got a miner on our jenkin for a day. I just call my aws sales rep and he get me a free lunch and a few credit to pay the business support for 1 month, then open the ticket through that business support. At the end of the week aws gave us extra credits around 10% of our yearly usage. I don’t think they will waived all yours 26k. Thats your dev team fault and also your finance team or whoever that don’t watch the billing. But they can give you a lot of aws credit for many reason (promising startup, loyal customer, big company, etc)

atatatat 5y ago
> watch the billing As a busy founder, I'll just go with the providers that don't chain me to their dashboard/email instead of providing meaningful caps.
- i_have_an_idea 5y ago
  you can setup whatever caps and alerts you want, it's not hard

caseymarquis 5y ago

So, what are best practices to avoid this situation in the first place? MFA. Billing alerts when estimated charges are over expected spending amounts. Anything else? Seems like a small mistake here could really harm a small business. Are there good ways to detect access that hasn't yet been exploited? Someone mentioned monitoring API calls, but what I'd googled on that seems fairly broad.

ryan29 5y ago
I've been looking into it and I think it would be possible to do if the documentation were better. They have `Budget Actions`, but it's fairly new so the documentation and examples are lacking. Based on what I've learned for myself, I would say the starter rules are: - Create a root account that's only used for consolidated billing and account recovery. Secure it with 2FA. I use TOTP saved on my Yubikey and my backup Yubikey with the setting that requires a physical touch to generate a code. - Create organizational accounts for every day use. The exact way to structure them can get complicated, but there's a fair bit of documentation on it. - Set up budget alerts and budget actions before you start using a resource type. - Only create users with permissions to access resource types with budget actions set up. It's easy to say, but very hard to do in practice based on my experience. The biggest problem is that if you want to use a couple of services (ex: EC2, S3), you have the complexity of 1000 services, IAM policies, etc. jammed in your face right at the start and it's almost impossible to figure out what permissions you need to do something. It reminds me of SELinux where the permissions are difficult enough to deal with that you can write an audit log while performing an action and simply enable all the permissions that were logged. The second biggest problem is that runaway billing is far worse for small users than for large users and big tech only cares about other big users because that's where the money is. Everything revolves around catering to huge users who don't care if they need to hire a consultant to tame their AWS billing, so the smaller users and startups are left with systems that are far too complex to meet their needs. I prefer the way Digital Ocean works, but there are some things you just can't do with them. For example, Lambdas and SES don't have good alternatives at DO. I also like Cloudflare Workers since I find it significantly easier to reason about price in the context of cost per execution instead of the complex formula used for Lambdas, etc.. I think Cloudflare is in a very good position to claw market share from the big clouds, but their Workers Unbound is pretty much a copy of Lambdas, Functions, etc. in terms of pricing structure, so it looks like they might be starting to go after those fat egress charges that everyone else makes their money from.

icedchai 5y ago

Did you set up billing alerts? If you had an alert at say, $1500/month, you would've noticed almost immediately.

Dah00n 5y ago
How does this help?
- icedchai 5y ago
  If it happens again, you can stop a small problem before it becomes a very big one. Also, it is a good practice in general.
  - TrueGeek 5y ago
    It's still crap though. If they spin up an instance billing $1k a day you'll find out after they've already billed several hundred, if not at least $1k. There needs to be a way to set actually limits, not alerts. You should be able to say "I'm not using this service, the limit should be $0." Of course, if OP didn't have MFA / let their keys leak this may not have helped anyway if the hacker was able to just remove the limits.
    - icedchai 5y ago
      If they had actual limits, you'd have people complaining about their sites getting shut down because someone broke into their account and spun up a bunch of instances. Or a developer did it accidentally. Alerts are much safer.
    - isbvhodnvemrwvn 5y ago
      > You should be able to say "I'm not using this service, the limit should be $0." In your organization, add a service control policy which denies access to services you don't use. This will prevent all member accounts from executing actions you don't want, including root users. You can also deny any action on any resource in region other than whatever you expect to use (with some exceptions due to legacy stuff).

readonthegoapp 5y ago

not OP i had set up some billing alerts on my AWS and it was just all terrible and sh*tty, clicking around in a million places to get not what i actually wanted. i thought about building my own service to just let me know a daily total of my expected bill at the end of the month then i'd add stuff to tell me about fast-increasing charges, as quickly as was necessary depending on the steepness of the charge rise curve. then i found Billgist, which i tried for a bit -- it worked great, looks great, etc., so i'm not building my own. no connection to them. https://www.billgist.com/ https://www.billgist.com/ i used it at first just to see if it worked at all, then to see if it sucked (in which case I would prob try to build my own), and then ultimately to try to help me get comfortable with the idea that i probably wasn't going to wake up one beautiful Saturday morning to a $50k AWS bill. that never worked -- that is, i never got comfortable with the idea that I would _not_ wake up to a $50k AWS bill one beautiful Saturday morning -- it just seems completely plausible, even likely. so i shut down most of my aws stuff (i was always particularly worried about my Lambda stuff), moved some things to Digital Ocean, and i'm guessing i'll revisit AWS at some point when i reach some critical mass of: * "i actually need AWS", and * "i actually have something to implement that has the possibility of making money", and * "i'm comfortable, thru my own alerts/billing limits/cutoffs/aws-expertise, that i probably won't wake up to that $50k hacker AWS bill". one thing i learned is that AWS charges you for _everything_ -- including your single daily API call to figure out how much you're going to owe at the end of the month -- the fee for that call is 3 cents per call, or at least for the first call. tho you can log in thru the console and check this estimate for free. one of the things you get charged for is 'Configs' -- pretty much any config setting you've customized in any way -- permissions, roles, tags (?), etc. i understand the logic, but damn -- i'm trying to use AWS so i can get things done, not so i can worry about the costs of every. single. not-completely-optimized. miniscule. design decision. down to the penny. nickle and diming would be a luxury. i can imagine my hypothetical company's AWS Cost Saving Specialist coming to me and saying, "I'm glad you've set up this incredibly fast and secure and resilient system, but....we need to save a few bucks, so....yeah, i'm gonna need you to come in tomorrow...." i may have a former co-worker that works there - if you run out of options i'll try to ping someone in my chain, see if i can contact them.

BelenusMordred 5y ago

> I'm not sure where to go next, can anyone give me any advice? Have you contacted the FBI and Europol? A police report is the first thing you need before any company starts taking you seriously about crime being committed on your billing accounts. https://www.fbi.gov/investigate/cyber https://www.fbi.gov/investigate/cyber https://www.europol.europa.eu/report-a-crime/report-cybercrime-online https://www.europol.europa.eu/report-a-crime/report-cybercri...

deleted 5y ago

[deleted]

usr1106 5y ago

AWS is a bad service because they don't let the customer set billing limits. It seems earning money from users' mistakes is part of their business model. Yes, resource limits are not a silver bullet. Users will complain when their important service goes down because it would have gone slightly over budget during "normal" use. Implementing it in a reasonable way is not perfectly simple. Probably you want separate limits for network, storage, and processing as well as different ways to enforce the limit. Deleting all S3 data might not be what many users would want. They might still be willing to pay for keeping the existing data. And obviously changing the limit must not be possible with the same credentials that allow you to use resources. Another fundamental challenge. But with the size of AWS's business there is no excuse not to implement anything. They just value profit over customers. (Sorry, not a reply to the original poster's question. I don't have anything significantly different from what has been mentioned by others.)