4 ms·

How I became $12,000 Amazon's debtor just in 24 hours

30 points by manlog 12y ago

WestCoastJustin 12y ago
There has probably been a half dozen of these threads on HN, and every single time, they have the fees waived. You see a much different strategy with public telco horror stories regarding roaming/data fees and things like that. There are a couple pro tips for anyone interested: - Configure billing alerts, so that if your bill goes over an expected threshold you will get an email/sms/etc [1]. If your bill is usually $20/month, then set the alert trigger for $30/month, and you will get an alert. In this case, it looks like they just racked up the bill, likely via crypto-currency mining, but what if they were deleting data, etc. You want to be pretty proactive about this stuff. This only takes a couple minutes. - Use dedicated role accounts, rather than the AWS root account, so that you can roll keys without a huge impact [2]. You can also give narrow rights to your role accounts, so that they do not have access to everything, this will greatly limit the scope of an attack. It seems that people mistakenly publish their root AWS keys to github, then this type of stuff happens, but luckily AWS is pretty good about it. Although, AWS cannot help if you had your database exfiltrated, or production instances deleted, so you really MUST create role accounts, and limit their abilities [3]. - Monitor the activity on your account [4]. Create some type of alerts so that you will get notified about suspicious activity on your account. For example, if you do no generally create new instances, have an alert fire on this event type, via cloudtrail [5]. Same goes for deleted instances, etc, then you will be in the loop at things are happening. [1] http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/billing-what-is.html http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/... [2] http://docs.aws.amazon.com/IAM/latest/UserGuide/roles-creatingrole.html http://docs.aws.amazon.com/IAM/latest/UserGuide/roles-creati... [3] http://arstechnica.com/security/2014/06/aws-console-breach-leads-to-demise-of-service-with-proven-backup-plan/ http://arstechnica.com/security/2014/06/aws-console-breach-l... [4] http://docs.aws.amazon.com/general/latest/gr/aws-security-audit-guide.html http://docs.aws.amazon.com/general/latest/gr/aws-security-au... [5] http://aws.amazon.com/cloudtrail/ http://aws.amazon.com/cloudtrail/
- manlog 12y ago
  You can't imagine the feelings a person experiences, when he sees 12k+ debt on an almost not used account!
altcognito 12y ago
Kudos and smart thinking on Amazon's part to vacate these fees. If you heard the opposite story (about how people were filing for bankruptcy as a result of this), there's no way I'd take the chance of using the service, and I would feel hesitant to recommend their services to small businesses I work with. 5k-10k for some of those groups is their entire operating budget. (mostly volunteers)
- manlog 12y ago
  You never know... unfortunately
sytelus 12y ago
Amazon has avoided allowing users to set hard billing cap and that makes me worried a lot too. If your website gets suddenly too popular or DDoSed then you can get toast and might even go bankrupt. Amazon has reasoned that they don't want to disrupt your service by enabling this feature (because technically their safest choice to implement hard cap is to shutdown all resources assigned to you until you intervine). But as a user that's exactly what I want! I rather have my service go down then me go bankrupt from my ad-free website. It should be customer's choice to enable this feature if they want to but Amazon doesn't want to do it regardless of their supposedly "sharp" focus on customer needs. The feature they did enabled is CloudWatch where you get alerts if you go over the limit. But that's rather weak. If it happened while I was sleeping or on vacation, it is very easy to miss these notifications and face giant financial losses in just few hours. Having a non-commercial service hosted on AWS is like a sword hanging over your head that can fall anytime on you.
- Terretta 12y ago
  You can't set a dollar figure but you absolutely can set resource count caps. The dollar figure is less useful. If you set $1000 should it kick in when you hit it 12 hours before the end of the month? What if you hit $990 by the 2nd day of month? These are wildly different, and the dollar figure doesn't know. Service Limits, on the other hand, let you be sure the max you use every day won't overrun your EOM budget. http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html http://docs.aws.amazon.com/general/latest/gr/aws_service_lim...
  - manlog 12y ago
    But, why the hack they don't allow this option? Those people who wants to do that - let them go ahead... What's the problem with that?
- manlog 12y ago
  Totally agree!
tbronchain 12y ago
I found that project recently https://code.google.com/p/amazon-pricewatcher/ https://code.google.com/p/amazon-pricewatcher/ It's old, and probably won't work. But the idea is interesting. It monitors your AWS account, and shutdowns EC2 instances/removes S3 files if your bill goes up. It could be interesting to re-write it using boto. Maybe even make it simpler: If the bill goes over a "hard" limit, shutdown anything that can be shutdown (but don't delete anything!). I guess I would prefer to have a short downtime than a crazy bill. Added to a well setup alert monitoring, it could be a great way to limit crazy billing. @WestCoastJustin point to use dedicates roles account seems like a quite import thing as well. I think I would also trend to avoid large-range ASG.
- manlog 12y ago
  Nice idea!

30 points by manlog 12y ago

There has probably been a half dozen of these threads on HN, and every single time, they have the fees waived. You see a much different strategy with public telco horror stories regarding roaming/data fees and things like that. There are a couple pro tips for anyone interested: - Configure billing alerts, so that if your bill goes over an expected threshold you will get an email/sms/etc [1]. If your bill is usually $20/month, then set the alert trigger for $30/month, and you will get an alert. In this case, it looks like they just racked up the bill, likely via crypto-currency mining, but what if they were deleting data, etc. You want to be pretty proactive about this stuff. This only takes a couple minutes. - Use dedicated role accounts, rather than the AWS root account, so that you can roll keys without a huge impact [2]. You can also give narrow rights to your role accounts, so that they do not have access to everything, this will greatly limit the scope of an attack. It seems that people mistakenly publish their root AWS keys to github, then this type of stuff happens, but luckily AWS is pretty good about it. Although, AWS cannot help if you had your database exfiltrated, or production instances deleted, so you really MUST create role accounts, and limit their abilities [3]. - Monitor the activity on your account [4]. Create some type of alerts so that you will get notified about suspicious activity on your account. For example, if you do no generally create new instances, have an alert fire on this event type, via cloudtrail [5]. Same goes for deleted instances, etc, then you will be in the loop at things are happening. [1] http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/billing-what-is.html http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/... [2] http://docs.aws.amazon.com/IAM/latest/UserGuide/roles-creatingrole.html http://docs.aws.amazon.com/IAM/latest/UserGuide/roles-creati... [3] http://arstechnica.com/security/2014/06/aws-console-breach-leads-to-demise-of-service-with-proven-backup-plan/ http://arstechnica.com/security/2014/06/aws-console-breach-l... [4] http://docs.aws.amazon.com/general/latest/gr/aws-security-audit-guide.html http://docs.aws.amazon.com/general/latest/gr/aws-security-au... [5] http://aws.amazon.com/cloudtrail/ http://aws.amazon.com/cloudtrail/

manlog 12y ago
You can't imagine the feelings a person experiences, when he sees 12k+ debt on an almost not used account!

altcognito 12y ago

Kudos and smart thinking on Amazon's part to vacate these fees. If you heard the opposite story (about how people were filing for bankruptcy as a result of this), there's no way I'd take the chance of using the service, and I would feel hesitant to recommend their services to small businesses I work with. 5k-10k for some of those groups is their entire operating budget. (mostly volunteers)

manlog 12y ago
You never know... unfortunately

sytelus 12y ago

Amazon has avoided allowing users to set hard billing cap and that makes me worried a lot too. If your website gets suddenly too popular or DDoSed then you can get toast and might even go bankrupt. Amazon has reasoned that they don't want to disrupt your service by enabling this feature (because technically their safest choice to implement hard cap is to shutdown all resources assigned to you until you intervine). But as a user that's exactly what I want! I rather have my service go down then me go bankrupt from my ad-free website. It should be customer's choice to enable this feature if they want to but Amazon doesn't want to do it regardless of their supposedly "sharp" focus on customer needs. The feature they did enabled is CloudWatch where you get alerts if you go over the limit. But that's rather weak. If it happened while I was sleeping or on vacation, it is very easy to miss these notifications and face giant financial losses in just few hours. Having a non-commercial service hosted on AWS is like a sword hanging over your head that can fall anytime on you.

Terretta 12y ago
You can't set a dollar figure but you absolutely can set resource count caps. The dollar figure is less useful. If you set $1000 should it kick in when you hit it 12 hours before the end of the month? What if you hit $990 by the 2nd day of month? These are wildly different, and the dollar figure doesn't know. Service Limits, on the other hand, let you be sure the max you use every day won't overrun your EOM budget. http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html http://docs.aws.amazon.com/general/latest/gr/aws_service_lim...
- manlog 12y ago
  But, why the hack they don't allow this option? Those people who wants to do that - let them go ahead... What's the problem with that?
manlog 12y ago
Totally agree!

tbronchain 12y ago

I found that project recently https://code.google.com/p/amazon-pricewatcher/ https://code.google.com/p/amazon-pricewatcher/ It's old, and probably won't work. But the idea is interesting. It monitors your AWS account, and shutdowns EC2 instances/removes S3 files if your bill goes up. It could be interesting to re-write it using boto. Maybe even make it simpler: If the bill goes over a "hard" limit, shutdown anything that can be shutdown (but don't delete anything!). I guess I would prefer to have a short downtime than a crazy bill. Added to a well setup alert monitoring, it could be a great way to limit crazy billing. @WestCoastJustin point to use dedicates roles account seems like a quite import thing as well. I think I would also trend to avoid large-range ASG.

manlog 12y ago
Nice idea!