5 ms·

Setting up billing alarms is easy for personal accounts, it's actually only a nightmare if you go through a reseller (which is common in government contacting).

Setting up billing alarms is easy for personal accounts, it's actually only a nightmare if you go through a reseller (which is common in government contacting). The author had an incident starting June 23rd, but didn't know about it until he got his bill July 7th, that's potentially ~14 days someone could have been abusing his account. A billing alarm would have reduced this to hours or minutes. I would be interested how you suggest stopping a bill before it happens otherwise.. should AWS disable your website because you got posted on hacker news and now have a bill over your $10 limit? If AWS needs to stop your billing at $10, then it might need to shut down your EC2 instance and destroy your data... If course they could offer the ability to configure the response, but if someone doesn't take the 2 minutes necessary to put a $25 billing alarm in place, what are the chances they will go through the effort of service/object based abuse policies? At the end of the day, the issue here was that the user posted something online that people could abuse.. I don't think any CDNs cover 30 TB in free tier...

ghaff 6y ago
>should AWS disable your website because you got posted on hacker news and now have a bill over your $10 limit? Maybe that should be an option? Probably not a good idea for a business but if you're just using AWS to learn and otherwise fool around with, there's a good argument that it would be nice to be able to have a hard circuit breaker for at least stateless services. I've actually heard people argue that they consider everything they put up on AWS is ephemeral so a hard circuit breaker should have the option to burn everything down but that seems like it would create its own set of problems for many people. Disabling data egress and EC2 seems as if it would go a long way towards stopping most of the unexpected bill stories. And, yes, I'm aware of billing alarms and even setting up Lambda functions to take actions but, especially if you use S3 to host files, it would be nice to cap expenses at mostly your storage costs. I was doing some research for a very small non-profit that needs some hosted storage and I think Backblaze B2 is a better choice for them for this reason.
- Turing_Machine 6y ago
  > Maybe that should be an option? It should absolutely be an option. Maybe not the default, but an option.
- ldoughty 6y ago
  I certainly would like the "don't book me over $X" option, but the devil is in the details. Perhaps just turning off where's would help, but if you account is compromised and 50 servers are running trying to mine crypto, the hacjer won't turn off those servers just because it's no longer submitting results... So AWS should probably stop all your servers.. which means users might lose data... Makes it a difficult choice
arpinum 6y ago
AWS billing alarms and cost explorer are not realtime services, you can expect a 6 hour delay. In this users case, this could be after $500 has already been spent. AWS does not do enough to prevent harm, even if a user is diligent with alarms.
hasenpfote 6y ago
So what does this really help? Do i get now a higher liablility because the alert alerted me and i now need to be aware of it? so i'm going on holiday and i'm fucked? You know, a billig alert is a nice thing, its not the solution. Host a few things on AWS, tell me your endpoints and your bill just might be already in deep shit before you read your bill.
- dathinab 6y ago
  > higher liablility The problem is you are always fully liable! The only reason Amazone did not charge the full price is because it's good marketing for them. I'm not sure if this also would have happened if the person would be less good connected/have gotten less visibility. > so i'm going on holiday and i'm fucked? That's why AWS is super problematic for any company which isn't big enough to make sure there is always someone reachable by billing alert 24/7. A more extream case happened btw. a while ago with google serverless. Small startup with limited budged did a software error which massively increased data queries and got good publicity when they lunched and then was instant bankrupt, well except that google was also not sharing them because the bad press that would have caused. > You know, a billig alert is a nice thing, its not the solution. Host a few things on AWS, tell me your endpoints and your bill just might be already in deep shit before you read your bill. I agree, I'm strongly in favor of soft- and hard limits. (Per "project", with a dynamic limit of x-times of what the cost had been previous month, etc.).
doctor_eval 6y ago
AWS could always send an alarm if current aggregate spend is more than say 2x previous monthly spend.
dathinab 6y ago
> herwise.. should AWS disable your website because you got posted on hacker news and now have a bill over your $10 limit? YES!!!!, if you ask them to do so. $10 is a arbitrary small amount, but what if it's a web app and we are now speaking about $1000? And tbh. for many especially young people just $100 alone is much more then they can afford. Or better have also a option for a dynamic hard limit based on a multiplier compared to normal traffic. Even besides private persons there is a good reason to have a hard limit for a lot of businesses (or parts of them). Especially if it's on non-essential things people don't rely on which have a limited budged. (Like a lot of things provided for marketing only.) Heck even for payed service providers a hard limit can make sense given that it's high enough, as a form of last defense against intentional or accidental DOS like situations (I mean in times of auto-scaling some DOS situations became cost explosion situations instead). The problem with billing alerts is that (assuming they work and get delivered in time and are set to an sensible value): - You need to read them in time. - You need to be able to react to them in time. Both NOT trivial in for private persons and also small companies which just got started (or do mismanagement or are to tight on budged). Like what happens if you get it when you are on a party? In the cinema? Asleep? In the hospital? On holiday currently outside of reception? Somehow the idea that you will react in just a view minutes and then fix the problem, too (or shut the service down). Is a bit to optimistic. Through yes in the given case he might have ended up with "just" a view $ in the best case or ~600 or so in the worst case. Still I think he would have been happy with service discontinuing once the price reached a view hounded $ (let's just arbitrary say 250).

ghaff 6y ago

>should AWS disable your website because you got posted on hacker news and now have a bill over your $10 limit? Maybe that should be an option? Probably not a good idea for a business but if you're just using AWS to learn and otherwise fool around with, there's a good argument that it would be nice to be able to have a hard circuit breaker for at least stateless services. I've actually heard people argue that they consider everything they put up on AWS is ephemeral so a hard circuit breaker should have the option to burn everything down but that seems like it would create its own set of problems for many people. Disabling data egress and EC2 seems as if it would go a long way towards stopping most of the unexpected bill stories. And, yes, I'm aware of billing alarms and even setting up Lambda functions to take actions but, especially if you use S3 to host files, it would be nice to cap expenses at mostly your storage costs. I was doing some research for a very small non-profit that needs some hosted storage and I think Backblaze B2 is a better choice for them for this reason.

Turing_Machine 6y ago
> Maybe that should be an option? It should absolutely be an option. Maybe not the default, but an option.
ldoughty 6y ago
I certainly would like the "don't book me over $X" option, but the devil is in the details. Perhaps just turning off where's would help, but if you account is compromised and 50 servers are running trying to mine crypto, the hacjer won't turn off those servers just because it's no longer submitting results... So AWS should probably stop all your servers.. which means users might lose data... Makes it a difficult choice

arpinum 6y ago

AWS billing alarms and cost explorer are not realtime services, you can expect a 6 hour delay. In this users case, this could be after $500 has already been spent. AWS does not do enough to prevent harm, even if a user is diligent with alarms.

hasenpfote 6y ago

So what does this really help? Do i get now a higher liablility because the alert alerted me and i now need to be aware of it? so i'm going on holiday and i'm fucked? You know, a billig alert is a nice thing, its not the solution. Host a few things on AWS, tell me your endpoints and your bill just might be already in deep shit before you read your bill.

dathinab 6y ago
> higher liablility The problem is you are always fully liable! The only reason Amazone did not charge the full price is because it's good marketing for them. I'm not sure if this also would have happened if the person would be less good connected/have gotten less visibility. > so i'm going on holiday and i'm fucked? That's why AWS is super problematic for any company which isn't big enough to make sure there is always someone reachable by billing alert 24/7. A more extream case happened btw. a while ago with google serverless. Small startup with limited budged did a software error which massively increased data queries and got good publicity when they lunched and then was instant bankrupt, well except that google was also not sharing them because the bad press that would have caused. > You know, a billig alert is a nice thing, its not the solution. Host a few things on AWS, tell me your endpoints and your bill just might be already in deep shit before you read your bill. I agree, I'm strongly in favor of soft- and hard limits. (Per "project", with a dynamic limit of x-times of what the cost had been previous month, etc.).

doctor_eval 6y ago

AWS could always send an alarm if current aggregate spend is more than say 2x previous monthly spend.

dathinab 6y ago

> herwise.. should AWS disable your website because you got posted on hacker news and now have a bill over your $10 limit? YES!!!!, if you ask them to do so. $10 is a arbitrary small amount, but what if it's a web app and we are now speaking about $1000? And tbh. for many especially young people just $100 alone is much more then they can afford. Or better have also a option for a dynamic hard limit based on a multiplier compared to normal traffic. Even besides private persons there is a good reason to have a hard limit for a lot of businesses (or parts of them). Especially if it's on non-essential things people don't rely on which have a limited budged. (Like a lot of things provided for marketing only.) Heck even for payed service providers a hard limit can make sense given that it's high enough, as a form of last defense against intentional or accidental DOS like situations (I mean in times of auto-scaling some DOS situations became cost explosion situations instead). The problem with billing alerts is that (assuming they work and get delivered in time and are set to an sensible value): - You need to read them in time. - You need to be able to react to them in time. Both NOT trivial in for private persons and also small companies which just got started (or do mismanagement or are to tight on budged). Like what happens if you get it when you are on a party? In the cinema? Asleep? In the hospital? On holiday currently outside of reception? Somehow the idea that you will react in just a view minutes and then fix the problem, too (or shut the service down). Is a bit to optimistic. Through yes in the given case he might have ended up with "just" a view $ in the best case or ~600 or so in the worst case. Still I think he would have been happy with service discontinuing once the price reached a view hounded $ (let's just arbitrary say 250).