5 ms·

The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/act

The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the customers users. And with billing alerts the customer is able to make the choice of whether the cost is worth it as it happens. Note: Principal at AWS. Have worked to issue many credits/bill amendments, but dont work in that area nor do I speak for AWS.

tpmx 6y ago
The upside for the noob trying out/learning is huge. I'm no longer that person, but I think GCP/AWS are just being lazy about this - perhaps because they earn a lot of money from engineer mistakes. Of course it's possible to create an actual limit. There'll be some engineering cost, like 0.5%-1% extra per service? Edit: Being European I think legislation might be the fix, since both Amazon and Google have demonstrated an unwillingness to fix this issue, for a very long time.
Alupis 6y ago
Why not alert thresholds, configurable by the user? Email me when we cross $X amount in one day, Text when we cross $Y, and Call when we cross $Z. Additionally, allow the user to configure a hard cut-off limit if they desire. Just provide the mechanisms and allow users to make the call. Google et al would have a much stronger leg to stand on when enforcing delinquent account collections if they provided these mechanisms and the user chose to ignore them. Additionally, Google et al should protect _themselves_ by tracking usage patterns, and reaching out to customers that grossly surpass the average billable amount - just like OP with their near $100k bill in 1 day. Zero vetting to even have a reasonable guarantee the individual or company is even capable of paying such a large bill. And then what? Sue a company that doesn't have $100k for $100k? This makes zero sense.
- patch_cable 6y ago
  Something like this for alerts? https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...
  - Alupis 6y ago
    Doesn't look like there's any cutoff mechanism there, and it's a separate, optional step instead of part of the setup flow with a mandatory opt-out warning. Nor does that address the other complaint - Google (and possibly others) seem to be willing to extend an unlimited credit line to all customers without any prior vetting for ability to pay. That's crazy.
- Retailer 6y ago
  Google has alert thresholds (you set it up under your Budget). But practically speaking, an alert is not enough - what if you are unavailable to get the alert, it comes in the middle of the night, etc? A better solution would have been 'limits' which they used to have (at least for Google App Engine) but which has been deprecated. We had to spend sometime to research and see if there was a work around because just like the author of the article, we were quite worried about suddenly consuming a huge amount of resources, getting a spike in our bill and our accounts being cut off/suspended because we hadn't paid the bill. We've documented our solution here https://retailingplatform.com/blog/how-to-control-costs-on-a-google-app-engine-project/ https://retailingplatform.com/blog/how-to-control-costs-on-a...
dralley 6y ago
>It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. Making the worst case scenario no worse than traditional infrastructure.
- speed_spread 6y ago
  Correct. That argument assumes that every penny spent autoscaling has a positive ROI.
  - freemint 6y ago
    I think this is a insightful way to think about. Thanks.
pstuart 6y ago
There are ways it could be done relatively benignly, such as defaulting to paranoid and explicitly opting out. And for those that are heading into that financial barrier it should be a straightforward problem to look at trending to anticipate the shutdown and send out an alert.
- dathinab 6y ago
  Or just ask for a default when opening a new account ;=)
mewpmewp2 6y ago
I was creating some side-project. I already incurred around $100 fee. I imagine if I made some looping/recursion bug I could've easily incurred a cost of $10,000 or frankly, infinite cost. How easy would it have been for me to get this pardoned? And at the very moment I would discover that I just lost $100,000 - would I know in advance that they are definitely going to forgive this because I'd be full panic mode? It was very scary for me to use cloud in this case. I didn't even have any customers at that point.
cogman10 6y ago
> And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. What? Why wouldn't this just be an opt in thing? It could even be tied to the account being used. It's not like AWS accounts are expensive or hard to setup. If a user opts in to the "kill if bill goes too high" and they kill a critical portion of their business, then that's on them. Similar to how a user choosing spot instances if their spot ends up being destroyed. You've already got that "I can kill your stuff if you opt into it" capability. > On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the customers users. Yeah, and what happens if someone isn't big enough to justify AWS's forgiveness? What if they get a rep that blows off their request or is having a bad day? You are at the mercy of your cloud provider to forgive a debt, which is a real shitty place to be for anyone. > And with billing alerts the customer is able to make the choice of whether the cost is worth it as it happens. And what do they do if they miss the alert? You can rack up a huge bill in very little time the right AWS services. The point of the kill switch cap is to guard against risk. The fact is that that while 72k isn't too big for some companies, it means bankruptcy for others. Its because you might want to give your devs a training account to play with AWS services to gain expertise, but you don't want them to blow $1 million dollars screwing around with Amazon satellite services.
- remus 6y ago
  > What? Why wouldn't this just be an opt in thing? "Oh cool, I'll set a $1k cap, never gonna spend that on this little side proj." Fast forward a year, the side proj has turned in to a critical piece of the business but the person who set it up has left and no one remembered to turn of the spending cap. Busy christmas shopping period comes along, AWS shuts down the whole account because they go over the spending cap, 6hr outage during peak hours, $20k sales down the pan. Of course it is technically the customers fault but it's a shit experience. Accidentally spending $72k is also technically the customers fault and also a shit experience. I don't think there is an easy solution to this problem.
  - freemint 6y ago
    There exist totally a solution. It is also user hostile enough so it might get's adopted. $cloud_vendor just has to (and probably will) constantly nudge people to loosen the limit. Have a red banner that says, "you spend 3% of your monthly budget already think about increasing it". Also routinely send out emails to remind people. " Black Friday comes up think about increasing your quota " when your service has nothing to do with E-Commerce.
dathinab 6y ago
> The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. This is simple wrong. Depending on your use-case disabling active resources is the right reasonable solution with less downsides. E.g. most (smaller) companies would prefer their miscellaneous (i.e. no core-product) website/app/service to be temporary unavailable then have a massive unexpected cost they might not be able to afort which might literally force them to fire people because they can't pay them.... I mean think about it, what worth is it if my app doesn't go temporary unavailable during it's free trial phase when it means I'm going bankrupt from today to tomorrow and in turn can't benefit from it at all. Sure hug companies can always throw more money at it and will likely prefer uninterrupted service. But then for every hug company there are hundreds smaller companies which have different priorities. In the end it should be the users choice, a configuration settings you can set (per preferably per project). And sure limits should probably be resource limits (like accumulated compute time) and not billing limits as prices might be in flux or dependent on your total resource usage or similar so computing it is non trivial or even impossible. I often have the feelings that hug companies like Amazone or Google often get so detached from how thinks work for literally every one else (who is not a hug company). That they don't realize that solutions proper for hug companies might not only be sub-optimal but literally crippling bad for medium and small companies.
vp8989 6y ago
"The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active." Lol what ... this is exactly what happens any time you hit a rate limit on any AWS service. The customers application is "catastrophically interrupted" during its most popular/active period. The only difference is in that case, it suits AWS to do that whereas in the case of respecting a billing limit, it doesn't.
- howlgarnish 6y ago
  If you hit a rate limit, the marginal portion of requests exceeding that limit is dropped: if you plot the requests, the graph gets clipped. Bad, but not catastrophic. If you hit a billing limit, everything beyond that point is dropped, and the graph of requests plunges to zero. You're effectively hard down in prod.
  - freemint 6y ago
    Just as it is with bank accounts. Once you run out of money you hit a hard floor.
  - franciscop 6y ago
    And for some companies/individuals, if you keep charging then THEY will plunge to a large negative debt. It's not even zero, it's a lot worse than that.
howlgarnish 6y ago
This. App Engine used to offer hard spending limits, and they were removed with precisely because so many users set them up to shoot themselves in the foot at precisely the worst possible moment.
- salmonlogs 6y ago
  ^^ this. Hard spending limits seem great until your app/service gets super popular and you have to explain to the CEO why you were down during the exact window you needed to be serving the demand.
Aeolun 6y ago
I feel less troubled by this in AWS because they actually have functional customer service.
arp242 6y ago
> The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. Well, this is true, but this is also true of a lot of limits, like limits.conf. Sometimes you really want to spawn loads of processes or open many files, but a lot of the time you don't, so a barrier to limit the damage makes sense. There is no one solution that will fit everyone: people should be able to choose: "scale to the max", "spend at most $100", etc. If my average bill is $100, then a limit of $500 would probably make sense, just as a proverbial seat belt. This should never be reached and prevents things going out of control (which is also the reason for limits.conf).
franciscop 6y ago
> "And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource." Well there's a very easy way, adding a checkbox and an input: [ ] I am just trying things out, don't charge me more than [ ] USD
welkie 6y ago
> It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. This could be ameliorated by using namespacing techniques to separate prod from dev resources. For example, GCP uses projects to namespace your resources. And you can delete everything in a project in one operation that is impossible to fail by just shutting down the project (no "you can't delete x, because y references it" messages). Aggressive billing alerts and events, that delete services when thresholds are met, could be used only in the development namespace. That way, fun little projects can be shut down and prod traffic can be free to use a bit more billing when it needs to.

tpmx 6y ago

The upside for the noob trying out/learning is huge. I'm no longer that person, but I think GCP/AWS are just being lazy about this - perhaps because they earn a lot of money from engineer mistakes. Of course it's possible to create an actual limit. There'll be some engineering cost, like 0.5%-1% extra per service? Edit: Being European I think legislation might be the fix, since both Amazon and Google have demonstrated an unwillingness to fix this issue, for a very long time.

Alupis 6y ago

Why not alert thresholds, configurable by the user? Email me when we cross $X amount in one day, Text when we cross $Y, and Call when we cross $Z. Additionally, allow the user to configure a hard cut-off limit if they desire. Just provide the mechanisms and allow users to make the call. Google et al would have a much stronger leg to stand on when enforcing delinquent account collections if they provided these mechanisms and the user chose to ignore them. Additionally, Google et al should protect _themselves_ by tracking usage patterns, and reaching out to customers that grossly surpass the average billable amount - just like OP with their near $100k bill in 1 day. Zero vetting to even have a reasonable guarantee the individual or company is even capable of paying such a large bill. And then what? Sue a company that doesn't have $100k for $100k? This makes zero sense.

patch_cable 6y ago
Something like this for alerts? https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...
- Alupis 6y ago
  Doesn't look like there's any cutoff mechanism there, and it's a separate, optional step instead of part of the setup flow with a mandatory opt-out warning. Nor does that address the other complaint - Google (and possibly others) seem to be willing to extend an unlimited credit line to all customers without any prior vetting for ability to pay. That's crazy.
Retailer 6y ago
Google has alert thresholds (you set it up under your Budget). But practically speaking, an alert is not enough - what if you are unavailable to get the alert, it comes in the middle of the night, etc? A better solution would have been 'limits' which they used to have (at least for Google App Engine) but which has been deprecated. We had to spend sometime to research and see if there was a work around because just like the author of the article, we were quite worried about suddenly consuming a huge amount of resources, getting a spike in our bill and our accounts being cut off/suspended because we hadn't paid the bill. We've documented our solution here https://retailingplatform.com/blog/how-to-control-costs-on-a-google-app-engine-project/ https://retailingplatform.com/blog/how-to-control-costs-on-a...

dralley 6y ago

>It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. Making the worst case scenario no worse than traditional infrastructure.

speed_spread 6y ago
Correct. That argument assumes that every penny spent autoscaling has a positive ROI.
- freemint 6y ago
  I think this is a insightful way to think about. Thanks.

pstuart 6y ago

There are ways it could be done relatively benignly, such as defaulting to paranoid and explicitly opting out. And for those that are heading into that financial barrier it should be a straightforward problem to look at trending to anticipate the shutdown and send out an alert.

dathinab 6y ago
Or just ask for a default when opening a new account ;=)

mewpmewp2 6y ago

I was creating some side-project. I already incurred around $100 fee. I imagine if I made some looping/recursion bug I could've easily incurred a cost of $10,000 or frankly, infinite cost. How easy would it have been for me to get this pardoned? And at the very moment I would discover that I just lost $100,000 - would I know in advance that they are definitely going to forgive this because I'd be full panic mode? It was very scary for me to use cloud in this case. I didn't even have any customers at that point.

cogman10 6y ago

> And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. What? Why wouldn't this just be an opt in thing? It could even be tied to the account being used. It's not like AWS accounts are expensive or hard to setup. If a user opts in to the "kill if bill goes too high" and they kill a critical portion of their business, then that's on them. Similar to how a user choosing spot instances if their spot ends up being destroyed. You've already got that "I can kill your stuff if you opt into it" capability. > On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the customers users. Yeah, and what happens if someone isn't big enough to justify AWS's forgiveness? What if they get a rep that blows off their request or is having a bad day? You are at the mercy of your cloud provider to forgive a debt, which is a real shitty place to be for anyone. > And with billing alerts the customer is able to make the choice of whether the cost is worth it as it happens. And what do they do if they miss the alert? You can rack up a huge bill in very little time the right AWS services. The point of the kill switch cap is to guard against risk. The fact is that that while 72k isn't too big for some companies, it means bankruptcy for others. Its because you might want to give your devs a training account to play with AWS services to gain expertise, but you don't want them to blow $1 million dollars screwing around with Amazon satellite services.

remus 6y ago
> What? Why wouldn't this just be an opt in thing? "Oh cool, I'll set a $1k cap, never gonna spend that on this little side proj." Fast forward a year, the side proj has turned in to a critical piece of the business but the person who set it up has left and no one remembered to turn of the spending cap. Busy christmas shopping period comes along, AWS shuts down the whole account because they go over the spending cap, 6hr outage during peak hours, $20k sales down the pan. Of course it is technically the customers fault but it's a shit experience. Accidentally spending $72k is also technically the customers fault and also a shit experience. I don't think there is an easy solution to this problem.
- freemint 6y ago
  There exist totally a solution. It is also user hostile enough so it might get's adopted. $cloud_vendor just has to (and probably will) constantly nudge people to loosen the limit. Have a red banner that says, "you spend 3% of your monthly budget already think about increasing it". Also routinely send out emails to remind people. " Black Friday comes up think about increasing your quota " when your service has nothing to do with E-Commerce.

dathinab 6y ago

> The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. This is simple wrong. Depending on your use-case disabling active resources is the right reasonable solution with less downsides. E.g. most (smaller) companies would prefer their miscellaneous (i.e. no core-product) website/app/service to be temporary unavailable then have a massive unexpected cost they might not be able to afort which might literally force them to fire people because they can't pay them.... I mean think about it, what worth is it if my app doesn't go temporary unavailable during it's free trial phase when it means I'm going bankrupt from today to tomorrow and in turn can't benefit from it at all. Sure hug companies can always throw more money at it and will likely prefer uninterrupted service. But then for every hug company there are hundreds smaller companies which have different priorities. In the end it should be the users choice, a configuration settings you can set (per preferably per project). And sure limits should probably be resource limits (like accumulated compute time) and not billing limits as prices might be in flux or dependent on your total resource usage or similar so computing it is non trivial or even impossible. I often have the feelings that hug companies like Amazone or Google often get so detached from how thinks work for literally every one else (who is not a hug company). That they don't realize that solutions proper for hug companies might not only be sub-optimal but literally crippling bad for medium and small companies.

vp8989 6y ago

"The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active." Lol what ... this is exactly what happens any time you hit a rate limit on any AWS service. The customers application is "catastrophically interrupted" during its most popular/active period. The only difference is in that case, it suits AWS to do that whereas in the case of respecting a billing limit, it doesn't.

howlgarnish 6y ago
If you hit a rate limit, the marginal portion of requests exceeding that limit is dropped: if you plot the requests, the graph gets clipped. Bad, but not catastrophic. If you hit a billing limit, everything beyond that point is dropped, and the graph of requests plunges to zero. You're effectively hard down in prod.
- freemint 6y ago
  Just as it is with bank accounts. Once you run out of money you hit a hard floor.
- franciscop 6y ago
  And for some companies/individuals, if you keep charging then THEY will plunge to a large negative debt. It's not even zero, it's a lot worse than that.

howlgarnish 6y ago

This. App Engine used to offer hard spending limits, and they were removed with precisely because so many users set them up to shoot themselves in the foot at precisely the worst possible moment.

salmonlogs 6y ago
^^ this. Hard spending limits seem great until your app/service gets super popular and you have to explain to the CEO why you were down during the exact window you needed to be serving the demand.

Aeolun 6y ago

I feel less troubled by this in AWS because they actually have functional customer service.

arp242 6y ago

franciscop 6y ago

> "And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource." Well there's a very easy way, adding a checkbox and an input: [ ] I am just trying things out, don't charge me more than [ ] USD

welkie 6y ago

> It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. This could be ameliorated by using namespacing techniques to separate prod from dev resources. For example, GCP uses projects to namespace your resources. And you can delete everything in a project in one operation that is impossible to fail by just shutting down the project (no "you can't delete x, because y references it" messages). Aggressive billing alerts and events, that delete services when thresholds are met, could be used only in the development namespace. That way, fun little projects can be shut down and prod traffic can be free to use a bit more billing when it needs to.