23 ms·

AWS bill capping feature request thread still unanswered after 10 years

358 points by shikkra 5y ago

lacker 5y ago
You really need to implement limits service-by-service, and with as many services as AWS has, it would make sense if some of them hadn't implemented limits yet. Think about the details... if you are paying something for both storage and bandwidth, and get a sudden surge of bandwidth, do you really want items in storage to be deleted? You basically never want storage to be automatically deleted even if your program suddenly uses a surprising amount; limits on its maximum size and alerts are much better. But once you realize that bill capping doesn't make sense for storage, well, many different services are essentially some type of storage. This is a feature that sounds good but in practice what people really need is something slightly different.
- akira2501 5y ago
  > But once you realize that bill capping doesn't make sense for storage I'd think you want something like a circuit breaker.. steady costs or slowly increasing costs aren't like to be a problem, but a sudden surge in costs is what I'm concerned with. Perhaps if it operated like a time-averaged quota.. don't let me incur _new_ costs if I slide too far out of my apparent range. Give me a knob to control that range or temporarily disable it, and maybe a way to monitor those events so I can react to them appropriately for my particular application.
- capableweb 5y ago
  > You basically never want storage to be automatically deleted even if your program suddenly uses a surprising amount In most use cases you're right. But never? Not true, not all storage is used the same way. And the thing is, why should Amazon decide which use case is valid or not? I might care more about being able to afford the service than to keep the data around, depending on what kind of service I offer my user. As a platform provider, they should be agnostic, but their greed for money is shining through their willingness to be a true platform provider.
  - philliphaydon 5y ago
    There are rules in S3 to move files or delete them after a period of time.
    - Hamuko 5y ago
      And perhaps most useful rule is to move them to cheaper tiers of storage. For example, Glacier Deep Archive is super cheap for storage since it's $0.99 per terabyte-month.
- p1necone 5y ago
  I don't think you ever want storage to be deleted as part of an automatic bill cap system. Just refuse access, or allow reads but not writes until the customer reviews it. There's even an HTTP code for it: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/402 https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/402 (non standard, but I know for a fact it's used by Azure). Actually deleting things should be done after a period of time specified in writing in the contract with the customer, and I'd hope you'd try to contact them first. But I think an automatic bill cap absolutely makes sense for storage, and I can't really see any reasons it couldn't work.
  - deleted 5y ago
    [deleted]
  - nucleardog 5y ago
    > and I can't really see any reasons it couldn't work. You’re assuming the existence of a function which can extrapolate the daily/monthly/etc costs. If I upload a 10GB file at $0.10/gbmo, how much will it cost me? We have no idea because we don’t know how long it will be up there. We have a process that downloads and streams massive reports into S3, then a process is immediately kicked off to handle processing and importing. As soon as that completes, the files are deleted. So we’re paying about 0.04 months of storage every month. The naive solution is estimating 1.00. You’re off by a factor of ~24. We could... not handle that. Now we’re in a situation where we’re consistently spending $100/mo on storage, set a liberal $150 budget and immediately everything breaks. We actually needed to set a $2400 budget! Expecting a $100 bill and getting a $2400 bill some month when something goes wrong is as good as doing nothing. You’re also going to need to account for the interaction of every single object in a bucket and the lifecycle rules set which is an absolutely ridiculous amount of overhead. Now extend this to... literally every one of the seemingly bazillion AWS services. The only cap that’s going to be easy and realistic to implement without a crystal ball is a hard cut-off once you’ve actually accrued the charges, and a “shut down all my servers, delete all my data, and just close my account” cap is useful to almost no one. If you want useful caps they need to be aware of your workload. The best person to implement those is you. The tools are there.
- forty 5y ago
  Exactly. I have heard stories of bill capping that ended up with infra and data being deleted, so I can really see why they wouldn't implement it. What they could do however would be to allow usage limit based on IAM policies for example (Deny s3:Getobject Condition: monthly bandwidth >= x)
karmasimida 5y ago
How is this going to be implemented? When it reach the cap, all your service stop working? This is incredibly user hostile. If you want some warning when utilization is high, it can be relatively easy to setup with AWS's existing feature, but you need to do some configuration/tweaking on your own.
- btilly 5y ago
  When it reach the cap, all your service stop working? This is incredibly user hostile. Not nearly as user hostile as the possibility of an unexpected DOS on your bank account while you're asleep. Which has actually happened to people.
  - handmodel 5y ago
    My experience is just one data point but: My expertise is not at all anything related to web development so when I first tried experimenting with AWS for a side project I was terrified. It was an experiment for me and i didn't care if it deleted the data since the data it was storing was test data I was uploading. However, my budget for my entire side project was $500 and it seemed totally feasible (at least to a first time user of such a service) that some wrong lines in the code could balloon the requests or storage I made. At the time, I didn't even have $5000 of liquid assets and it would have been a nightmare.
- technion 5y ago
  That would be substantively less user hostile to me than a massive bill on a personal project, which is easy to see occurring for any number of reasons.
- exactlysolved 5y ago
  > When it reach the cap, all your service stop working? This is incredibly user hostile. So set your cap to a very high number, much higher than any amount you expect to be billed. Surely for everyone there is some number at which they would rather their services go down, than have to pay it?
- varispeed 5y ago
e_commerce 5y ago
They're on the decline and are now in the "extract every last cent" phase of the organization.
- samzer 5y ago
  What makes you say that they are in decline?
- eeegnu 5y ago
  I don't see how this conclusion could follow from a bill limit feature not ever being implemented. That would only make sense if this was about them removing such a feature.
- neximo64 5y ago
  I feel like Amazon's teams have actually the opposite mindset of this if you actually use their product/services. No affiliation to amazon whatsoever saying this.
- mkl 5y ago
  So you think they were already declining ten years ago, and still are? Amazon's market cap has gone up by a factor of 20 in that time.
tjoff 5y ago
Don't give companies that does this money.
- arthurcolle 5y ago
  Sorry, just a nit pick but I think the grammar for this should be: * Don't give companies that do this money
- toomanybeersies 5y ago
  Is there any cloud provider that actually implements a hard cap on billing?
  - BackBlast 5y ago
    Vultr allows you to use a pure pre-pay option. You can even pay with crypto and forgo the credit card. Your account gets shut down when you run out of funds.
Hamuko 5y ago
It's never too late. 21 years after being reported, Firefox finally has native macOS context menus in the nightly version. https://bugzilla.mozilla.org/show_bug.cgi?id=34572 https://bugzilla.mozilla.org/show_bug.cgi?id=34572
- Austin_Conlon 5y ago
  "i can't argue, but it's a time thing."
- lloeki 5y ago
  Ah, maybe in 10 years they'll use the native file pickers.
  - mjevans 5y ago
    That would be wonderful on Linux... I would really really love to use my desktop environment's picker which integrates with the rest of the experience and isn't the preschooler file picker UI that firefox defaults to.
- tgv 5y ago
  I've been using Firefox for a long time (first in its "Camino" guise), but I've never noticed.
guywhocodes 5y ago
If this happens it will happen service by service as it's most definitely not been a design criteria. I've worked on account leasing systems for temporary accounts and talked to the internal team for AWS event engine. And they don't have much more than the official tools. Edit: not been..
lend000 5y ago
Do Google/Azure have this?
- amanzi 5y ago
  Not sure about Google, but Azure has it with some subscription types but not all. I have an Azure subscription that gets some free monthly credits. When I go over the limit on that subscription, everything freezes until I sort it out.
  - jiggawatts 5y ago
    Microsoft only cared to implement spending caps when it affects their bottom line.
- aoetalks 5y ago
  Most Azure services don’t, but Azure Monitor does. I guess it’s easy to deploy something that emits tons of logs or metrics inadvertently, so having a quota might be nice. I think it’s tricky to implement, though.
- gnopgnip 5y ago
  GCP has billing alerts, and a cap. But it can take up to 24 hours after the cap is hit before you stop receiving new charges in some case so it isn't perfect.
runnerup 5y ago
My understanding is that this is a difficult problem to solve "perfectly" due to lag between incurring a cost and recording the cost. I believe GCP currently has the best feature for this. You can set both billing alerts as well as caps. However, I also believe that it can take up to 24 hours between incurring a cost and it showing up on your billing report. So even on GCP (which is the most forward-thinking cloud service for this feature), you can incur up to 24 hours of charges over your maximum billing threshold cutoff. I'm also not 100% sure if GCP's billing threshold is really designed to be a "hard cap" per se. The real question is whether AWS should let perfect be the enemy of good; and/or whether providing a somewhat "broken" service like GCP's would mislead customers into feeling more protected than they actually are. See here where someone set a Firebase billing budget of $7 but an infinite recursion generated $72,000 in charges. When the founders started seeing the charges come in, all they could do was watch as it grew and grew....because their screen was merely reflecting what had already happened in hours past. https://www.theregister.com/2020/12/10/google_cloud_over_run/ https://www.theregister.com/2020/12/10/google_cloud_over_run... Discussed here: https://news.ycombinator.com/item?id=25398148 https://news.ycombinator.com/item?id=25398148
- andrewguenther 5y ago
  GCP doesn't support a cap, only alerts. You can use those alerts to implement your own cap mechanism, same as how it works on AWS except AWS billing is only on a one hour delay I believe so I'd say AWS wins here.
  - runnerup 5y ago
    Indeed, here is GCP's own guide which confirms what you say and provides a hacky way to implement a cap mechanism. https://cloud.google.com/billing/docs/how-to/notify#cap_disable_billing_to_stop_usage https://cloud.google.com/billing/docs/how-to/notify#cap_disa...
  - Agingcoder 5y ago
    We talked about caps with the Google reps at my day job. The short answer was 'we can, but don't want to' (note : this may be completely unrelated to what Google thinks internally, and is just what the fairly high up the food chain rep told us)
andrewguenther 5y ago
It hasn't been implemented because it is nonsensical. So you hit your limit. Is all your storage deleted? You can't receive an alert because that costs something (even if it's a fraction of a cent) to send. Are your domains forfeit? Audit logs destroyed? There's no reasonable way to implement this. Billing alerts are the best you can do on this problem and I think it would be a good faith move for AWS to enable some by default but a limit just doesn't make sense on any level. EDIT: Lots of people proposing solutions that work for them. AWS has to think of everyone. And they did. That's why budget alerts exist and you can respond in the way you choose. Everyone's conflicting ideas for how to solve this can be implemented today on top of billing alerts/actions[1]. Case closed. [1] https://aws.amazon.com/blogs/aws-cost-management/get-started-with-aws-budgets-actions/ https://aws.amazon.com/blogs/aws-cost-management/get-started...
- viraptor 5y ago
  Resources can be divided into persistent and not. People could opt into "we'll cut off traffic and prevent spawning new resources" limit without affecting storage, domains, logs, etc. This also aligns well with the "planned vs unexpected" costs.
- darkwizard42 5y ago
  I feel like this response has to be said every time this sort of thread comes up when someone discusses their unexpectedly large bill. The worst case scenario is that the user configures their limits in a poor way then turns around and blames Amazon when something breaks in their project and Amazon points back at the limit. It would just be bad all around...so all in all, I think alerts are the best way to do this...
- heipei 5y ago
  At the very least allow users to roughly define what happens when the threshold is reached: S3 will still store data but maybe not serve anything, Lambdas will still be defined but not run, EC2 instances will keep running, etc. It's not supposed to be a hard cap as in "don't spend a penny more than my $5" but more as in "do everything possible and reasonable so the customer doesn't wake up to a sudden $20k overnight AWS bill".
ablekh 5y ago
This is ironic in a very sad way, considering that Amazon declares customer obsession their top leadership principle [1]. [1] https://www.amazon.jobs/en/principles https://www.amazon.jobs/en/principles
- varispeed 5y ago
  That seems to be an obsession, but in an abusive vein. Here have so many tools and millions of options we made for you, just be careful, it would be a shame if you had to mortgage your house if you forgot to configure something. But we are here for you.
ghoomketu 5y ago
Yes this would be a super useful feature especially for dormant accounts. One of my old aws account was recently hacked and racked up bills in thousands of dollars. The AWS support was super nice and took care of it for me but it gave me some sleepless nights until it was resolved. If this was a feature I would have most definitely capped my account at 1k (my max bill being $50). They can always send alerts to increase your cap based on projections like it shows in cost explorer. Also since it's an opt-in feature I'm kinda taking responsibility that I'm okay if services are suspended due to hitting the caps.
- thitcanh 5y ago
  They could easily factor natural growth and ping you when it feels like bills are getting too close to the cap. Even “bursts” could be allowed x% above the cap if you get an email within an hour. The lack of caps is just laziness or a dark pattern by Amazon. There’s no excuse to not implement them, it’s not a hard problem.
aoetalks 5y ago
From the service side...What happens when the user quota resets? You see a flood of traffic to the service. That might trigger anomaly alerts or you run out of capacity. Another issue is how often does the quota reset? Daily? Hourly? Monthly? That’ll determine how big those spikes are when the quota lifts.
kioleanu 5y ago
Doesn't Google Cloud have a cap and it's completely worthless? https://dev-blog.tomilkieway.com/72k-1/ https://dev-blog.tomilkieway.com/72k-1/
- varispeed 5y ago
  If AWS doesn't do it, then GC has no incentive to do it either. I think this should be mandated by law, so all companies will have to introduce caps that actually work. If the customer sets $100, this is the maximum they should pay, regardless if the operator has any delays or other problems. It should be their problem. I think once the law mandates it, they'll quickly find a solution.
867-5309 5y ago
>Seriously, 10 years unanswered... amazing. You can add +1 to customers lost due to this problem. that's but a drop in the digital ocean
cr1pablo 5y ago
As a junior developer I'm always afraid using any AWS service to make a mistake that cost me a lot of money. I'm not talking about millions, simply 2k would be really bad for me.
- GrantZvolsky 5y ago
  I had been in the same position for about ten years until I set up a limited liability company. It does take some effort to set up and costs about $20/month[1], but to me the peace of mind is well worth it. [1]: https://www.ukpostbox.com/address/business-address-service https://www.ukpostbox.com/address/business-address-service
  - _0o6v 5y ago
    A limited liability company may mean you're personally not liable, but your business can still be made bankrupt which isn't much better.
  - Sohcahtoa82 5y ago
    The cost to own an LLC varies a lot on jurisdiction. Even in the USA, it can vary considerably from state to state.
    - petecoop 5y ago
      Also they link to a business address service, which isn't a business. Actual cost of just owning a business in the UK is £13/year
  - varispeed 5y ago
    This is insane that a user has to setup a separate company just to protect themselves from sudden bills that may be not their fault.
- oliwarner 5y ago
  Anyone who is personally liable for the the bill should be terrified of AWS. It's enough to lose your business to a usage spike, much worse to lose your home and car. I struggle to consider a scenario where setting up a limited liability organisation wouldn't make sense, even if only for interacting with Amazon.
forty 5y ago
I'm told some really fancy restaurants would not show the prices on the rational that "if you need to know the price, then it's too expensive for you" Well, I think the same point can be made here: if you need a price cap, then AWS is too expensive for you ;)
- deleted 5y ago
  [deleted]
- thitcanh 5y ago
  I use a few free APIs with usage limits. Maybe I pay 10 cents a year. I’m always afraid to one time open an invoice to find out it’s way more than that for whatever reason. The problem is that AWS markets itself as “pay what you use” and generally that’s extremely little for me, but if I screw up with Glacier I suppose I could get a $100 bill because I restored some backups the wrong way.
  - toomanybeersies 5y ago
    > but if I screw up with Glacier I suppose I could get a $100 bill because I restored some backups the wrong way. You wouldn't be the first person to do that: https://medium.com/@karppinen/how-i-ended-up-paying-150-for-a-single-60gb-download-from-amazon-glacier-6cb77b288c3e https://medium.com/@karppinen/how-i-ended-up-paying-150-for-...
- NoPicklez 5y ago
  Well that's just rubbish. The whole idea of services like AWS is that it is scalable as a solution for startups, small businesses and large global corporations. Not just large corporations that would be considered customers of a "fancy" web services provider.
  - forty 5y ago
    Sorry about that, I thought the ;) would have made it clear that this is a joke.
ddtaylor 5y ago
This is one of the reasons I really like Vultr. They have a simple interface and their pricing has always been solid for me. I don't ever go to sleep worried I'm going to wake up with a $10k bill. At one of the companies years ago a developer accidentally leaked an API credential on GitHub and the company woke up with a $30k bill.
fabian2k 5y ago
I understand that there's an endless number of edge cases that are hard or impossible to solve here. But the unlimited potential charges eliminate AWS for certain scenarios. I would not use AWS for private stuff because of this, it is simply not worth it to risk huge bills to me personally. Of course what I might spend on AWS is peanuts, but it could translate into using AWS with my employer because I'm familiar with certain parts of it. And even professionally this is kind of a risk, especially if you're working for a small company. And even more if you're not that experienced with AWS, there's a lot of parts to it and it can be really hard to figure out what is costing you money exactly, if you're not an AWS expert. I once created a relatively small, but noticeable charge on AWS due to a recurring script transferring a lot more data than intended. With my limited knowledge it was pretty much impossible to figure out the exact source from AWS tools. I more or less stumbled upon the issue randomly while looking into this, which is kinda scary to me. All this does make AWS less attractive compared to classic hosting/renting a server, if that fits your use case and the lesser flexibility isn't an issue.
alkonaut 5y ago
+1 will never use a cloud servcie that doesn't have a hard cap. Not just alerts. Hard cap.
mjevans 5y ago
Back of the napkin algorithm: Borrow what DHCP does for leases, but with payment quotas. First layer is a burst quota, up to that maximum in any spike. This only regenerates when the (monthly?) average quota has enough slack at current that an additional burst can be harvested without the remaining average being under the initial set value. (The math would probably be different, remaining period quota less burst average over time greater than period quota average, but that's the sales description of the concept.) The second layer is the monthly (or some other period) average for the maximum expenditure allowed. A billing endpoint would maintain a fractional bucket of spending (divided up as makes sense) but in the case of a single quota consumer would receive an estimate for the period (ideally the burst) and allow up to half of that to be used. At that point it will 'renew' the quota (lease), and flush billing, including sending alerts. If there isn't any further quota the remaining released balance would be consumed and then requests fail.
Zealotux 5y ago
I'm using AWS for my side-project and while I enjoy how cheap it can be, I'm also terrified of it and always have the Cost Explorer opened in a tab that I check every day.
elongatedMusku 5y ago
Build your own monitoring services. lol.
bkovacev 5y ago
If anyone from AWS is reading this - is there any way we can get multiple devices for 2FA?
haywirez 5y ago
This finally answers my bafflement over why my requests for a transfer cap feature have been ignored by all thin-second-layer providers of the Vercel etc. type...
aranelsurion 5y ago
I believe at the very least they could offer a more flexible way of managing Service Quotas. Currently it's too much of a hassle to increase/decrease account-wide limits: it takes too much time, depends on AWS Support, UX is nowhere as nice as, say instance reservations etc. As a result of this people tend to increase the limits once in a while just to make sure it doesn't become an issue at a very unfortunate time, and that's it. It never gets decreased/managed again. It's obviously not seen as a use-case by AWS, since they only have a "Request quota increase" button in the UI, and no "decrease" button. [1] SQ on its own wouldn't cover all kinds of costs (for on-demand items like Lambda executions they offer limits on rates, not total count), but it'd be better than nothing, it'd prevent scenarios where you'd wake up to 100 GPU instances mining bitcoin, and it's within the quotas since Bob asked for an increase two years ago to try something. [1] Just having a button there would still be an awful UX, yet I believe its complete omission is noteworthy.
auggierose 5y ago
It's very clear why they are not doing it. Putting the risk on you instead of themselves a) is better for them for obvious reasons, and b) makes it less likely for something bad to happen in the first place. Also, I can imagine clients worried about this are not the money-making kind of clients anyway.
- varispeed 5y ago
  And if you put it this way, I am not sure why this is not an illegal practice? Companies shouldn't discriminate the access to their services based on the level of anxiety you can handle.
  - deleted 5y ago
    [deleted]
  - auggierose 5y ago
    I don't think that's a law.
varispeed 5y ago
That's why I don't use AWS. I am sure I forget to disable or enable something and then wake up with life ruining bill. Since they don't want to fix it, it seems like this is still a part of their business model. Maybe this business requires more regulation - if company cannot resolve an issue negatively impacting the consumer, then government should step in and mandate bill capping features to be implemented. Maybe we should start writing to our representatives to come up with something, so companies like Amazon know their place.
londons_explore 5y ago
I understand that designing a real-time billing system is near impossible. But couldn't Google/Amazon simply pretend they have real-time bill capping, and then simply swallow the costs incurred by any delays? Doesn't seem like rocket science, probably won't cost much, and might bring in new kinds of business (a lot of businesses won't allow most employees to sign an effectively blank check)
- sigotirandolas 5y ago
  To avoid penalizing customers who don't use it, they could make you incur an instant overcharge fee when you go over your limit, say 5% of your limit, which would be pooled to compensate for any extra cost Amazon could need to incur to salvage customer data. Like an insurance, fundamentally. Now that I think about it, a third party could do this as well... except that if it were built in to AWS there would be much less friction.
whoknew1122 5y ago
Disclaimer: Work for AWS Support. Good and bad things about my employer. Opinions my own. One of the biggest questions here is what are customers asking for? And the necessary follow-up question: Do customers want to actually bear responsibility for their choices? Recently, I had a customer lodge a support case because their programmatic access keys were leaked. A malicious actor was then able to use those credentials to exfiltrate their S3 data, and delete it. Now the S3 data is being ransomed. The customer opened a case asking if there was any way to get the data back. If we had to go the 'extra mile', the customer demanded we do that. The answer is simply: No. We can't get that data back. Customers demand that they own the data they upload into S3. They don't want AWS to be able to read the data, nor do they want us to store the data internally as a backup. That's what customers demand, and what AWS gives them. Now something regrettable happened (a customer got pwned) and now a customer wants AWS to bear the responsibility for backing up the data. I bet a week ago they would've demanded that they have absolute sovereignty over their data. Shit changes when shit hits the fan. --- Hard caps are doable, but the question is: Do customers want responsibility for this feature? I read a comment wherein someone's startup was killed due to $30k of bandwidth costs because of misconfigurations that led to users abusing their platform. That sucks. It's not in AWS's interest to put their customers out of business. But let's look at the flip side. What if someone in the finance department puts a hard cap on an account. But then a tweet goes viral, and business is pouring in. Everyone, in their rush to keep everything scaling appropriately, forgets there's a hard cap. They hit the business and now there's a hard outage in the middle of the biggest business event they've ever had. Who's the customer going to hold responsible for the outage? As someone who deals with customer misconfigurations on AWS for a living, I assure you the customer won't be calling themselves demanding an explanation. What happens if the person in charge of budgets doesn't lift the cap before the Christmas holidays? Again, hard down. Again, customer won't be mad at themselves. --- It's better to let AWS run as the customer configures it to run than bring everything to an abrupt stop. Billing alerts exist, and you can use Lambda to turn off resources that are above their billing threshold. But doing a hard cap on an account? Something a subsection of customers might want, but something not many customers will want to take responsibility for when something goes sideways.
zadkey 5y ago
I don't think they are going to implement it. It's too juicy financially to do so. Furthermore, it's bad PR to say they aren't going to do it. So they will keep stringing us along as long as they can. Well, that's what the incentives would suggest.
tyrex2017 5y ago
Many comments say it cant be done, but this is disingenious. As if we were bureaucrats with no imagination: Eg just have 2 caps: First cap triggers a red alarm prohibiting provisioning, or storing, of new data. The second one (when you are like 50% above the first cap) is dark red, closing down everything except storage. Sure, decisions have to be made for every single service, but you don’t need a perfect solution if it is optional. Eg, you could start ONLY with EC2 and will have fixed 50% of the problem.
Elect2 5y ago
AWS should do it at least on bandwidth cost.

358 points by shikkra 5y ago

You really need to implement limits service-by-service, and with as many services as AWS has, it would make sense if some of them hadn't implemented limits yet. Think about the details... if you are paying something for both storage and bandwidth, and get a sudden surge of bandwidth, do you really want items in storage to be deleted? You basically never want storage to be automatically deleted even if your program suddenly uses a surprising amount; limits on its maximum size and alerts are much better. But once you realize that bill capping doesn't make sense for storage, well, many different services are essentially some type of storage. This is a feature that sounds good but in practice what people really need is something slightly different.

akira2501 5y ago
> But once you realize that bill capping doesn't make sense for storage I'd think you want something like a circuit breaker.. steady costs or slowly increasing costs aren't like to be a problem, but a sudden surge in costs is what I'm concerned with. Perhaps if it operated like a time-averaged quota.. don't let me incur _new_ costs if I slide too far out of my apparent range. Give me a knob to control that range or temporarily disable it, and maybe a way to monitor those events so I can react to them appropriately for my particular application.
capableweb 5y ago
> You basically never want storage to be automatically deleted even if your program suddenly uses a surprising amount In most use cases you're right. But never? Not true, not all storage is used the same way. And the thing is, why should Amazon decide which use case is valid or not? I might care more about being able to afford the service than to keep the data around, depending on what kind of service I offer my user. As a platform provider, they should be agnostic, but their greed for money is shining through their willingness to be a true platform provider.
- philliphaydon 5y ago
  There are rules in S3 to move files or delete them after a period of time.
  - Hamuko 5y ago
    And perhaps most useful rule is to move them to cheaper tiers of storage. For example, Glacier Deep Archive is super cheap for storage since it's $0.99 per terabyte-month.
p1necone 5y ago
I don't think you ever want storage to be deleted as part of an automatic bill cap system. Just refuse access, or allow reads but not writes until the customer reviews it. There's even an HTTP code for it: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/402 https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/402 (non standard, but I know for a fact it's used by Azure). Actually deleting things should be done after a period of time specified in writing in the contract with the customer, and I'd hope you'd try to contact them first. But I think an automatic bill cap absolutely makes sense for storage, and I can't really see any reasons it couldn't work.
- deleted 5y ago
  [deleted]
- nucleardog 5y ago
  > and I can't really see any reasons it couldn't work. You’re assuming the existence of a function which can extrapolate the daily/monthly/etc costs. If I upload a 10GB file at $0.10/gbmo, how much will it cost me? We have no idea because we don’t know how long it will be up there. We have a process that downloads and streams massive reports into S3, then a process is immediately kicked off to handle processing and importing. As soon as that completes, the files are deleted. So we’re paying about 0.04 months of storage every month. The naive solution is estimating 1.00. You’re off by a factor of ~24. We could... not handle that. Now we’re in a situation where we’re consistently spending $100/mo on storage, set a liberal $150 budget and immediately everything breaks. We actually needed to set a $2400 budget! Expecting a $100 bill and getting a $2400 bill some month when something goes wrong is as good as doing nothing. You’re also going to need to account for the interaction of every single object in a bucket and the lifecycle rules set which is an absolutely ridiculous amount of overhead. Now extend this to... literally every one of the seemingly bazillion AWS services. The only cap that’s going to be easy and realistic to implement without a crystal ball is a hard cut-off once you’ve actually accrued the charges, and a “shut down all my servers, delete all my data, and just close my account” cap is useful to almost no one. If you want useful caps they need to be aware of your workload. The best person to implement those is you. The tools are there.
forty 5y ago
Exactly. I have heard stories of bill capping that ended up with infra and data being deleted, so I can really see why they wouldn't implement it. What they could do however would be to allow usage limit based on IAM policies for example (Deny s3:Getobject Condition: monthly bandwidth >= x)

karmasimida 5y ago

How is this going to be implemented? When it reach the cap, all your service stop working? This is incredibly user hostile. If you want some warning when utilization is high, it can be relatively easy to setup with AWS's existing feature, but you need to do some configuration/tweaking on your own.

btilly 5y ago
When it reach the cap, all your service stop working? This is incredibly user hostile. Not nearly as user hostile as the possibility of an unexpected DOS on your bank account while you're asleep. Which has actually happened to people.
- handmodel 5y ago
  My experience is just one data point but: My expertise is not at all anything related to web development so when I first tried experimenting with AWS for a side project I was terrified. It was an experiment for me and i didn't care if it deleted the data since the data it was storing was test data I was uploading. However, my budget for my entire side project was $500 and it seemed totally feasible (at least to a first time user of such a service) that some wrong lines in the code could balloon the requests or storage I made. At the time, I didn't even have $5000 of liquid assets and it would have been a nightmare.
technion 5y ago
That would be substantively less user hostile to me than a massive bill on a personal project, which is easy to see occurring for any number of reasons.
exactlysolved 5y ago
> When it reach the cap, all your service stop working? This is incredibly user hostile. So set your cap to a very high number, much higher than any amount you expect to be billed. Surely for everyone there is some number at which they would rather their services go down, than have to pay it?
varispeed 5y ago

e_commerce 5y ago

They're on the decline and are now in the "extract every last cent" phase of the organization.

samzer 5y ago
What makes you say that they are in decline?
eeegnu 5y ago
I don't see how this conclusion could follow from a bill limit feature not ever being implemented. That would only make sense if this was about them removing such a feature.
neximo64 5y ago
I feel like Amazon's teams have actually the opposite mindset of this if you actually use their product/services. No affiliation to amazon whatsoever saying this.
mkl 5y ago
So you think they were already declining ten years ago, and still are? Amazon's market cap has gone up by a factor of 20 in that time.

tjoff 5y ago

Don't give companies that does this money.

arthurcolle 5y ago
Sorry, just a nit pick but I think the grammar for this should be: * Don't give companies that do this money
toomanybeersies 5y ago
Is there any cloud provider that actually implements a hard cap on billing?
- BackBlast 5y ago
  Vultr allows you to use a pure pre-pay option. You can even pay with crypto and forgo the credit card. Your account gets shut down when you run out of funds.

Hamuko 5y ago

It's never too late. 21 years after being reported, Firefox finally has native macOS context menus in the nightly version. https://bugzilla.mozilla.org/show_bug.cgi?id=34572 https://bugzilla.mozilla.org/show_bug.cgi?id=34572

Austin_Conlon 5y ago
"i can't argue, but it's a time thing."
lloeki 5y ago
Ah, maybe in 10 years they'll use the native file pickers.
- mjevans 5y ago
  That would be wonderful on Linux... I would really really love to use my desktop environment's picker which integrates with the rest of the experience and isn't the preschooler file picker UI that firefox defaults to.
tgv 5y ago
I've been using Firefox for a long time (first in its "Camino" guise), but I've never noticed.

guywhocodes 5y ago

If this happens it will happen service by service as it's most definitely not been a design criteria. I've worked on account leasing systems for temporary accounts and talked to the internal team for AWS event engine. And they don't have much more than the official tools. Edit: not been..

lend000 5y ago

Do Google/Azure have this?

amanzi 5y ago
Not sure about Google, but Azure has it with some subscription types but not all. I have an Azure subscription that gets some free monthly credits. When I go over the limit on that subscription, everything freezes until I sort it out.
- jiggawatts 5y ago
  Microsoft only cared to implement spending caps when it affects their bottom line.
aoetalks 5y ago
Most Azure services don’t, but Azure Monitor does. I guess it’s easy to deploy something that emits tons of logs or metrics inadvertently, so having a quota might be nice. I think it’s tricky to implement, though.
gnopgnip 5y ago
GCP has billing alerts, and a cap. But it can take up to 24 hours after the cap is hit before you stop receiving new charges in some case so it isn't perfect.

runnerup 5y ago

My understanding is that this is a difficult problem to solve "perfectly" due to lag between incurring a cost and recording the cost. I believe GCP currently has the best feature for this. You can set both billing alerts as well as caps. However, I also believe that it can take up to 24 hours between incurring a cost and it showing up on your billing report. So even on GCP (which is the most forward-thinking cloud service for this feature), you can incur up to 24 hours of charges over your maximum billing threshold cutoff. I'm also not 100% sure if GCP's billing threshold is really designed to be a "hard cap" per se. The real question is whether AWS should let perfect be the enemy of good; and/or whether providing a somewhat "broken" service like GCP's would mislead customers into feeling more protected than they actually are. See here where someone set a Firebase billing budget of $7 but an infinite recursion generated $72,000 in charges. When the founders started seeing the charges come in, all they could do was watch as it grew and grew....because their screen was merely reflecting what had already happened in hours past. https://www.theregister.com/2020/12/10/google_cloud_over_run/ https://www.theregister.com/2020/12/10/google_cloud_over_run... Discussed here: https://news.ycombinator.com/item?id=25398148 https://news.ycombinator.com/item?id=25398148

andrewguenther 5y ago
GCP doesn't support a cap, only alerts. You can use those alerts to implement your own cap mechanism, same as how it works on AWS except AWS billing is only on a one hour delay I believe so I'd say AWS wins here.
- runnerup 5y ago
  Indeed, here is GCP's own guide which confirms what you say and provides a hacky way to implement a cap mechanism. https://cloud.google.com/billing/docs/how-to/notify#cap_disable_billing_to_stop_usage https://cloud.google.com/billing/docs/how-to/notify#cap_disa...
- Agingcoder 5y ago
  We talked about caps with the Google reps at my day job. The short answer was 'we can, but don't want to' (note : this may be completely unrelated to what Google thinks internally, and is just what the fairly high up the food chain rep told us)

andrewguenther 5y ago

It hasn't been implemented because it is nonsensical. So you hit your limit. Is all your storage deleted? You can't receive an alert because that costs something (even if it's a fraction of a cent) to send. Are your domains forfeit? Audit logs destroyed? There's no reasonable way to implement this. Billing alerts are the best you can do on this problem and I think it would be a good faith move for AWS to enable some by default but a limit just doesn't make sense on any level. EDIT: Lots of people proposing solutions that work for them. AWS has to think of everyone. And they did. That's why budget alerts exist and you can respond in the way you choose. Everyone's conflicting ideas for how to solve this can be implemented today on top of billing alerts/actions[1]. Case closed. [1] https://aws.amazon.com/blogs/aws-cost-management/get-started-with-aws-budgets-actions/ https://aws.amazon.com/blogs/aws-cost-management/get-started...

viraptor 5y ago
Resources can be divided into persistent and not. People could opt into "we'll cut off traffic and prevent spawning new resources" limit without affecting storage, domains, logs, etc. This also aligns well with the "planned vs unexpected" costs.
darkwizard42 5y ago
I feel like this response has to be said every time this sort of thread comes up when someone discusses their unexpectedly large bill. The worst case scenario is that the user configures their limits in a poor way then turns around and blames Amazon when something breaks in their project and Amazon points back at the limit. It would just be bad all around...so all in all, I think alerts are the best way to do this...
heipei 5y ago
At the very least allow users to roughly define what happens when the threshold is reached: S3 will still store data but maybe not serve anything, Lambdas will still be defined but not run, EC2 instances will keep running, etc. It's not supposed to be a hard cap as in "don't spend a penny more than my $5" but more as in "do everything possible and reasonable so the customer doesn't wake up to a sudden $20k overnight AWS bill".

ablekh 5y ago

This is ironic in a very sad way, considering that Amazon declares customer obsession their top leadership principle [1]. [1] https://www.amazon.jobs/en/principles https://www.amazon.jobs/en/principles

varispeed 5y ago
That seems to be an obsession, but in an abusive vein. Here have so many tools and millions of options we made for you, just be careful, it would be a shame if you had to mortgage your house if you forgot to configure something. But we are here for you.

ghoomketu 5y ago

Yes this would be a super useful feature especially for dormant accounts. One of my old aws account was recently hacked and racked up bills in thousands of dollars. The AWS support was super nice and took care of it for me but it gave me some sleepless nights until it was resolved. If this was a feature I would have most definitely capped my account at 1k (my max bill being $50). They can always send alerts to increase your cap based on projections like it shows in cost explorer. Also since it's an opt-in feature I'm kinda taking responsibility that I'm okay if services are suspended due to hitting the caps.

thitcanh 5y ago
They could easily factor natural growth and ping you when it feels like bills are getting too close to the cap. Even “bursts” could be allowed x% above the cap if you get an email within an hour. The lack of caps is just laziness or a dark pattern by Amazon. There’s no excuse to not implement them, it’s not a hard problem.

aoetalks 5y ago

From the service side...What happens when the user quota resets? You see a flood of traffic to the service. That might trigger anomaly alerts or you run out of capacity. Another issue is how often does the quota reset? Daily? Hourly? Monthly? That’ll determine how big those spikes are when the quota lifts.

kioleanu 5y ago

Doesn't Google Cloud have a cap and it's completely worthless? https://dev-blog.tomilkieway.com/72k-1/ https://dev-blog.tomilkieway.com/72k-1/

varispeed 5y ago
If AWS doesn't do it, then GC has no incentive to do it either. I think this should be mandated by law, so all companies will have to introduce caps that actually work. If the customer sets $100, this is the maximum they should pay, regardless if the operator has any delays or other problems. It should be their problem. I think once the law mandates it, they'll quickly find a solution.

867-5309 5y ago

>Seriously, 10 years unanswered... amazing. You can add +1 to customers lost due to this problem. that's but a drop in the digital ocean

cr1pablo 5y ago

As a junior developer I'm always afraid using any AWS service to make a mistake that cost me a lot of money. I'm not talking about millions, simply 2k would be really bad for me.

GrantZvolsky 5y ago
I had been in the same position for about ten years until I set up a limited liability company. It does take some effort to set up and costs about $20/month[1], but to me the peace of mind is well worth it. [1]: https://www.ukpostbox.com/address/business-address-service https://www.ukpostbox.com/address/business-address-service
- _0o6v 5y ago
  A limited liability company may mean you're personally not liable, but your business can still be made bankrupt which isn't much better.
- Sohcahtoa82 5y ago
  The cost to own an LLC varies a lot on jurisdiction. Even in the USA, it can vary considerably from state to state.
  - petecoop 5y ago
    Also they link to a business address service, which isn't a business. Actual cost of just owning a business in the UK is £13/year
- varispeed 5y ago
  This is insane that a user has to setup a separate company just to protect themselves from sudden bills that may be not their fault.
oliwarner 5y ago
Anyone who is personally liable for the the bill should be terrified of AWS. It's enough to lose your business to a usage spike, much worse to lose your home and car. I struggle to consider a scenario where setting up a limited liability organisation wouldn't make sense, even if only for interacting with Amazon.

forty 5y ago

I'm told some really fancy restaurants would not show the prices on the rational that "if you need to know the price, then it's too expensive for you" Well, I think the same point can be made here: if you need a price cap, then AWS is too expensive for you ;)

deleted 5y ago
[deleted]
thitcanh 5y ago
I use a few free APIs with usage limits. Maybe I pay 10 cents a year. I’m always afraid to one time open an invoice to find out it’s way more than that for whatever reason. The problem is that AWS markets itself as “pay what you use” and generally that’s extremely little for me, but if I screw up with Glacier I suppose I could get a $100 bill because I restored some backups the wrong way.
- toomanybeersies 5y ago
  > but if I screw up with Glacier I suppose I could get a $100 bill because I restored some backups the wrong way. You wouldn't be the first person to do that: https://medium.com/@karppinen/how-i-ended-up-paying-150-for-a-single-60gb-download-from-amazon-glacier-6cb77b288c3e https://medium.com/@karppinen/how-i-ended-up-paying-150-for-...
NoPicklez 5y ago
Well that's just rubbish. The whole idea of services like AWS is that it is scalable as a solution for startups, small businesses and large global corporations. Not just large corporations that would be considered customers of a "fancy" web services provider.
- forty 5y ago
  Sorry about that, I thought the ;) would have made it clear that this is a joke.

ddtaylor 5y ago

This is one of the reasons I really like Vultr. They have a simple interface and their pricing has always been solid for me. I don't ever go to sleep worried I'm going to wake up with a $10k bill. At one of the companies years ago a developer accidentally leaked an API credential on GitHub and the company woke up with a $30k bill.

fabian2k 5y ago

I understand that there's an endless number of edge cases that are hard or impossible to solve here. But the unlimited potential charges eliminate AWS for certain scenarios. I would not use AWS for private stuff because of this, it is simply not worth it to risk huge bills to me personally. Of course what I might spend on AWS is peanuts, but it could translate into using AWS with my employer because I'm familiar with certain parts of it. And even professionally this is kind of a risk, especially if you're working for a small company. And even more if you're not that experienced with AWS, there's a lot of parts to it and it can be really hard to figure out what is costing you money exactly, if you're not an AWS expert. I once created a relatively small, but noticeable charge on AWS due to a recurring script transferring a lot more data than intended. With my limited knowledge it was pretty much impossible to figure out the exact source from AWS tools. I more or less stumbled upon the issue randomly while looking into this, which is kinda scary to me. All this does make AWS less attractive compared to classic hosting/renting a server, if that fits your use case and the lesser flexibility isn't an issue.

alkonaut 5y ago

+1 will never use a cloud servcie that doesn't have a hard cap. Not just alerts. Hard cap.

mjevans 5y ago

Back of the napkin algorithm: Borrow what DHCP does for leases, but with payment quotas. First layer is a burst quota, up to that maximum in any spike. This only regenerates when the (monthly?) average quota has enough slack at current that an additional burst can be harvested without the remaining average being under the initial set value. (The math would probably be different, remaining period quota less burst average over time greater than period quota average, but that's the sales description of the concept.) The second layer is the monthly (or some other period) average for the maximum expenditure allowed. A billing endpoint would maintain a fractional bucket of spending (divided up as makes sense) but in the case of a single quota consumer would receive an estimate for the period (ideally the burst) and allow up to half of that to be used. At that point it will 'renew' the quota (lease), and flush billing, including sending alerts. If there isn't any further quota the remaining released balance would be consumed and then requests fail.

Zealotux 5y ago

I'm using AWS for my side-project and while I enjoy how cheap it can be, I'm also terrified of it and always have the Cost Explorer opened in a tab that I check every day.

elongatedMusku 5y ago

Build your own monitoring services. lol.

bkovacev 5y ago

If anyone from AWS is reading this - is there any way we can get multiple devices for 2FA?

haywirez 5y ago

This finally answers my bafflement over why my requests for a transfer cap feature have been ignored by all thin-second-layer providers of the Vercel etc. type...

aranelsurion 5y ago

I believe at the very least they could offer a more flexible way of managing Service Quotas. Currently it's too much of a hassle to increase/decrease account-wide limits: it takes too much time, depends on AWS Support, UX is nowhere as nice as, say instance reservations etc. As a result of this people tend to increase the limits once in a while just to make sure it doesn't become an issue at a very unfortunate time, and that's it. It never gets decreased/managed again. It's obviously not seen as a use-case by AWS, since they only have a "Request quota increase" button in the UI, and no "decrease" button. [1] SQ on its own wouldn't cover all kinds of costs (for on-demand items like Lambda executions they offer limits on rates, not total count), but it'd be better than nothing, it'd prevent scenarios where you'd wake up to 100 GPU instances mining bitcoin, and it's within the quotas since Bob asked for an increase two years ago to try something. [1] Just having a button there would still be an awful UX, yet I believe its complete omission is noteworthy.

auggierose 5y ago

It's very clear why they are not doing it. Putting the risk on you instead of themselves a) is better for them for obvious reasons, and b) makes it less likely for something bad to happen in the first place. Also, I can imagine clients worried about this are not the money-making kind of clients anyway.

varispeed 5y ago
And if you put it this way, I am not sure why this is not an illegal practice? Companies shouldn't discriminate the access to their services based on the level of anxiety you can handle.
- deleted 5y ago
  [deleted]
- auggierose 5y ago
  I don't think that's a law.

varispeed 5y ago

That's why I don't use AWS. I am sure I forget to disable or enable something and then wake up with life ruining bill. Since they don't want to fix it, it seems like this is still a part of their business model. Maybe this business requires more regulation - if company cannot resolve an issue negatively impacting the consumer, then government should step in and mandate bill capping features to be implemented. Maybe we should start writing to our representatives to come up with something, so companies like Amazon know their place.

londons_explore 5y ago

I understand that designing a real-time billing system is near impossible. But couldn't Google/Amazon simply pretend they have real-time bill capping, and then simply swallow the costs incurred by any delays? Doesn't seem like rocket science, probably won't cost much, and might bring in new kinds of business (a lot of businesses won't allow most employees to sign an effectively blank check)

sigotirandolas 5y ago
To avoid penalizing customers who don't use it, they could make you incur an instant overcharge fee when you go over your limit, say 5% of your limit, which would be pooled to compensate for any extra cost Amazon could need to incur to salvage customer data. Like an insurance, fundamentally. Now that I think about it, a third party could do this as well... except that if it were built in to AWS there would be much less friction.

whoknew1122 5y ago

Disclaimer: Work for AWS Support. Good and bad things about my employer. Opinions my own. One of the biggest questions here is what are customers asking for? And the necessary follow-up question: Do customers want to actually bear responsibility for their choices? Recently, I had a customer lodge a support case because their programmatic access keys were leaked. A malicious actor was then able to use those credentials to exfiltrate their S3 data, and delete it. Now the S3 data is being ransomed. The customer opened a case asking if there was any way to get the data back. If we had to go the 'extra mile', the customer demanded we do that. The answer is simply: No. We can't get that data back. Customers demand that they own the data they upload into S3. They don't want AWS to be able to read the data, nor do they want us to store the data internally as a backup. That's what customers demand, and what AWS gives them. Now something regrettable happened (a customer got pwned) and now a customer wants AWS to bear the responsibility for backing up the data. I bet a week ago they would've demanded that they have absolute sovereignty over their data. Shit changes when shit hits the fan. --- Hard caps are doable, but the question is: Do customers want responsibility for this feature? I read a comment wherein someone's startup was killed due to $30k of bandwidth costs because of misconfigurations that led to users abusing their platform. That sucks. It's not in AWS's interest to put their customers out of business. But let's look at the flip side. What if someone in the finance department puts a hard cap on an account. But then a tweet goes viral, and business is pouring in. Everyone, in their rush to keep everything scaling appropriately, forgets there's a hard cap. They hit the business and now there's a hard outage in the middle of the biggest business event they've ever had. Who's the customer going to hold responsible for the outage? As someone who deals with customer misconfigurations on AWS for a living, I assure you the customer won't be calling themselves demanding an explanation. What happens if the person in charge of budgets doesn't lift the cap before the Christmas holidays? Again, hard down. Again, customer won't be mad at themselves. --- It's better to let AWS run as the customer configures it to run than bring everything to an abrupt stop. Billing alerts exist, and you can use Lambda to turn off resources that are above their billing threshold. But doing a hard cap on an account? Something a subsection of customers might want, but something not many customers will want to take responsibility for when something goes sideways.

zadkey 5y ago

I don't think they are going to implement it. It's too juicy financially to do so. Furthermore, it's bad PR to say they aren't going to do it. So they will keep stringing us along as long as they can. Well, that's what the incentives would suggest.

tyrex2017 5y ago

Many comments say it cant be done, but this is disingenious. As if we were bureaucrats with no imagination: Eg just have 2 caps: First cap triggers a red alarm prohibiting provisioning, or storing, of new data. The second one (when you are like 50% above the first cap) is dark red, closing down everything except storage. Sure, decisions have to be made for every single service, but you don’t need a perfect solution if it is optional. Eg, you could start ONLY with EC2 and will have fixed 50% of the problem.

Elect2 5y ago

AWS should do it at least on bandwidth cost.