7 ms·

I really wish cloud providers would allow users to set a hard budget that just stops the service if you exceed a threshold. I got a surprise bill from AWS this

by paulgb 7y ago

I really wish cloud providers would allow users to set a hard budget that just stops the service if you exceed a threshold. I got a surprise bill from AWS this month (fortunately orders of magnitude less than this, but still ~15x my usual) and am thinking of moving from Lambda to a VPS just so the possibility of this doesn't keep me up at night.

stickfigure 7y ago
You can? I have daily limits for my Google App Engine apps.
- EugeneOZ 7y ago
  Don't forget they are only warnings, they will not shut down your app and their update rate is not as frequent as you might expect.
  - inlined 7y ago
    If the concern is DB operations adding up you can lower your quota as well which will be strongly enforced and shut down your app.
  - asciimike 7y ago
    App Engine PM: the GAE Daily Spend limits are hard caps that will shut down services when hit. They are different from the broader GCP Billing Alerts which are simply notifications. The issue is that they only apply to certain GAE services (compute, legacy APIs, etc.) and not across the platform.
    - EugeneOZ 7y ago
      Thank you for correction!
- inlined 7y ago
  Firebase also has a fixed price plan that can shut off your app if it blows through quotas. It’s called “Flame”
jsty 7y ago
I think there'd be two big issues - (1) what is reasonable behaviour when you reach your budget, and (2) the consistency of billing. On (1), if you get to say the 14th of the month and run out of budget, what would be the expected behaviour? - shut down all your services? delete all your data so that's not incurring a charge too? It'd probably have to be configurable on a service by service basis, and that's an awful lot of complexity to introduce for a tiny minority of their revenue base. On (2), their billing systems are almost certainly eventually consistent in logging and charging usage, so they'd either have to shut you down early in anticipation of delayed charges, or let any delayed charges bring you over budget. Either is liable to make customers fairly unhappy one way or the other. The TL;DR version - it'd be nigh on impossible to have good UX with such a feature, would be complicated to implement, and it's not likely to move the needle on any metric they really care about considerably.
- kkarakk 7y ago
  > it's not likely to move the needle on any metric they really care about considerably. what about devs too scared to put their CC no in the till coz they've heard horror stories about being overcharged for mistakes? Just recently had to sort out a feature that a junior dev created using openstreetmaps coz they didn't want to be on the hook by putting their card into google maps... It was for an internal portal, usage wasn't even high enough to hit the monthly free tier! Hour 1 experience also matters.
  - p0llard 7y ago
    Why would a junior dev be using _their_ card details? Surely your company has a corporate billing account with Google for this purpose? Perhaps I've misunderstood the situation, but if I were working as a dev at a company and was expected to provide my own card details to pay for APIs, I would walk out the door.
    - kkarakk 7y ago
      They were just told to make the portal as a learning teambuilding exercise, choice of tools was entirely up to them - i even told the person company would pony up for anything they ended up spending. HR was being a pain about getting card details(it's not easy getting that stuff in india) The dude bounced off google maps as soon as he saw the form to enter payment info and went straight to open street maps.
dvfjsdhgfv 7y ago
Only Amazon is adamant at refusing this, Microsoft already made it possible some years ago, and I believe Google too.
- donmcronald 7y ago
  Azure doesn’t have it for pay as you go. https://feedback.azure.com/forums/906772-cost-management/suggestions/3238796-spending-limit-or-maximum-cost-cap-for-azure https://feedback.azure.com/forums/906772-cost-management/sug... A 6+ year wait for such a reasonable feature sucks.
  - GFischer 7y ago
    And they have it for almost all other plans, so the technology is there. It sounds like an intentional product decision and not a technical obstacle.
tcgv 7y ago
Amazon provides Billing Alarms for monitoring charges, which is very useful for anticipating unplanned cost surges: - https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...
- donmcronald 7y ago
  Alarms are like a garage leaving you a voicemail message before doing $25k of repairs on your car. It’s ridiculous that I can’t set a hard cap on my spending.
  - gtsteve 7y ago
    I guess you could have the SNS topic invoke a lambda that shuts down the specific service if you wanted that but I agree a budgets feature would be nice, especially when provisioning test accounts for developers.
    - crankylinuxuser 7y ago
      Fine and dandy, but you can't use Lambda in fedramp. You can't use most aws things in fedramp scope.
  - brianwawok 7y ago
    On the flip side I would never want my real business to shut off. Hobby project yes! Real business no,
    - faster 7y ago
      Assuming your revenue and your AWS bill are strongly correlated, I agree. However, when a free service or a bug can increase costs by orders of magnitude without a corresponding increase in revenue your choices are to eat it (not sustainable, for most of us), throttle, or stop. Throttling is hard, so stopping is often the best choice.
StreamBright 7y ago
I guess you had not setup any budget alarms.
- paulgb 7y ago
  The alarms are why it wasn't higher than 15x.
  - StreamBright 7y ago
    At what percentage you had the alarm then?
7ewis 7y ago
How did you manage to increase your bill 15x in Lambda?
- paulgb 7y ago
  It seems to have been a combination of a traffic surge and a third-party API being temporarily slow. The slow API meant that I ended up paying for more idle CPU time during the request round-trip.
grepthisab 7y ago
Same here, with AWS specifically. I'm used to AWS so generally stay in that ecosystem. Designed a web app and backed part of it by lambda and api gateway. Via a surge in traffic, my monthly bill went from about $6/mo to $150 for a month. Didn't catch it until I got the bill for the month, the timing of which meant my next month's bill was about $80.
OJFord 7y ago
I use VPSs for personal use for exactly this reason - the bill's the same (small amount) every month, and if load spikes I'm happier not being able to handle it than handling it for a surprise Xx bill.
vertis 7y ago
A few years ago I got a ~$3500 Amazon bill when someone broke into my account (through keys I'd inadvertently added to GitHub)[1]. I had no expectation of that money being waived, but Amazon did anyway. Since then I've had billing alerts turned on at about the $150 mark (which is about double what my personal monthly Amazon bill is). [1]: https://news.ycombinator.com/item?id=6911908 https://news.ycombinator.com/item?id=6911908
- hoaxcracker 7y ago
  Shameless plug: I know this plug should not be here but I think I can help some people with cost problem via (the easiest automation platform for AWS)[1]. I have a startup plan where you can run 60 executions in a month free of cost. Just reach out to me via https://totalcloud.io/solutions.html https://totalcloud.io/solutions.html and refer YComb. [1]: https://totalcloud.io https://totalcloud.io
GFischer 7y ago
Microsoft Azure has such a feature (although not for Enterprise users, which is evil), I´m surprised that AWS doesn´t. https://azure.microsoft.com/en-us/support/legal/offer-details/ https://azure.microsoft.com/en-us/support/legal/offer-detail...
solidasparagus 7y ago
You can set budget alerts, which aren’t as good, but they can let you know if the bill’s going off the tracks. I set up alerts at 50% and 75% of what I’m willing to pay and that’s gotten rid of my ‘unexpected bill’ stress.
cavisne 7y ago
You can limit the max concurrency of lambda functions which can help a bit here. Imo the reason is this is for business/enterprise use cases configuring this wrong could be catastrophic, for side project/personal use cases its pretty well known you get 1 chance for a refund from the public clouds, after some manual checking to make sure you aren't trying to abuse the service.

by paulgb 7y ago

stickfigure 7y ago

You can? I have daily limits for my Google App Engine apps.

EugeneOZ 7y ago
Don't forget they are only warnings, they will not shut down your app and their update rate is not as frequent as you might expect.
- inlined 7y ago
  If the concern is DB operations adding up you can lower your quota as well which will be strongly enforced and shut down your app.
- asciimike 7y ago
  App Engine PM: the GAE Daily Spend limits are hard caps that will shut down services when hit. They are different from the broader GCP Billing Alerts which are simply notifications. The issue is that they only apply to certain GAE services (compute, legacy APIs, etc.) and not across the platform.
  - EugeneOZ 7y ago
    Thank you for correction!
inlined 7y ago
Firebase also has a fixed price plan that can shut off your app if it blows through quotas. It’s called “Flame”

jsty 7y ago

I think there'd be two big issues - (1) what is reasonable behaviour when you reach your budget, and (2) the consistency of billing. On (1), if you get to say the 14th of the month and run out of budget, what would be the expected behaviour? - shut down all your services? delete all your data so that's not incurring a charge too? It'd probably have to be configurable on a service by service basis, and that's an awful lot of complexity to introduce for a tiny minority of their revenue base. On (2), their billing systems are almost certainly eventually consistent in logging and charging usage, so they'd either have to shut you down early in anticipation of delayed charges, or let any delayed charges bring you over budget. Either is liable to make customers fairly unhappy one way or the other. The TL;DR version - it'd be nigh on impossible to have good UX with such a feature, would be complicated to implement, and it's not likely to move the needle on any metric they really care about considerably.

kkarakk 7y ago
> it's not likely to move the needle on any metric they really care about considerably. what about devs too scared to put their CC no in the till coz they've heard horror stories about being overcharged for mistakes? Just recently had to sort out a feature that a junior dev created using openstreetmaps coz they didn't want to be on the hook by putting their card into google maps... It was for an internal portal, usage wasn't even high enough to hit the monthly free tier! Hour 1 experience also matters.
- p0llard 7y ago
  Why would a junior dev be using _their_ card details? Surely your company has a corporate billing account with Google for this purpose? Perhaps I've misunderstood the situation, but if I were working as a dev at a company and was expected to provide my own card details to pay for APIs, I would walk out the door.
  - kkarakk 7y ago
    They were just told to make the portal as a learning teambuilding exercise, choice of tools was entirely up to them - i even told the person company would pony up for anything they ended up spending. HR was being a pain about getting card details(it's not easy getting that stuff in india) The dude bounced off google maps as soon as he saw the form to enter payment info and went straight to open street maps.

dvfjsdhgfv 7y ago

Only Amazon is adamant at refusing this, Microsoft already made it possible some years ago, and I believe Google too.

donmcronald 7y ago
Azure doesn’t have it for pay as you go. https://feedback.azure.com/forums/906772-cost-management/suggestions/3238796-spending-limit-or-maximum-cost-cap-for-azure https://feedback.azure.com/forums/906772-cost-management/sug... A 6+ year wait for such a reasonable feature sucks.
- GFischer 7y ago
  And they have it for almost all other plans, so the technology is there. It sounds like an intentional product decision and not a technical obstacle.

tcgv 7y ago

Amazon provides Billing Alarms for monitoring charges, which is very useful for anticipating unplanned cost surges: - https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

donmcronald 7y ago
Alarms are like a garage leaving you a voicemail message before doing $25k of repairs on your car. It’s ridiculous that I can’t set a hard cap on my spending.
- gtsteve 7y ago
  I guess you could have the SNS topic invoke a lambda that shuts down the specific service if you wanted that but I agree a budgets feature would be nice, especially when provisioning test accounts for developers.
  - crankylinuxuser 7y ago
    Fine and dandy, but you can't use Lambda in fedramp. You can't use most aws things in fedramp scope.
- brianwawok 7y ago
  On the flip side I would never want my real business to shut off. Hobby project yes! Real business no,
  - faster 7y ago
    Assuming your revenue and your AWS bill are strongly correlated, I agree. However, when a free service or a bug can increase costs by orders of magnitude without a corresponding increase in revenue your choices are to eat it (not sustainable, for most of us), throttle, or stop. Throttling is hard, so stopping is often the best choice.

StreamBright 7y ago

I guess you had not setup any budget alarms.

paulgb 7y ago
The alarms are why it wasn't higher than 15x.
- StreamBright 7y ago
  At what percentage you had the alarm then?

7ewis 7y ago

How did you manage to increase your bill 15x in Lambda?

paulgb 7y ago
It seems to have been a combination of a traffic surge and a third-party API being temporarily slow. The slow API meant that I ended up paying for more idle CPU time during the request round-trip.

grepthisab 7y ago

Same here, with AWS specifically. I'm used to AWS so generally stay in that ecosystem. Designed a web app and backed part of it by lambda and api gateway. Via a surge in traffic, my monthly bill went from about $6/mo to $150 for a month. Didn't catch it until I got the bill for the month, the timing of which meant my next month's bill was about $80.

OJFord 7y ago

I use VPSs for personal use for exactly this reason - the bill's the same (small amount) every month, and if load spikes I'm happier not being able to handle it than handling it for a surprise Xx bill.

vertis 7y ago

A few years ago I got a ~$3500 Amazon bill when someone broke into my account (through keys I'd inadvertently added to GitHub)[1]. I had no expectation of that money being waived, but Amazon did anyway. Since then I've had billing alerts turned on at about the $150 mark (which is about double what my personal monthly Amazon bill is). [1]: https://news.ycombinator.com/item?id=6911908 https://news.ycombinator.com/item?id=6911908

hoaxcracker 7y ago
Shameless plug: I know this plug should not be here but I think I can help some people with cost problem via (the easiest automation platform for AWS)[1]. I have a startup plan where you can run 60 executions in a month free of cost. Just reach out to me via https://totalcloud.io/solutions.html https://totalcloud.io/solutions.html and refer YComb. [1]: https://totalcloud.io https://totalcloud.io

GFischer 7y ago

Microsoft Azure has such a feature (although not for Enterprise users, which is evil), I´m surprised that AWS doesn´t. https://azure.microsoft.com/en-us/support/legal/offer-details/ https://azure.microsoft.com/en-us/support/legal/offer-detail...

solidasparagus 7y ago

You can set budget alerts, which aren’t as good, but they can let you know if the bill’s going off the tracks. I set up alerts at 50% and 75% of what I’m willing to pay and that’s gotten rid of my ‘unexpected bill’ stress.

cavisne 7y ago

You can limit the max concurrency of lambda functions which can help a bit here. Imo the reason is this is for business/enterprise use cases configuring this wrong could be catastrophic, for side project/personal use cases its pretty well known you get 1 chance for a refund from the public clouds, after some manual checking to make sure you aren't trying to abuse the service.