6 ms·

Is there seriously no way in AWS/Azure/GCP to specify "Here's my budget, shut everything down if I exceed $X"? I don't use those platforms much but was always s

Is there seriously no way in AWS/Azure/GCP to specify "Here's my budget, shut everything down if I exceed $X"? I don't use those platforms much but was always surprised I couldn't find anything like that right off the bat. I'll build cloud stuff if it makes sense at work, but if I'm footing the bill I'll stick to something that can provide an actual upper limit.

josephorjoe 5y ago
You can set price alerts to email you on the day when you cross a monthly spending limit, but it is not the simplest thing to figure out. I run a small server and have a few odds and ends and get emails at $5, $15, $50, $75, and $100. Haven't broken $75 limit yet...
- PeterisP 5y ago
  The problem is that a billing alert at $10 won't prevent you from accidentally starting up something that will bring you a $1000 bill before you can react to that email - there needs to be a process that cuts off service at some hard limit instead of just sending an email and continuing to spend money.
  - josephorjoe 5y ago
    completely agree. even though it creates a new class of problem, there should be a "cut me off immediately at $XXXX per month" option.
newobj 5y ago
Of course there's a way. But it's opt-in, not opt-out.
- deleted 5y ago
  [deleted]
deleted 5y ago
[deleted]
NullPrefix 5y ago
IIRC the excuse is that billing is a separate department and they count all the usage dollars way after you done using it, not in realtime. You would still be able to go over your limit and then who should foot the bill? Realtime counting would be too difficult to figure out, our brightest minds are busy figuring out engagement metrics.
- NathanKP 5y ago
  This is my personal opinion, though I do work at AWS: It's not that real time counting is difficult, it is that the amount of compute resources and electricity that would be needed to power real time billing at AWS scale would be astronomical. There is a reason why banks and financial institutions generally do batch processing in the off peak hours when electricity is cheaper and there is less demand for the compute resources. Now imagine AWS billing, which is arguably far more difficult in scale and complexity.
  - czgs2woh4ue3 5y ago
    I also work at AWS (nowhere near billing), so the usual disclaimers apply, but: I actually have no idea if billing is real-time or not? I think it's mostly batch, but the records aren't necessarily, though they may be aggregated a bit. The general point in this thread certainly holds: our systems provide service first, bill second, and that by throwing a record over the wall to some other internal system. It's not unthinkable they could tally up your costs as you go, but the expense has fundamentally already happened, and that's the disconnect. It would be hard to react ahead of time. Small, cheap things like "a single DynamoDB query" or "a byte of bandwidth" are often also high-volume, and you don't want to introduce billing checks to critical paths for reliability / latency / cost reasons. Expensive big-bang on/off things, probably simpler, though I can think of a few sticking points. It would be hard to react after the fact, too. Where does a bill come from? My own team is deeply internal, several layers removed from anything you're recognize on an invoice, but we're the one generating and measuring costs. Precise attribution would be a problem in and of itself- cutting off service means traversing those layers in reverse, then figuring out what "cut off" means in our specific context. That's new systems and new code all around, repeat for all of AWS- there's a huge organizational problem here on top of the technical one. I could see some individual teams doing something like this for just their services, but AWS-wide would be a big undertaking. I wish we had it- I'd sleep a little better at night, myself- but from my limited perspective, it sure looks like we're fundamentally not designed for this.
diveanon 5y ago
It's not real time billing, so there is no real way to disable service until costs have been determined. I agree that it seems like a solvable problem, but it doesn't make them money so why should they care?
snorkel 5y ago
A spending kill switch can be setup on AWS using AWS Budgets alerts and Lambda but it’s a DIY project, not a built-in feature.
- cesarb 5y ago
  The whole point of a "spending kill switch" is as a backstop when you make a mistake; but if you do it as a "DIY project", what prevents you from making a mistake on it? It has to be a built-in feature.
  - ghaff 5y ago
    So what services do you kill? Everything? Including databases and S3?
    - yjftsjthsd-h 5y ago
      Ideally tunable, but sure. At least provide that option.
- sixothree 5y ago
  If it can be done DIY, it can be done first party.
  - dragonwriter 5y ago
    If its done first party, someone (realistically, af AWS’ scale, a substantial number of someones) will mess up using it and nuke their account. Customer service can correct “we screwed up and have a giant bill” easier than “we screwed up and lost all our data”. So its not going to happen first party. (It’s also not really possible DIY as a hard certain cutoff at or before a limit, only at an indefinite interval in time and money beyond a limit. So you still have potentially unbounded expense.)
maxgashkov 5y ago
That’s extremely hard to design, at least with the current state of what AWS bills and does not bill. Example: let’s assume you’ve set the cut-off budget too strict, spun off another shard for your stateful service (DB for example), it received and stored some data during the short window before some other service brought whole account over budget (i.e. paid egress crossed the threshold). To bring VM and EBS charges to 0 (to implement your idea of ‘shut down everything’) AWS will have to delete production data. While it may be OK when you’re experimenting, it’s really hard to tell in automated way. So, to fully comply w/o risking losing customer data, AWS will have to stop charging for not running instances and inactive EBS volumes which most definitely bring on many kinds of abuse. — There may be some other way to do this, maybe mark some services expendable or not, so they are stopped in the event of overspend.
- markus_zhang 5y ago
  They can simply add a nuclear option. Ordinary business probably won't enable that option but individuals can and should set it up.
  - nucleardog 5y ago
    It’s a lot easier for them to refund the individuals who make mistakes than deal with the fallout and bad press from the small businesses they’ve destroyed when they incorrectly enable that option.
- ghaff 5y ago
  I think there's a good argument that they could do better. But there's also probably an argument that harder circuit breakers would result in posts like "AWS destroyed my business right when I was featured in the New York Times"--including things like deleting production data. I'm sort of sympathetic to the idea that AWS should have an experimental "rip it all down" option and kill all stateless services option but that adds another layer of complexity and introduces more edge cases.
  - fapjacks 5y ago
nineplay 5y ago
Azure will alert you when you exceed a budget, but it won't disable anything. Azure MongoDB billing was insanity. I was up to $800 to host a couple of GB that wasn't doing anything. I'm still not sure what happened, even their support kept saying "it's priced by request units" and I kept saying "How does a handful of queries a day translate to $40 in request units?" A year later, I think that I had a lot of collections and they seem to charge per-collection, but I'm still not even sure. Thank goodness we moved off of it after only a month or two.
- jd_mongodb 5y ago
  I work for MongoDB. The writer is referring to CosmosDB, the Microsoft clone of MongoDB. You can runa real MongoDB cluster on Azure with MongoDB Atlas. The pricing model for Atlas is per Node size + disk + memory + data transfer. It's generally easier to predict your costs using this model. Users tend to over-provision with this model so we recommend using auto-scaling. This will allow your cluster to scale up or down based on load (price will adjust as well).
  - nineplay 5y ago
    We switched to MongoDB Atlas which was awesome but I didn't want to sound like a stealth corporate shill. Atlas rocks!
    - jd_mongodb 5y ago
      Love to hear that.
jrockway 5y ago
I think the big problem is that usage collection is a few days out of date, at least for GCP. Autoscaling can react in seconds to increased load, but it takes about 3 days before that shows up on your cost reports. You can burn through a lot of cloud resources in 3 days. GCP at least has some provision to get very detailed information about usage (but not cost) that updates in less than an hour. That, to me, is the tool for building something like "shut down our account if usage is too high". It is annoying that you have to code this yourself, but ultimately, it kind of makes sense to me. Cloud providers exist to rent you some hardware (often with "value-add" software); it's the developer and operator's responsibility to account for every machine they request, every byte they send to the network, every query they make to the database, etc. and to have a good reason for that. To some extent, if you don't know where you're sending bytes, or what queries you're making, how do you know if your product is working? How do you know that you're not hacked? Reliability and cost go hand in hand here -- if you're monitoring what you need to assure reliability, costs probably aren't confusingly accumulating out of control.
- nlitened 5y ago
  Are you being sarcastic? > I think the big problem is that usage collection is a few days out of date, at least for GCP. Autoscaling can react in seconds to increased load, but it takes about 3 days before that shows up on your cost reports. That does not sound like a good reason, but more like a crappy implementation of usage collection. I don’t see why a bunch of Google engineers can’t implement real-time billing properly, and see no reason to defend their inability to do their job.
ta988 5y ago
You can get alerts. And there are some limits for young accounts. But they really need a "beginner" mode. Or a budget mode, put $x on the account can't spend more. But I guess they are making a lot of money with "mistakes" so there may not be kuch incentives
manigandham 5y ago
On top of the other reasons for complexity and delay, this just creates another potential mistake where people delete their entire accounts or stop production services. It's far easier to negotiate dollar amounts than lost data or service uptime.
dragonwriter 5y ago
> Is there seriously no way in AWS/Azure/GCP to specify "Here's my budget, shut everything down if I exceed $X"? All of them have billing APIs which should, in principal, allow you to build “Shutdown everything at some indefinite time interval (and additional spend) after I exceed $X”, though you’ll need to do nontrivial custom coding to tie together the various pieces to do it, and actually stopping all spend is likely going to mean nuking storage, not just halting “active” services. None of them provide anyway to more than probabilistically do “Shut everything down before I exceed, and in a way which prevents exceeding, $X.”
- Havoc 5y ago
  >though you’ll need to do nontrivial custom coding to tie together the various pieces to do it, Let me guess...hosted on the cloud we're trying to shut down?
  - dragonwriter 5y ago
    > Let me guess...hosted on the cloud we're trying to shut down? It needs to consume APIs from that cloud, but there is no reason it would need to be hosted there.
HomeDeLaPot 5y ago
A lot of people have thoughfully responded with reasons why this doesn't or can't exist: real-time billing is far too expensive to implement, better to get a huge bill than to lose data or shut down critical systems, etc. I guess it makes sense—ideally you are monitoring your stuff, whether you're using your own tools or built-in ones—and you know ahead of time when your usage is creeping up. Also, I suppose only the customer can really know which systems can be shut down/deallocated to save money and which ones would kill the company if shut down. It sounds like if you're a small startup strapped for resources, you can avoid these bills either by self-hosting or by being careful about how you build your cloud infrastructure. I.e. maybe you could host your app on your own box OR on an equivalent VM in Azure that's just going to fail if it runs out of disk/CPU/outgoing bandwidth instead of autoscaling to outrageous levels.

josephorjoe 5y ago

You can set price alerts to email you on the day when you cross a monthly spending limit, but it is not the simplest thing to figure out. I run a small server and have a few odds and ends and get emails at $5, $15, $50, $75, and $100. Haven't broken $75 limit yet...

PeterisP 5y ago
The problem is that a billing alert at $10 won't prevent you from accidentally starting up something that will bring you a $1000 bill before you can react to that email - there needs to be a process that cuts off service at some hard limit instead of just sending an email and continuing to spend money.
- josephorjoe 5y ago
  completely agree. even though it creates a new class of problem, there should be a "cut me off immediately at $XXXX per month" option.

newobj 5y ago

Of course there's a way. But it's opt-in, not opt-out.

deleted 5y ago
[deleted]

deleted 5y ago

[deleted]

NullPrefix 5y ago

IIRC the excuse is that billing is a separate department and they count all the usage dollars way after you done using it, not in realtime. You would still be able to go over your limit and then who should foot the bill? Realtime counting would be too difficult to figure out, our brightest minds are busy figuring out engagement metrics.

NathanKP 5y ago
This is my personal opinion, though I do work at AWS: It's not that real time counting is difficult, it is that the amount of compute resources and electricity that would be needed to power real time billing at AWS scale would be astronomical. There is a reason why banks and financial institutions generally do batch processing in the off peak hours when electricity is cheaper and there is less demand for the compute resources. Now imagine AWS billing, which is arguably far more difficult in scale and complexity.
- czgs2woh4ue3 5y ago
  I also work at AWS (nowhere near billing), so the usual disclaimers apply, but: I actually have no idea if billing is real-time or not? I think it's mostly batch, but the records aren't necessarily, though they may be aggregated a bit. The general point in this thread certainly holds: our systems provide service first, bill second, and that by throwing a record over the wall to some other internal system. It's not unthinkable they could tally up your costs as you go, but the expense has fundamentally already happened, and that's the disconnect. It would be hard to react ahead of time. Small, cheap things like "a single DynamoDB query" or "a byte of bandwidth" are often also high-volume, and you don't want to introduce billing checks to critical paths for reliability / latency / cost reasons. Expensive big-bang on/off things, probably simpler, though I can think of a few sticking points. It would be hard to react after the fact, too. Where does a bill come from? My own team is deeply internal, several layers removed from anything you're recognize on an invoice, but we're the one generating and measuring costs. Precise attribution would be a problem in and of itself- cutting off service means traversing those layers in reverse, then figuring out what "cut off" means in our specific context. That's new systems and new code all around, repeat for all of AWS- there's a huge organizational problem here on top of the technical one. I could see some individual teams doing something like this for just their services, but AWS-wide would be a big undertaking. I wish we had it- I'd sleep a little better at night, myself- but from my limited perspective, it sure looks like we're fundamentally not designed for this.

diveanon 5y ago

It's not real time billing, so there is no real way to disable service until costs have been determined. I agree that it seems like a solvable problem, but it doesn't make them money so why should they care?

snorkel 5y ago

A spending kill switch can be setup on AWS using AWS Budgets alerts and Lambda but it’s a DIY project, not a built-in feature.

cesarb 5y ago
The whole point of a "spending kill switch" is as a backstop when you make a mistake; but if you do it as a "DIY project", what prevents you from making a mistake on it? It has to be a built-in feature.
- ghaff 5y ago
  So what services do you kill? Everything? Including databases and S3?
  - yjftsjthsd-h 5y ago
    Ideally tunable, but sure. At least provide that option.
sixothree 5y ago
If it can be done DIY, it can be done first party.
- dragonwriter 5y ago
  If its done first party, someone (realistically, af AWS’ scale, a substantial number of someones) will mess up using it and nuke their account. Customer service can correct “we screwed up and have a giant bill” easier than “we screwed up and lost all our data”. So its not going to happen first party. (It’s also not really possible DIY as a hard certain cutoff at or before a limit, only at an indefinite interval in time and money beyond a limit. So you still have potentially unbounded expense.)

maxgashkov 5y ago

That’s extremely hard to design, at least with the current state of what AWS bills and does not bill. Example: let’s assume you’ve set the cut-off budget too strict, spun off another shard for your stateful service (DB for example), it received and stored some data during the short window before some other service brought whole account over budget (i.e. paid egress crossed the threshold). To bring VM and EBS charges to 0 (to implement your idea of ‘shut down everything’) AWS will have to delete production data. While it may be OK when you’re experimenting, it’s really hard to tell in automated way. So, to fully comply w/o risking losing customer data, AWS will have to stop charging for not running instances and inactive EBS volumes which most definitely bring on many kinds of abuse. — There may be some other way to do this, maybe mark some services expendable or not, so they are stopped in the event of overspend.

markus_zhang 5y ago
They can simply add a nuclear option. Ordinary business probably won't enable that option but individuals can and should set it up.
- nucleardog 5y ago
  It’s a lot easier for them to refund the individuals who make mistakes than deal with the fallout and bad press from the small businesses they’ve destroyed when they incorrectly enable that option.
ghaff 5y ago
I think there's a good argument that they could do better. But there's also probably an argument that harder circuit breakers would result in posts like "AWS destroyed my business right when I was featured in the New York Times"--including things like deleting production data. I'm sort of sympathetic to the idea that AWS should have an experimental "rip it all down" option and kill all stateless services option but that adds another layer of complexity and introduces more edge cases.
- fapjacks 5y ago

nineplay 5y ago

Azure will alert you when you exceed a budget, but it won't disable anything. Azure MongoDB billing was insanity. I was up to $800 to host a couple of GB that wasn't doing anything. I'm still not sure what happened, even their support kept saying "it's priced by request units" and I kept saying "How does a handful of queries a day translate to $40 in request units?" A year later, I think that I had a lot of collections and they seem to charge per-collection, but I'm still not even sure. Thank goodness we moved off of it after only a month or two.

jd_mongodb 5y ago
I work for MongoDB. The writer is referring to CosmosDB, the Microsoft clone of MongoDB. You can runa real MongoDB cluster on Azure with MongoDB Atlas. The pricing model for Atlas is per Node size + disk + memory + data transfer. It's generally easier to predict your costs using this model. Users tend to over-provision with this model so we recommend using auto-scaling. This will allow your cluster to scale up or down based on load (price will adjust as well).
- nineplay 5y ago
  We switched to MongoDB Atlas which was awesome but I didn't want to sound like a stealth corporate shill. Atlas rocks!
  - jd_mongodb 5y ago
    Love to hear that.

jrockway 5y ago

I think the big problem is that usage collection is a few days out of date, at least for GCP. Autoscaling can react in seconds to increased load, but it takes about 3 days before that shows up on your cost reports. You can burn through a lot of cloud resources in 3 days. GCP at least has some provision to get very detailed information about usage (but not cost) that updates in less than an hour. That, to me, is the tool for building something like "shut down our account if usage is too high". It is annoying that you have to code this yourself, but ultimately, it kind of makes sense to me. Cloud providers exist to rent you some hardware (often with "value-add" software); it's the developer and operator's responsibility to account for every machine they request, every byte they send to the network, every query they make to the database, etc. and to have a good reason for that. To some extent, if you don't know where you're sending bytes, or what queries you're making, how do you know if your product is working? How do you know that you're not hacked? Reliability and cost go hand in hand here -- if you're monitoring what you need to assure reliability, costs probably aren't confusingly accumulating out of control.

nlitened 5y ago
Are you being sarcastic? > I think the big problem is that usage collection is a few days out of date, at least for GCP. Autoscaling can react in seconds to increased load, but it takes about 3 days before that shows up on your cost reports. That does not sound like a good reason, but more like a crappy implementation of usage collection. I don’t see why a bunch of Google engineers can’t implement real-time billing properly, and see no reason to defend their inability to do their job.

ta988 5y ago

You can get alerts. And there are some limits for young accounts. But they really need a "beginner" mode. Or a budget mode, put $x on the account can't spend more. But I guess they are making a lot of money with "mistakes" so there may not be kuch incentives

manigandham 5y ago

On top of the other reasons for complexity and delay, this just creates another potential mistake where people delete their entire accounts or stop production services. It's far easier to negotiate dollar amounts than lost data or service uptime.

dragonwriter 5y ago

> Is there seriously no way in AWS/Azure/GCP to specify "Here's my budget, shut everything down if I exceed $X"? All of them have billing APIs which should, in principal, allow you to build “Shutdown everything at some indefinite time interval (and additional spend) after I exceed $X”, though you’ll need to do nontrivial custom coding to tie together the various pieces to do it, and actually stopping all spend is likely going to mean nuking storage, not just halting “active” services. None of them provide anyway to more than probabilistically do “Shut everything down before I exceed, and in a way which prevents exceeding, $X.”

Havoc 5y ago
>though you’ll need to do nontrivial custom coding to tie together the various pieces to do it, Let me guess...hosted on the cloud we're trying to shut down?
- dragonwriter 5y ago
  > Let me guess...hosted on the cloud we're trying to shut down? It needs to consume APIs from that cloud, but there is no reason it would need to be hosted there.

HomeDeLaPot 5y ago

A lot of people have thoughfully responded with reasons why this doesn't or can't exist: real-time billing is far too expensive to implement, better to get a huge bill than to lose data or shut down critical systems, etc. I guess it makes sense—ideally you are monitoring your stuff, whether you're using your own tools or built-in ones—and you know ahead of time when your usage is creeping up. Also, I suppose only the customer can really know which systems can be shut down/deallocated to save money and which ones would kill the company if shut down. It sounds like if you're a small startup strapped for resources, you can avoid these bills either by self-hosting or by being careful about how you build your cloud infrastructure. I.e. maybe you could host your app on your own box OR on an equivalent VM in Azure that's just going to fail if it runs out of disk/CPU/outgoing bandwidth instead of autoscaling to outrageous levels.