8 ms·

It's surprisingly complex to do that. Let's take a simple example and say your cloud account is doing 2 things - compute & storage. Compute is an active resour

It's surprisingly complex to do that. Let's take a simple example and say your cloud account is doing 2 things - compute & storage. Compute is an active resource, when you exceed your budget it can be automatically shutdown. Storage is a passive resource, when you exceed your budget it can be automatically....deleted? That's almost always the wrong action. Providing fine-grained cost limits help some, as passive resources usually don't have massive cost spikes while active resources do, so you can better "protect" your passive resources by setting more aggressive cost limits on the active resources. This quickly gets more complicated. Another example is most monitoring services are a combination of active (actual metric monitoring) and passive (metric history) resources. A cost limit on that monitoring service likely won't provide sub-service granularity, mostly depending on whether the service even has different charges for monitoring vs history. Oh, also, even for a passive resource like storage, you also have active resource charges whenever you upload/download your data. Ugh, what a mess. The best thing to do is pay attention to your spending, just like you do with your personal & corporate budget.

Closi 6y ago
It's almost like you could make it configurable so users can choose what happens if they go over, and to what extent.
AngusH 6y ago
But we've had disk quotas before that mostly worked? If anything it seems an easier problem than processor time. I recall disk quotas on shared systems at university back in 1998 and I'm sure they existed before that. Two thresholds IIRC, one at which you get a warning, second at which you can't write any further and the disk write operation fails. I don't think they deleted files, it was just you couldn't write more than [quota] bytes to your disk. Is there something particular about cloud based systems that prevent this from working? ie. is this a specific problem with distributed storage? edit:tone
- robrtsql 6y ago
  S3 costs money to keep your files in, even if you're not touching them, so just preventing further uploads wouldn't do much to prevent your AWS bill from increasing.
  - camjohnson26 6y ago
    It would let you set an upper limit on the price you pay though. Better than accidentally misconfiguring a logging service and writing gigabytes of unneeded data.
- Spooky23 6y ago
  The IaaS services are easy. Other services are more difficult. Something like BigQuery ML could generate massive bills pretty easily.
- user5994461 6y ago
  >>> But we've had disk quotas before that mostly worked? AWS has quotas on everything, including quotas on EBS storage per region. You will realize that after you spin up some instances with disks and it's failing because you've hit 10 TB of EBS storage. Have to raise a ticket to raise the limit.
cesarb 6y ago
> Storage is a passive resource, when you exceed your budget it can be automatically....deleted? That's almost always the wrong action. A better option would be to automatically reduce the budget by the amount it would cost to keep the storage forever. If doing that would reduce the budget to zero, do not allow increasing the amount of storage. That is: assume the storage will not be deleted, and budget according to that.
- asciimike 6y ago
  How does this actually work? It clearly can't be forever, since any non-zero dollar amount * infinity months is infinity dollars, which is going to reduce the budget below zero since any non-infinite number minus infinity is less than zero... thus locking it immediately. Even if we say "you get N months of storage before we delete it" and subtract N * current storage cost/month, what happens after you're locked out of all actions because you added an extra GB? Storage APIs cost money to use, so you would get locked out of those too (note that if you're not, people would set arbitrarily low limits and get storage access for free) and couldn't retrieve anything. The only remaining actions are delete (which is free) or raise the quota and do the whole rodeo over again. Abuse is impossible to ignore at public cloud scale, so "free storage forever" (or even, storage at a one time fixed price) as the fallback isn't a viable option. Lastly, from an optics perspective, which blog post would you rather see on the front page of HN: "I did something dumb and spent too much money on Cloud" or "Google is holding our data hostage" (or "Google deleted all my data")? Source: I launched Firebase Storage, which has a GCS bucket that has a hard limit.
  - deleted 6y ago
    [deleted]
  - betwixthewires 6y ago
    Off the top of my head I'd say that if you're budgeting for storage, the max you can afford for the time period that you'd need to recover data in the event of a budget overrun taking into considerations the delay time for notifications would be a way to calculate that. And that sounds like something that is reasonable to put on the customer to calculate.
modeless 6y ago
It's not really that complex. All compute should shut down. All API calls should fail. Storage should be (optionally) preserved at normal cost. Your examples are simple given this framework. Uploading/downloading data to storage is an API call. Monitoring is compute. Metric history storage is storage.
- jasonpeacock 6y ago
  But storage costs continue to add up even when you're not accessing them - there's a cost to storage existing which continues to accrue with time. When there's no budget left, what do you do with those accruing costs for existing storage?
  - asciimike 6y ago
    Moreover, once API calls are locked, what next? You can't delete files, and even if you can delete them, you aren't able to retrieve them before deletion... If a platform allows you to do those actions, then it's rife for abuse, and at public cloud scale that ends up being a far, far bigger problem than the occasional blog post that ends up as a refund (because the other blog post is "I got free storage forever with this one weird trick"). It's really not a simple problem because the next action depends on the choice the developer wants to make: do they increase the budget or decrease usage, and no cloud provider wants to make this choice because no matter what the choice is it will be viewed as wrong. The best they can do is provide developers the best insight and tooling to make this choice themselves.
    - AngusH 6y ago
      You need separate costed quotas for each type of activity with a combined total at the bottom. You could also have a setting in the admin panel as to what the system should do: [ ] I want to keep going beyond my quotas (but email me) [ ] Please shutdown my site
      - asciimike 6y ago
        If the answer is "you have a dollar limit set of GCS GETs, GCS PUTs, etc." I guess I could see this working, but hot damn that'll be a horrific interface. The other issue is that many large customers pay different prices, so billing and quota aren't really tied to each other, and it wouldn't be easy to reconcile this. As for the button... having been on the product side of building this button, there is no right answer: people will say they never got the email (or it went to the wrong inbox, or their dog ate their phone...) or that they never checked the box to "shut down the site" ("I didn't think it would do X that made my app not work").
onion2k 6y ago
You've explained why it's hard for Google to not give me resources I can't pay for, but that's not what I care about, or what I'm asking for. What I'm asking for is a feature where I set a hard limit of $100 and that's the most I get billed - if my account accidently uses $5000 of resources before Google reconciles the usage with my budget then Google automatically waives the additional $4900 and then limits my account in some way until the problem is rectified. Practically every time these blog posts come up they end with the provider refunding the costs. I just want that refund to be a feature.
- jasonpeacock 6y ago
  So...you're saying that Google should give away $4900 of usage? How will Google automatically differentiate between an "honest mistake" and someone taking advantage of this feature?
  - onion2k 6y ago
    So...you're saying that Google should give away $4900 of usage? Yes. But they should also develop mechanisms to warn users that they've made a mistake before it happens, and improve the speed they can detect mistakes to lower the cost, and invent some way to detect someone intentionally abusing the feature. But mostly they should make the fact they do give away $4900 when a mistake happens explicit. That isn't actually a change. They just need to make it clear that's what happens.

Closi 6y ago

It's almost like you could make it configurable so users can choose what happens if they go over, and to what extent.

AngusH 6y ago

But we've had disk quotas before that mostly worked? If anything it seems an easier problem than processor time. I recall disk quotas on shared systems at university back in 1998 and I'm sure they existed before that. Two thresholds IIRC, one at which you get a warning, second at which you can't write any further and the disk write operation fails. I don't think they deleted files, it was just you couldn't write more than [quota] bytes to your disk. Is there something particular about cloud based systems that prevent this from working? ie. is this a specific problem with distributed storage? edit:tone

robrtsql 6y ago
S3 costs money to keep your files in, even if you're not touching them, so just preventing further uploads wouldn't do much to prevent your AWS bill from increasing.
- camjohnson26 6y ago
  It would let you set an upper limit on the price you pay though. Better than accidentally misconfiguring a logging service and writing gigabytes of unneeded data.
Spooky23 6y ago
The IaaS services are easy. Other services are more difficult. Something like BigQuery ML could generate massive bills pretty easily.
user5994461 6y ago
>>> But we've had disk quotas before that mostly worked? AWS has quotas on everything, including quotas on EBS storage per region. You will realize that after you spin up some instances with disks and it's failing because you've hit 10 TB of EBS storage. Have to raise a ticket to raise the limit.

cesarb 6y ago

> Storage is a passive resource, when you exceed your budget it can be automatically....deleted? That's almost always the wrong action. A better option would be to automatically reduce the budget by the amount it would cost to keep the storage forever. If doing that would reduce the budget to zero, do not allow increasing the amount of storage. That is: assume the storage will not be deleted, and budget according to that.

asciimike 6y ago
How does this actually work? It clearly can't be forever, since any non-zero dollar amount * infinity months is infinity dollars, which is going to reduce the budget below zero since any non-infinite number minus infinity is less than zero... thus locking it immediately. Even if we say "you get N months of storage before we delete it" and subtract N * current storage cost/month, what happens after you're locked out of all actions because you added an extra GB? Storage APIs cost money to use, so you would get locked out of those too (note that if you're not, people would set arbitrarily low limits and get storage access for free) and couldn't retrieve anything. The only remaining actions are delete (which is free) or raise the quota and do the whole rodeo over again. Abuse is impossible to ignore at public cloud scale, so "free storage forever" (or even, storage at a one time fixed price) as the fallback isn't a viable option. Lastly, from an optics perspective, which blog post would you rather see on the front page of HN: "I did something dumb and spent too much money on Cloud" or "Google is holding our data hostage" (or "Google deleted all my data")? Source: I launched Firebase Storage, which has a GCS bucket that has a hard limit.
- deleted 6y ago
  [deleted]
- betwixthewires 6y ago
  Off the top of my head I'd say that if you're budgeting for storage, the max you can afford for the time period that you'd need to recover data in the event of a budget overrun taking into considerations the delay time for notifications would be a way to calculate that. And that sounds like something that is reasonable to put on the customer to calculate.

modeless 6y ago

It's not really that complex. All compute should shut down. All API calls should fail. Storage should be (optionally) preserved at normal cost. Your examples are simple given this framework. Uploading/downloading data to storage is an API call. Monitoring is compute. Metric history storage is storage.

jasonpeacock 6y ago
But storage costs continue to add up even when you're not accessing them - there's a cost to storage existing which continues to accrue with time. When there's no budget left, what do you do with those accruing costs for existing storage?
- asciimike 6y ago
  Moreover, once API calls are locked, what next? You can't delete files, and even if you can delete them, you aren't able to retrieve them before deletion... If a platform allows you to do those actions, then it's rife for abuse, and at public cloud scale that ends up being a far, far bigger problem than the occasional blog post that ends up as a refund (because the other blog post is "I got free storage forever with this one weird trick"). It's really not a simple problem because the next action depends on the choice the developer wants to make: do they increase the budget or decrease usage, and no cloud provider wants to make this choice because no matter what the choice is it will be viewed as wrong. The best they can do is provide developers the best insight and tooling to make this choice themselves.
  - AngusH 6y ago
    You need separate costed quotas for each type of activity with a combined total at the bottom. You could also have a setting in the admin panel as to what the system should do: [ ] I want to keep going beyond my quotas (but email me) [ ] Please shutdown my site
    - asciimike 6y ago
      If the answer is "you have a dollar limit set of GCS GETs, GCS PUTs, etc." I guess I could see this working, but hot damn that'll be a horrific interface. The other issue is that many large customers pay different prices, so billing and quota aren't really tied to each other, and it wouldn't be easy to reconcile this. As for the button... having been on the product side of building this button, there is no right answer: people will say they never got the email (or it went to the wrong inbox, or their dog ate their phone...) or that they never checked the box to "shut down the site" ("I didn't think it would do X that made my app not work").

onion2k 6y ago

You've explained why it's hard for Google to not give me resources I can't pay for, but that's not what I care about, or what I'm asking for. What I'm asking for is a feature where I set a hard limit of $100 and that's the most I get billed - if my account accidently uses $5000 of resources before Google reconciles the usage with my budget then Google automatically waives the additional $4900 and then limits my account in some way until the problem is rectified. Practically every time these blog posts come up they end with the provider refunding the costs. I just want that refund to be a feature.

jasonpeacock 6y ago
So...you're saying that Google should give away $4900 of usage? How will Google automatically differentiate between an "honest mistake" and someone taking advantage of this feature?
- onion2k 6y ago
  So...you're saying that Google should give away $4900 of usage? Yes. But they should also develop mechanisms to warn users that they've made a mistake before it happens, and improve the speed they can detect mistakes to lower the cost, and invent some way to detect someone intentionally abusing the feature. But mostly they should make the fact they do give away $4900 when a mistake happens explicit. That isn't actually a change. They just need to make it clear that's what happens.