3 ms·

I understand joking about this and the possible downsides, but there are good reasons why cloud services are set up this way: 1) it's impossible to bill at scal

I understand joking about this and the possible downsides, but there are good reasons why cloud services are set up this way: 1) it's impossible to bill at scale and exactly cut off service usage globally when a target is hit, and 2) most companies don't want a single point of failure like a misconfigured budget to bring down their production services. The answer is probably quota management, where a limit on the number of VMs or size of database or something, caps the worst-case scenario, and where it's arguably easier to monitor an approach to that quota as it's more granular than billing. Personally I think cloud providers could have an explicit "hobby mode" that limits certain things in such a way that the spend can't run away like this, with the trade-off that they're not really production grade in a sense, but then again those accounts are probably worth anything so I understand not building that out. That said, whenever I've seen one of these things happen, it always ends with "FooCloud said that as a one-time gesture of good will they would write off this accidental usage", so while briefly scary, maybe this is the system working fine overall?

jsheard 2y ago
Azure does actually have the ability to force-kill your resources when you hit a certain billing threshold, but only for things like free trials and student accounts. The instant you switch to a regular pay-as-you-go account that functionality disappears for uh... reasons. https://learn.microsoft.com/en-us/azure/cost-management-billing/manage/spending-limit https://learn.microsoft.com/en-us/azure/cost-management-bill...
- danpalmer 2y ago
  That's good, it makes sense it would be for only those sorts of accounts. Imagine building a billing system that could always do that, any time you accept a request, you need to check against a database if the user has hit their billing limit or if you can charge them for it. It obviously can't work. I imagine the way MS make it work is probably to slow down resource consumption for those accounts dramatically, and then just take the hit on overspend, knowing that it will be almost nothing.
  - jsheard 2y ago
    I understand that a precise billing limit is probably impossible at that scale, but if they have the ability to send you an email some time after you go over a limit then they have the ability to automatically pull the plug at the same time rather than waiting for you to notice the email and do it yourself. They just don't want to. I'm sure people would accept a best-effort system where setting a billing limit for $100 means you may be billed $140 because your spending overshot the limit before the system noticed. It still beats the alternative of waking up to a $20,000 bill out of nowhere.
    - danpalmer 2y ago
      So I've stored some data in a bucket, the storage time goes over my budget, what do you do now? Just delete the data? I think most people would think that's a terrible outcome. Let's say the answer is yes, you just delete all the data as I can no longer afford it... delete operations are billable, so do you charge the user for those deletes or not? Let's say the answer is yes, and you bill the deletes. What happens if too many deletes are required and suddenly you're at 2x the bill cap? Now you can't document the bill cap as being able to go over by up to 1.5x. This may be unlikely, but customers use cloud services in weird and wonderful ways. This is just one resource type, there are many different resource types on a typical cloud provider, each with multiple axes of billing, each of which has hard decisions to not just be made, but documented and communicated to customers in such a way that they understand the impact. Oh and also it's the "I just put my credit card in and go" crowd who you have to explain it to, who aren't engaging in sales conversations, not those on business contracts who might actually listen or read the documentation. It's not at all obvious to me that this is preferable to just having someone look at these incidents on a case by case basis and seeing who should be refunded.
      - thot_experiment 2y ago
        This is an insane take, there are many options here. The idea that the reason to structure billing this way is anything OTHER than that it's the most profitable way to do things is ludicrous. It doesn't matter if you can construct some other plausible reason, as long as it's the most profitable way to operate why would I believe the cause to be something other than profit maximization.
thaumasiotes 2y ago
> That said, whenever I've seen one of these things happen, it always ends with "FooCloud said that as a one-time gesture of good will they would write off this accidental usage", so while briefly scary, maybe this is the system working fine overall? I suspect that if they tried to sue you over an unpaid bill, they'd lose over issues of proper notice to you. You can't actually bill someone for a service they didn't want and didn't ask for; that's why the wash-your-car-in-traffic people are considered scammers.
- danpalmer 2y ago
  I'm not a lawyer, maybe it wouldn't hold up in court, but I imagine that good will is the driving force, even if way down the line they may not manage to actually reclaim the money. My experience of working with cloud providers is that they are almost always happy to take a short term loss (extended trial, more test resources, refunds for accidental usage, etc) to get an account that is going to grow and stick with the provider. This does make sense, customer acquisition and churn are expensive, and recurring revenue is great.
esperent 2y ago
> it's impossible to bill at scale and exactly cut off service usage globally when a target is hit This seems unlikely to me. What is the technical reason for this?
- danpalmer 2y ago
  To do this you would need to check in with a central billing service every time you want to charge, and that central billing service must keep a consistent view per customer to ensure it can't spend over the cap. This is not too hard if the billable event is, say, creating a VM. Creating the VM takes a few seconds, it's easy to add a quick API call to check billing. But what about the next hour, when you charge for the VM again? You now have the number of VMs checking in every hour (worst case, at the same time), and you need to handle all those hourly checkins consistently per customer. That's still probably easy enough, but what if it's not a VM hour, but an API gateway request? Now on every single request you have to check in with the billing service. API gateways need to be very low latency, but you've just added a request to that process that possibly needs to go across the world to check in with the billing service running on another continent. What if the billable resource is "database query seconds", and now you need to know how many seconds a query is going to take before you start it? Oh, and add the check in time to every database query. What if the billable resource is streaming video, do you check in on every packet you send out? What if it's CDN downloads, do you have every one of thousands of points of presence around the world all check in, even though the point of the product is to be faster than having a single far away delivery node? There are bad workarounds for each of these, but they all mean either the cloud provider losing money (which, assuming a certain scale of customer, is too expensive), the customer over-spending (which assuming a certain scale, could still be waaay over their budget), or slowing down most services to the point that they functionally don't work anymore.
  - collingreen 2y ago
    All these arguments seem very much like throwing the baby out with the bathwater - I don't think we should pretend it makes sense to say "if we can't have perfect billing cutoff down to each individual api call we shouldn't have a billing cutoff at all". You've listed super achievable ways to prevent a $50/mo spend from ballooning to $70,000. Additionally, it feels hollow to not have billing cutoff at the same rate as authorization would cutoff if they shut off my account.
theodric 2y ago
I strongly disagree that this is in any way defensible, despite that both Google and AWS do it. You should be able to set a limit, and even if they can't cut off exactly at the limit, they should be able to cut off when you hit (say) 2* your limit, or in the absence of a limit, perhaps issue progressively louder warnings until you hit 10* your usual spend, with the ability for you to give affirmative consent to uncap if you're hoping your app will suddenly take off. Nobody needs to lose their house or their life savings to some teracorp just because "it's hard."
- adriand 2y ago
  I totally agree. You should be able to choose to have things shut down rather go past a limit. I don’t have a single client that would choose $70k versus, say, a couple hours downtime while we figure out what is happening with a resource that is going crazy. This is where I question the risk of serverless. Although, now that I think about it, while my one client’s EC2 instances are essentially capped in terms of capacity and spend, we also use S3 etc. I suppose it would be entirely possible to accidentally write a huge amount of data to S3. But again I would rather get warnings from the app that writing to S3 has failed due to limits than get a huge bill!
  - danpalmer 2y ago
    But you also pay for data at rest in S3. Should S3 stop storing that data while you figure things out? Should they bill the customer for the deletion of that data as they normally would? I don't disagree on wanting this feature, but it's just not something that's possible to implement in totality when you dig into the details.
    - adriand 2y ago
      Could it work based on deviance from norms? Like a setting where you say, I’m okay with up to two standard deviations from our typical write volume but that’s it? Or, alternatively, just let me set the size of the volume. Treat it like a hard drive? In terms of your point about the data at rest, part of the issue is that we get a bill once per month, and that’s probably when we would notice it. Of course there is probably a Cloudwatch alarm or something we could set (I assume) but there’s so many damn services…
ndriscoll 2y ago
Hobby mode is using something like Hetzner instead of messing about with "cloud native" nonsense. For the $50/mo they expected, they could get an absolute monster like a Ryzen 5 3600. Cloud native stuff is for when you need to be SOC2 compliant or something and want to minimize access to everything. Clouds charge you worse than enterprise pricing (enterprises negotiate discounts).
trog 2y ago
> it's impossible to bill at scale and exactly cut off service usage globally when a target is hit, How much does the problem change if you remove the word 'exactly' from here, though? Like, I don't mind if I end up paying a couple of extra bucks. Or even tens of bucks! Some people might not mind hundreds or thousands, or even more depending on their scale. But blowing out several orders of magnitude past my usual monthly spend is the problem I'd like to avoid.
rpcope1 2y ago
AWS has a "hobby mode": they call it Lightsail.

jsheard 2y ago

Azure does actually have the ability to force-kill your resources when you hit a certain billing threshold, but only for things like free trials and student accounts. The instant you switch to a regular pay-as-you-go account that functionality disappears for uh... reasons. https://learn.microsoft.com/en-us/azure/cost-management-billing/manage/spending-limit https://learn.microsoft.com/en-us/azure/cost-management-bill...

danpalmer 2y ago
That's good, it makes sense it would be for only those sorts of accounts. Imagine building a billing system that could always do that, any time you accept a request, you need to check against a database if the user has hit their billing limit or if you can charge them for it. It obviously can't work. I imagine the way MS make it work is probably to slow down resource consumption for those accounts dramatically, and then just take the hit on overspend, knowing that it will be almost nothing.
- jsheard 2y ago
  I understand that a precise billing limit is probably impossible at that scale, but if they have the ability to send you an email some time after you go over a limit then they have the ability to automatically pull the plug at the same time rather than waiting for you to notice the email and do it yourself. They just don't want to. I'm sure people would accept a best-effort system where setting a billing limit for $100 means you may be billed $140 because your spending overshot the limit before the system noticed. It still beats the alternative of waking up to a $20,000 bill out of nowhere.
  - danpalmer 2y ago
    So I've stored some data in a bucket, the storage time goes over my budget, what do you do now? Just delete the data? I think most people would think that's a terrible outcome. Let's say the answer is yes, you just delete all the data as I can no longer afford it... delete operations are billable, so do you charge the user for those deletes or not? Let's say the answer is yes, and you bill the deletes. What happens if too many deletes are required and suddenly you're at 2x the bill cap? Now you can't document the bill cap as being able to go over by up to 1.5x. This may be unlikely, but customers use cloud services in weird and wonderful ways. This is just one resource type, there are many different resource types on a typical cloud provider, each with multiple axes of billing, each of which has hard decisions to not just be made, but documented and communicated to customers in such a way that they understand the impact. Oh and also it's the "I just put my credit card in and go" crowd who you have to explain it to, who aren't engaging in sales conversations, not those on business contracts who might actually listen or read the documentation. It's not at all obvious to me that this is preferable to just having someone look at these incidents on a case by case basis and seeing who should be refunded.
    - thot_experiment 2y ago
      This is an insane take, there are many options here. The idea that the reason to structure billing this way is anything OTHER than that it's the most profitable way to do things is ludicrous. It doesn't matter if you can construct some other plausible reason, as long as it's the most profitable way to operate why would I believe the cause to be something other than profit maximization.

thaumasiotes 2y ago

> That said, whenever I've seen one of these things happen, it always ends with "FooCloud said that as a one-time gesture of good will they would write off this accidental usage", so while briefly scary, maybe this is the system working fine overall? I suspect that if they tried to sue you over an unpaid bill, they'd lose over issues of proper notice to you. You can't actually bill someone for a service they didn't want and didn't ask for; that's why the wash-your-car-in-traffic people are considered scammers.

danpalmer 2y ago
I'm not a lawyer, maybe it wouldn't hold up in court, but I imagine that good will is the driving force, even if way down the line they may not manage to actually reclaim the money. My experience of working with cloud providers is that they are almost always happy to take a short term loss (extended trial, more test resources, refunds for accidental usage, etc) to get an account that is going to grow and stick with the provider. This does make sense, customer acquisition and churn are expensive, and recurring revenue is great.

esperent 2y ago

> it's impossible to bill at scale and exactly cut off service usage globally when a target is hit This seems unlikely to me. What is the technical reason for this?

danpalmer 2y ago
To do this you would need to check in with a central billing service every time you want to charge, and that central billing service must keep a consistent view per customer to ensure it can't spend over the cap. This is not too hard if the billable event is, say, creating a VM. Creating the VM takes a few seconds, it's easy to add a quick API call to check billing. But what about the next hour, when you charge for the VM again? You now have the number of VMs checking in every hour (worst case, at the same time), and you need to handle all those hourly checkins consistently per customer. That's still probably easy enough, but what if it's not a VM hour, but an API gateway request? Now on every single request you have to check in with the billing service. API gateways need to be very low latency, but you've just added a request to that process that possibly needs to go across the world to check in with the billing service running on another continent. What if the billable resource is "database query seconds", and now you need to know how many seconds a query is going to take before you start it? Oh, and add the check in time to every database query. What if the billable resource is streaming video, do you check in on every packet you send out? What if it's CDN downloads, do you have every one of thousands of points of presence around the world all check in, even though the point of the product is to be faster than having a single far away delivery node? There are bad workarounds for each of these, but they all mean either the cloud provider losing money (which, assuming a certain scale of customer, is too expensive), the customer over-spending (which assuming a certain scale, could still be waaay over their budget), or slowing down most services to the point that they functionally don't work anymore.
- collingreen 2y ago
  All these arguments seem very much like throwing the baby out with the bathwater - I don't think we should pretend it makes sense to say "if we can't have perfect billing cutoff down to each individual api call we shouldn't have a billing cutoff at all". You've listed super achievable ways to prevent a $50/mo spend from ballooning to $70,000. Additionally, it feels hollow to not have billing cutoff at the same rate as authorization would cutoff if they shut off my account.

theodric 2y ago

I strongly disagree that this is in any way defensible, despite that both Google and AWS do it. You should be able to set a limit, and even if they can't cut off exactly at the limit, they should be able to cut off when you hit (say) 2* your limit, or in the absence of a limit, perhaps issue progressively louder warnings until you hit 10* your usual spend, with the ability for you to give affirmative consent to uncap if you're hoping your app will suddenly take off. Nobody needs to lose their house or their life savings to some teracorp just because "it's hard."

adriand 2y ago
I totally agree. You should be able to choose to have things shut down rather go past a limit. I don’t have a single client that would choose $70k versus, say, a couple hours downtime while we figure out what is happening with a resource that is going crazy. This is where I question the risk of serverless. Although, now that I think about it, while my one client’s EC2 instances are essentially capped in terms of capacity and spend, we also use S3 etc. I suppose it would be entirely possible to accidentally write a huge amount of data to S3. But again I would rather get warnings from the app that writing to S3 has failed due to limits than get a huge bill!
- danpalmer 2y ago
  But you also pay for data at rest in S3. Should S3 stop storing that data while you figure things out? Should they bill the customer for the deletion of that data as they normally would? I don't disagree on wanting this feature, but it's just not something that's possible to implement in totality when you dig into the details.
  - adriand 2y ago
    Could it work based on deviance from norms? Like a setting where you say, I’m okay with up to two standard deviations from our typical write volume but that’s it? Or, alternatively, just let me set the size of the volume. Treat it like a hard drive? In terms of your point about the data at rest, part of the issue is that we get a bill once per month, and that’s probably when we would notice it. Of course there is probably a Cloudwatch alarm or something we could set (I assume) but there’s so many damn services…

ndriscoll 2y ago

Hobby mode is using something like Hetzner instead of messing about with "cloud native" nonsense. For the $50/mo they expected, they could get an absolute monster like a Ryzen 5 3600. Cloud native stuff is for when you need to be SOC2 compliant or something and want to minimize access to everything. Clouds charge you worse than enterprise pricing (enterprises negotiate discounts).

trog 2y ago

> it's impossible to bill at scale and exactly cut off service usage globally when a target is hit, How much does the problem change if you remove the word 'exactly' from here, though? Like, I don't mind if I end up paying a couple of extra bucks. Or even tens of bucks! Some people might not mind hundreds or thousands, or even more depending on their scale. But blowing out several orders of magnitude past my usual monthly spend is the problem I'd like to avoid.

rpcope1 2y ago

AWS has a "hobby mode": they call it Lightsail.