4 ms·

There is a very obvious fix for surprise billing. Enforce a billing cap and terminate service if it's met. Even better if you send alerts when the cap is approa

There is a very obvious fix for surprise billing. Enforce a billing cap and terminate service if it's met. Even better if you send alerts when the cap is approaching. If I pay $39/month, a default cap should be $39 per-month. Otherwise, let me set a cap I am comfortable with. Surprise billing is never good for customers, only the business.

klabb3 2y ago
Yes, but I imagined that you’d have a ceiling for rolling average / token bucket over a sensible time window, with two limits: - a soft alert limit, which you set to the threshold of “hmm something is wrong but we’ll bear the cost until we figure it out” - a hard limit which fails until more tokens trickle back in, without shutting down service I really don’t want to rely on forgiveness, it’s just encouraging reckless behavior and submitting to the incomprehensibility of cloud pricing. Everyone wants these limits, why not design products with that in mind from the get go? It feels like such an afterthought
tptacek 2y ago
I promise, you are not the first person to have thought of this, and, believe it or not, there are reasons other than malice and avarice that cloud providers don't terminate service based on billing caps. Terminating service is a big deal. We agree completely about surprise billing.
- sigseg1v 2y ago
  For many company's use cases (make EC2s a few times a year, otherwise leave them running) I would imagine they would really feel better if there was an option that said "if charges incurred is over (configurable threshold you set to 10k higher than your normal monthly spend), then robocall the customer, and if they do not reply after 3 automated calls then block all additional AWS API calls that would incur more than 10% of my monthly spend if left running for 24h." This would still allow all production services to run, but would stop someone from spinning up 200 crypto miners. I'm sure AWS is capable of implementing this, and I don't want to say it's "easy" but I would be shocked if they lacked the technical expertise to do this.
  - maccard 2y ago
    We’ve now gone from hard billing caps to “soft billing with alerting”. If you look at lots of these threads you’ll see that many people don’t want to provide phone numbers, lots of people ignore emails, even repeated ones, directly to them, from billing. This isn’t a technical problem, it’s a service problem. I can see the hn posts already “my site went viral and <HOST> shut me down”
- karmajunkie 2y ago
  The only thing in cloud that's worse than a surprise bill is a surprise outage.
  - lkbm 2y ago
    Depends. For my hobby project that hasn't been monetized, being charged $100k is way worse than it going down. If it were critical infrastructure, or monetized in a way that brought in revenue to cover the charge, then maybe I don't want it to shut down despite skyrocketing costs, but that's hardly the only situation you could be in.
    - tptacek 2y ago
      Nobody is going to charge you $100k.
      - immibis 2y ago
        (ignores the multiple hobbyists who accidentally got charged $100k)
        tptacek 2y ago
        My belief, and you can correct me on this if you have evidence to the contrary, is that nobody actually pays those bills.
        withinboredom 2y ago
        Someone paid those bills to somebody. AWS has costs incurred from it (support, opportunity, etc) and they paid them.
        bee_rider 2y ago
        From that point of view, the cap seems more like a common-sense self defense feature that the cloud provider would want to implement. But we have cloud providers in this thread saying they don’t want to implement caps, so, I dunno.
- aidos 2y ago
  Having flashbacks to the time where we had paid for a server and were paying for rack space for a customer and they were refusing to pay their bill. Our lawyers told us in no uncertain terms that turning off the server would be a terrible idea. “Obstruction of service” is the term that comes to mind.
  - bitbasher 2y ago
    Cloud squatting?
    - psd1 2y ago
      Node eviction
  - alpinisme 2y ago
    While the parent point about cloud providers having arrived at their policies thoughtfully, this particular issue is likely not part of the equation. There are plenty of services that run on a quota system (chat gpt, sentry, etc). There is a difference between shutting off a service the customer reasonably expected to be always on and shutting off a service when it reaches a threshold set by the customer as part of their purchase. The former is more like repossessing a physical good willy-nilly if the customer misses a payment or you find a check bounces…you can’t do that.
  - fauigerzigerk 2y ago
    Surely it depends on the type of contract. Prepaid SIM cards stop working the second you run out of credit.
- hinkley 2y ago
  When you build an app for resiliency you end up with classes of service where the app fails in stages. But to extend that to the billing case, you’d have to have a partnership with your customers, not just a dashboard where they push buttons and an API where you add or delete machines. Maybe the website goes read only except for admin traffic when the budget is exceeded, for instance. Not as a bespoke process each company has to reinvent, but as functionality provided by the vendor.
- easton 2y ago
  I understand it’s super hard, but Azure has pay as you go and credits that do exactly this. Seems like if you were designing a billing system it’d be a good idea to include this feature.
- tmnvix 2y ago
  My preferred option would be to have an optional billing cap that I can enable knowing full well that if it is exceeded the service would be terminated (obviously with notifications as that cap is approached). I could then apply this to simple hobby projects and such, while not having the risk of termination apply to more serious applications (though a 'soft cap' would be nice here so that I could still receive notifications as it approaches).
  - maccard 2y ago
    aws has billing alerts that trigger lambdas. > aws ec2 describe-instances --query 'Reservations[].Instances[].InstanceId' --output text | xargs -n1 aws ec2 stop-instances --instance-ids Will stop all ec2 instances. The real fix is scoping credentials on aws - if you don’t use an account or role with limited permissions then even if they had this toggle the first step in an attack would be to disable this option.
  - jiveturkey 2y ago
    As a certain kind of user, you probably do think that. But I also think I should be able to have a root level admin account without MFA. The consensus is that no, that should not be up to the customer. It's different here, sure, but the providers optimize for not letting customers shoot themselves in the foot, and remediation via bill forgiveness is a fine solution -- from the provider POV.
    - genewitch 2y ago
      I don't have MFA on my root level account, is it because my account is 16 years old or so at this point? Like my personal AWS account is tied directly to my "order more dish soap" amazon account, because that's how it worked back then, i guess.
    - franga2000 2y ago
      You should be able to have a root level admin account with no 2FA! I would print mine and keep it in a tamper-evident envelope in a safe at my lawyer's office with instructions for when and who can get it. A company isn't liable if their customer gets themselves hacked because they decided to not use any of the many MFA options available to them and neither is a company liable if the customer set a billing limit rule that they executed correctly. Companies can simply not be trusted to tell the difference between a foot-gun and a..whatever a good kind of gun would be...
- School-Cotton 2y ago
  Okay, what are the reasons?
  - ec109685 2y ago
    Big spike of real traffic and your site / app / database / system goes down.
    - School-Cotton 2y ago
      Only if you chose to configure an unreasonably low cap.
- tgsovlerkhgsel 2y ago
  Terminating service is a big deal for commercial customers' production environments. Reversibly (i.e. shut down compute, don't delete anything, allow the customer to review, fix and reinstate quickly) terminating service is a minor annoyance for hobby/experimental setups, and in those, it's much more preferable than having to open a support ticket to deal with a massive bill. Having quotas that the customer can increase themselves (but has to manually choose to increase) on storage prevents storage related surprise bills, and the rest you can shut down (optionally, make the user choose up front what they would prefer). What am I missing? Too many commercial customers picking "experimental" initially and forgetting to change it?
  - solidasparagus 2y ago
    No one really cares enough about hobby developers as a customer segment to rebuild the billing infrastructure to make this possible. Scalable billing at huge scale is solved at the cost of latency and being "eventually correct" (unless it has changed). To add a price cap feature, eventually correct isn't enough and then you ask yourself who would actually use it and you have to scroll really far down your list of biggest customers until you get to someone who wants it.
    - tgsovlerkhgsel 2y ago
      The problem with not caring about hobby developers is that it means developers won't be as familiar with your cloud environment when time comes to pick one for the next "real" project. I would also expect a price cap feature to be useful for experimental/no-approval-required projects at work. In fact, if I ran a cloud project for work as a small team-internal project, a cost explosion would become an even bigger bureaucratic nightmare than if it happened at home.
      - deleted 2y ago
        [deleted]
- wokwokwok 2y ago
  Let's not sugar coat it. The problem is 100% technical. Detecting unexpected charges, scaling and restriction in real time is hard. It's easier to just charge people money than come up with good ways to avoid charging them, and deal with edge cases as a manual process. Sure. I get it. What company has an internal team that's like "ooo... lets find ways to cap the amount of money people pay us". No one. That's why. > there are reasons other than malice and avarice Right. It's just avarice. There's no other reason.
  - tptacek 2y ago
    > The problem is 100% technical. Detecting unexpected charges, scaling and restriction in real time is hard. > It's just avarice. There's no other reason.
    - wokwokwok 2y ago
      Yes. Not doing things that are hard because they’re being greedy, and don’t want to cap the amount people can be charged. The problem is technically solvable, it’s good for customers. All you need to do is prioritise actually doing it. In what stupid world do we pretend this is somehow good for consumers? Racking up thousands of dollars in bills? It’s just the status quo. They’re not not fixing it because it’s hard; that’s just the convenient excuse; they’re not fixing it because they don’t want to.
- figassis 2y ago
  Here's the thing. We have apis for everything and their grandmother. You create an instance and there are apis for adding tags, labels, nicknames...but not for spending caps? I understand I don't know all of the complexities involved, but if you can bill by the second or by the hour, you can certainly alert by the same metrics. We have been measuring CPU, MEM with extreme granularity, how about considering price as a resource and measuring the same way, so that a service with a price cap can self manage and self terminate according to some priority field? This might not be the actual solution, but we have been at this for a very long time, seems like there is not even a hint of an attempt at solving it by the giants. This is about incentives, sorry.
  - mnahkies 2y ago
    I was pleasantly surprised when I was messing around with the Google maps API and found I was able to adjust a quota to put an upper cap on daily spend. It made me feel much more comfortable hacking around and not needing to worry that I'd accidentally create a render loop or something that could rack up a bill whilst I wasn't looking
tonnydourado 2y ago
When you're paying $39/month for something that generates $0/month, that is a very sensible policy. When you're paying $50,000/month for something that generates $200,000/month in value, or if an outage can generate $100,000/month in costs, or if the people that can fix an outage cost $100,000/year, then it's not.
- bitbasher 2y ago
  Then that company's threshold for "sensible monthly cost" would be a lot higher. 500k? 1m? Give them the option to set something.
  - foota 2y ago
    Why would you prefer this when the alternative is no downtime and the provider forgiving the bill?
    - yyhhsj0521 2y ago
      > provider forgiving the bill That eventually will be factor into the price like credit card fraud insurance. Better have it be more transparent.
    - naniwaduni 2y ago
      Because you don't want to be relying on the service provider's whim in "forgiving" the bill?
mcbain 2y ago
It's easy for (say) AWS to terminate your EC2 instances. Do they also delete your DB backups? Delete your S3 buckets? All of these incur costs. How hard a cap do you want?
lo3p0ez 2y ago
Yes, and it shouldn't be too hard to implement at the business side. Deno just announced it: https://deno.com/blog/deploy-spend-limits https://deno.com/blog/deploy-spend-limits
- tptacek 2y ago
  (on their Pro plan) I'm not talking it down. Maybe people are right about this. We'll see.
solatic 2y ago
Because data storage costs money, hard billing caps require deleting both your data and its backups to stay under the hard cap. There are very few use cases where that's actually acceptable, not even development environments where people will get upset for losing their work that they haven't pushed to somewhere else yet.

klabb3 2y ago

Yes, but I imagined that you’d have a ceiling for rolling average / token bucket over a sensible time window, with two limits: - a soft alert limit, which you set to the threshold of “hmm something is wrong but we’ll bear the cost until we figure it out” - a hard limit which fails until more tokens trickle back in, without shutting down service I really don’t want to rely on forgiveness, it’s just encouraging reckless behavior and submitting to the incomprehensibility of cloud pricing. Everyone wants these limits, why not design products with that in mind from the get go? It feels like such an afterthought

tptacek 2y ago

I promise, you are not the first person to have thought of this, and, believe it or not, there are reasons other than malice and avarice that cloud providers don't terminate service based on billing caps. Terminating service is a big deal. We agree completely about surprise billing.

sigseg1v 2y ago
For many company's use cases (make EC2s a few times a year, otherwise leave them running) I would imagine they would really feel better if there was an option that said "if charges incurred is over (configurable threshold you set to 10k higher than your normal monthly spend), then robocall the customer, and if they do not reply after 3 automated calls then block all additional AWS API calls that would incur more than 10% of my monthly spend if left running for 24h." This would still allow all production services to run, but would stop someone from spinning up 200 crypto miners. I'm sure AWS is capable of implementing this, and I don't want to say it's "easy" but I would be shocked if they lacked the technical expertise to do this.
- maccard 2y ago
  We’ve now gone from hard billing caps to “soft billing with alerting”. If you look at lots of these threads you’ll see that many people don’t want to provide phone numbers, lots of people ignore emails, even repeated ones, directly to them, from billing. This isn’t a technical problem, it’s a service problem. I can see the hn posts already “my site went viral and <HOST> shut me down”
karmajunkie 2y ago
The only thing in cloud that's worse than a surprise bill is a surprise outage.
- lkbm 2y ago
  Depends. For my hobby project that hasn't been monetized, being charged $100k is way worse than it going down. If it were critical infrastructure, or monetized in a way that brought in revenue to cover the charge, then maybe I don't want it to shut down despite skyrocketing costs, but that's hardly the only situation you could be in.
  - tptacek 2y ago
    Nobody is going to charge you $100k.
    - immibis 2y ago
      (ignores the multiple hobbyists who accidentally got charged $100k)
      - tptacek 2y ago
        My belief, and you can correct me on this if you have evidence to the contrary, is that nobody actually pays those bills.
        withinboredom 2y ago
        Someone paid those bills to somebody. AWS has costs incurred from it (support, opportunity, etc) and they paid them.
        bee_rider 2y ago
        From that point of view, the cap seems more like a common-sense self defense feature that the cloud provider would want to implement. But we have cloud providers in this thread saying they don’t want to implement caps, so, I dunno.
aidos 2y ago
Having flashbacks to the time where we had paid for a server and were paying for rack space for a customer and they were refusing to pay their bill. Our lawyers told us in no uncertain terms that turning off the server would be a terrible idea. “Obstruction of service” is the term that comes to mind.
- bitbasher 2y ago
  Cloud squatting?
  - psd1 2y ago
    Node eviction
- alpinisme 2y ago
  While the parent point about cloud providers having arrived at their policies thoughtfully, this particular issue is likely not part of the equation. There are plenty of services that run on a quota system (chat gpt, sentry, etc). There is a difference between shutting off a service the customer reasonably expected to be always on and shutting off a service when it reaches a threshold set by the customer as part of their purchase. The former is more like repossessing a physical good willy-nilly if the customer misses a payment or you find a check bounces…you can’t do that.
- fauigerzigerk 2y ago
  Surely it depends on the type of contract. Prepaid SIM cards stop working the second you run out of credit.
hinkley 2y ago
When you build an app for resiliency you end up with classes of service where the app fails in stages. But to extend that to the billing case, you’d have to have a partnership with your customers, not just a dashboard where they push buttons and an API where you add or delete machines. Maybe the website goes read only except for admin traffic when the budget is exceeded, for instance. Not as a bespoke process each company has to reinvent, but as functionality provided by the vendor.
easton 2y ago
I understand it’s super hard, but Azure has pay as you go and credits that do exactly this. Seems like if you were designing a billing system it’d be a good idea to include this feature.
tmnvix 2y ago
My preferred option would be to have an optional billing cap that I can enable knowing full well that if it is exceeded the service would be terminated (obviously with notifications as that cap is approached). I could then apply this to simple hobby projects and such, while not having the risk of termination apply to more serious applications (though a 'soft cap' would be nice here so that I could still receive notifications as it approaches).
- maccard 2y ago
  aws has billing alerts that trigger lambdas. > aws ec2 describe-instances --query 'Reservations[].Instances[].InstanceId' --output text | xargs -n1 aws ec2 stop-instances --instance-ids Will stop all ec2 instances. The real fix is scoping credentials on aws - if you don’t use an account or role with limited permissions then even if they had this toggle the first step in an attack would be to disable this option.
- jiveturkey 2y ago
  As a certain kind of user, you probably do think that. But I also think I should be able to have a root level admin account without MFA. The consensus is that no, that should not be up to the customer. It's different here, sure, but the providers optimize for not letting customers shoot themselves in the foot, and remediation via bill forgiveness is a fine solution -- from the provider POV.
  - genewitch 2y ago
    I don't have MFA on my root level account, is it because my account is 16 years old or so at this point? Like my personal AWS account is tied directly to my "order more dish soap" amazon account, because that's how it worked back then, i guess.
  - franga2000 2y ago
    You should be able to have a root level admin account with no 2FA! I would print mine and keep it in a tamper-evident envelope in a safe at my lawyer's office with instructions for when and who can get it. A company isn't liable if their customer gets themselves hacked because they decided to not use any of the many MFA options available to them and neither is a company liable if the customer set a billing limit rule that they executed correctly. Companies can simply not be trusted to tell the difference between a foot-gun and a..whatever a good kind of gun would be...
School-Cotton 2y ago
Okay, what are the reasons?
- ec109685 2y ago
  Big spike of real traffic and your site / app / database / system goes down.
  - School-Cotton 2y ago
    Only if you chose to configure an unreasonably low cap.
tgsovlerkhgsel 2y ago
Terminating service is a big deal for commercial customers' production environments. Reversibly (i.e. shut down compute, don't delete anything, allow the customer to review, fix and reinstate quickly) terminating service is a minor annoyance for hobby/experimental setups, and in those, it's much more preferable than having to open a support ticket to deal with a massive bill. Having quotas that the customer can increase themselves (but has to manually choose to increase) on storage prevents storage related surprise bills, and the rest you can shut down (optionally, make the user choose up front what they would prefer). What am I missing? Too many commercial customers picking "experimental" initially and forgetting to change it?
- solidasparagus 2y ago
  No one really cares enough about hobby developers as a customer segment to rebuild the billing infrastructure to make this possible. Scalable billing at huge scale is solved at the cost of latency and being "eventually correct" (unless it has changed). To add a price cap feature, eventually correct isn't enough and then you ask yourself who would actually use it and you have to scroll really far down your list of biggest customers until you get to someone who wants it.
  - tgsovlerkhgsel 2y ago
    The problem with not caring about hobby developers is that it means developers won't be as familiar with your cloud environment when time comes to pick one for the next "real" project. I would also expect a price cap feature to be useful for experimental/no-approval-required projects at work. In fact, if I ran a cloud project for work as a small team-internal project, a cost explosion would become an even bigger bureaucratic nightmare than if it happened at home.
    - deleted 2y ago
      [deleted]
wokwokwok 2y ago
Let's not sugar coat it. The problem is 100% technical. Detecting unexpected charges, scaling and restriction in real time is hard. It's easier to just charge people money than come up with good ways to avoid charging them, and deal with edge cases as a manual process. Sure. I get it. What company has an internal team that's like "ooo... lets find ways to cap the amount of money people pay us". No one. That's why. > there are reasons other than malice and avarice Right. It's just avarice. There's no other reason.
- tptacek 2y ago
  > The problem is 100% technical. Detecting unexpected charges, scaling and restriction in real time is hard. > It's just avarice. There's no other reason.
  - wokwokwok 2y ago
    Yes. Not doing things that are hard because they’re being greedy, and don’t want to cap the amount people can be charged. The problem is technically solvable, it’s good for customers. All you need to do is prioritise actually doing it. In what stupid world do we pretend this is somehow good for consumers? Racking up thousands of dollars in bills? It’s just the status quo. They’re not not fixing it because it’s hard; that’s just the convenient excuse; they’re not fixing it because they don’t want to.
figassis 2y ago
Here's the thing. We have apis for everything and their grandmother. You create an instance and there are apis for adding tags, labels, nicknames...but not for spending caps? I understand I don't know all of the complexities involved, but if you can bill by the second or by the hour, you can certainly alert by the same metrics. We have been measuring CPU, MEM with extreme granularity, how about considering price as a resource and measuring the same way, so that a service with a price cap can self manage and self terminate according to some priority field? This might not be the actual solution, but we have been at this for a very long time, seems like there is not even a hint of an attempt at solving it by the giants. This is about incentives, sorry.
- mnahkies 2y ago
  I was pleasantly surprised when I was messing around with the Google maps API and found I was able to adjust a quota to put an upper cap on daily spend. It made me feel much more comfortable hacking around and not needing to worry that I'd accidentally create a render loop or something that could rack up a bill whilst I wasn't looking

tonnydourado 2y ago

When you're paying $39/month for something that generates $0/month, that is a very sensible policy. When you're paying $50,000/month for something that generates $200,000/month in value, or if an outage can generate $100,000/month in costs, or if the people that can fix an outage cost $100,000/year, then it's not.

bitbasher 2y ago
Then that company's threshold for "sensible monthly cost" would be a lot higher. 500k? 1m? Give them the option to set something.
- foota 2y ago
  Why would you prefer this when the alternative is no downtime and the provider forgiving the bill?
  - yyhhsj0521 2y ago
    > provider forgiving the bill That eventually will be factor into the price like credit card fraud insurance. Better have it be more transparent.
  - naniwaduni 2y ago
    Because you don't want to be relying on the service provider's whim in "forgiving" the bill?

mcbain 2y ago

It's easy for (say) AWS to terminate your EC2 instances. Do they also delete your DB backups? Delete your S3 buckets? All of these incur costs. How hard a cap do you want?

lo3p0ez 2y ago

Yes, and it shouldn't be too hard to implement at the business side. Deno just announced it: https://deno.com/blog/deploy-spend-limits https://deno.com/blog/deploy-spend-limits

tptacek 2y ago
(on their Pro plan) I'm not talking it down. Maybe people are right about this. We'll see.

solatic 2y ago

Because data storage costs money, hard billing caps require deleting both your data and its backups to stay under the hard cap. There are very few use cases where that's actually acceptable, not even development environments where people will get upset for losing their work that they haven't pushed to somewhere else yet.