6 ms·

If it is a "free tier", Amazon should halt the application when it exceeds quota. Moving the account to a paid tier and charging $100k is not the right thing t

If it is a "free tier", Amazon should halt the application when it exceeds quota. Moving the account to a paid tier and charging $100k is not the right thing to do.

cperciva 1y ago
Good news! This is exactly how the free tier works now.
christophilus 1y ago
I agree, but I could also see how someone would complain about that: “Our e-commerce site was taken down by Amazon right on our biggest day of the year. They should have just moved us up to the next tier.”
- ndsipa_pomu 1y ago
  Seems like the most flexible option is to put a spending limit in place by default and make it obvious that it can affect availability of the service if the limit is reached. My credit cards have credit limits, so it makes sense that a variable cost service should easily be able to support a spending limit too.
- pixl97 1y ago
  Then let that be the non default option.
  - derefr 1y ago
    The default option is always going to be the one that makes the majority of Amazon's paying customers happy.
    - mjevans 1y ago
      Maybe offer 'Sales day rush auto-scale' as a setting.
- magicalhippo 1y ago
  That would get caught during the pre-peak stress testing. You do do stress testing ahead of peak season, right?
tekno45 1y ago
stop putting stuff on the internet you don't understand.
mmh0000 1y ago
Yes. They said it was free then they surprise charge you $100k. That’s an insane amount of both money and stress. You’re at Amazon’s mercy if they will or will not refund it. And while this is in process you’re wondering if your entire financial future is ruined.
- JustExAWS 1y ago
  I have never in 8 years of being in the AWS ecosystem and reading forums and Reddits on the internet had anyone report that AWS wouldn’t refund their money. If you go over your budget with AWS, what should AWS do automatically? Delete your objects from S3? Terminate your databases and EC2 instances? Besides, billing data collection doesn’t happen anywhere near realtime, consider it a fire hose of streaming data that is captured asynchronously.
  - Bjartr 1y ago
    Provide the user the tools to make these choices. Give the option to explicitly choose how durable to extreme traffic you want to be. Have the free tier default to "not very durable"
    - kyleee 1y ago
      Bam, you said. They’d do it if they cared, but they don’t and prefer the status quo. 100k surprise bill is the type of thing people kill themselves over. Horrific
    - raw_anon_1111 1y ago
      You mean like having a billing alert send an event that allows you to trigger custom actions to turn things off? That already exists. It has for years.
      - crote 1y ago
        So why isn't it the default yet? Why isn't unlimited scaling something you have to turn on?
derefr 1y ago
You're misunderstanding the offering. (Maybe that's their fault for using intentionally misleading language... but using that language in this way is pretty common nowadays, so this is important to understand.) For a postpaid service with usage-based billing, there are no separate "free" and "paid" plans (= what you're clearly thinking of when you're saying "tiers" here.) The "free tier" of these services, is a set of per-usage-SKU monthly usage credit bonuses, that are set up in such a way that if you are using reasonable "just testing" amounts of resources, your bill for the month will be credited down to $0. And yes, this does mean that even when you're paying for some AWS services, you're still benefitting from the "free tier" for any service whose usage isn't exceeding those free-tier limits. That's why it's a [per-SKU usage] tier, rather than a "plan." If you're familiar with electricity providers telling you that you're about to hit a "step-up rate" for your electricity usage for the month — that's exactly the same type of usage tier system. Except theirs goes [cheap usage] -> [expensive usage], whereas IaaS providers' tiers go [free usage] -> [costed usage]. > Amazon should halt the application when it exceeds quota. There is no easy way to do this in a distributed system (which is why IaaS services don't even try; and why their billing dashboards are always these weird detached things that surface billing only in monthly statements and coarse-grained charts, with no visibility into the raw usage numbers.) There's a lot of inherent complexity of converting "usage" into "billable usage." It involves not just muxing usage credit-spend together, but also classifying spend from each system into a SKU [where the appropriate bucket for the same usage can change over time]; and then a lot of lookups into various control-plane systems to figure out whether any bounded or continuous discounts and credits should be applied to each SKU. And that means that this conversion process can't happen in the services themselves. It needs to be a separate process pushed out to some specific billing system. Usually, this means that the services that generate billable usage are just asynchronously pushing out "usage-credit spend events" into something like a log or message queue; and then a billing system is, asynchronously, sucking these up and crunching through them to emit/checkpoint "SKU billing events" against an invoice object tied to a billing account. Due to all of the extra steps involved in this pipeline, the cumulative usage that an IaaS knows about for a given billing account (i.e. can fire a webhook when one of those billing events hits an MQ topic) might be something like 5 minutes out-of-date of the actual incoming usage-credit-spend. Which means that, by the time any "trigger" to shut down your application because it exceeded a "quota" went through, your application would have already spent 5 minutes more of credits. And again, for a large, heavily-loaded application — the kind these services are designed around — that extra five minutes of usage could correspond to millions of dollars of extra spend. Which is, obviously, unacceptable from a customer perspective. No customer would accept a "quota system" that says you're in a free plan, yet charges you, because you accrued an extra 5 minutes of usage beyond the free plan's limits before the quota could "kick in." But nor would the IaaS itself just be willing to eat that bill for the actual underlying costs of serving that extra 5 minutes of traffic, because that traffic could very well have an underlying cost of "millions of dollars." So instead they just say "no, we won't implement a data-plane billable-usage-quota feature; if you want it, you can either implement it yourself [since your L7 app can observe its usage 'live' much better than our infra can] or, more idiomatically to our infra, you can ensure that any development project is configured with appropriate sandboxing + other protections to never get into a situation where any resource could exceed its the free-tier-credited usage in the first place."

If it is a "free tier", Amazon should halt the application when it exceeds quota. Moving the account to a paid tier and charging $100k is not the right thing to do.

cperciva 1y ago

Good news! This is exactly how the free tier works now.

christophilus 1y ago

I agree, but I could also see how someone would complain about that: “Our e-commerce site was taken down by Amazon right on our biggest day of the year. They should have just moved us up to the next tier.”

ndsipa_pomu 1y ago
Seems like the most flexible option is to put a spending limit in place by default and make it obvious that it can affect availability of the service if the limit is reached. My credit cards have credit limits, so it makes sense that a variable cost service should easily be able to support a spending limit too.
pixl97 1y ago
Then let that be the non default option.
- derefr 1y ago
  The default option is always going to be the one that makes the majority of Amazon's paying customers happy.
  - mjevans 1y ago
    Maybe offer 'Sales day rush auto-scale' as a setting.
magicalhippo 1y ago
That would get caught during the pre-peak stress testing. You do do stress testing ahead of peak season, right?

tekno45 1y ago

stop putting stuff on the internet you don't understand.

mmh0000 1y ago

Yes. They said it was free then they surprise charge you $100k. That’s an insane amount of both money and stress. You’re at Amazon’s mercy if they will or will not refund it. And while this is in process you’re wondering if your entire financial future is ruined.

JustExAWS 1y ago
I have never in 8 years of being in the AWS ecosystem and reading forums and Reddits on the internet had anyone report that AWS wouldn’t refund their money. If you go over your budget with AWS, what should AWS do automatically? Delete your objects from S3? Terminate your databases and EC2 instances? Besides, billing data collection doesn’t happen anywhere near realtime, consider it a fire hose of streaming data that is captured asynchronously.
- Bjartr 1y ago
  Provide the user the tools to make these choices. Give the option to explicitly choose how durable to extreme traffic you want to be. Have the free tier default to "not very durable"
  - kyleee 1y ago
    Bam, you said. They’d do it if they cared, but they don’t and prefer the status quo. 100k surprise bill is the type of thing people kill themselves over. Horrific
  - raw_anon_1111 1y ago
    You mean like having a billing alert send an event that allows you to trigger custom actions to turn things off? That already exists. It has for years.
    - crote 1y ago
      So why isn't it the default yet? Why isn't unlimited scaling something you have to turn on?

derefr 1y ago

You're misunderstanding the offering. (Maybe that's their fault for using intentionally misleading language... but using that language in this way is pretty common nowadays, so this is important to understand.) For a postpaid service with usage-based billing, there are no separate "free" and "paid" plans (= what you're clearly thinking of when you're saying "tiers" here.) The "free tier" of these services, is a set of per-usage-SKU monthly usage credit bonuses, that are set up in such a way that if you are using reasonable "just testing" amounts of resources, your bill for the month will be credited down to $0. And yes, this does mean that even when you're paying for some AWS services, you're still benefitting from the "free tier" for any service whose usage isn't exceeding those free-tier limits. That's why it's a [per-SKU usage] tier, rather than a "plan." If you're familiar with electricity providers telling you that you're about to hit a "step-up rate" for your electricity usage for the month — that's exactly the same type of usage tier system. Except theirs goes [cheap usage] -> [expensive usage], whereas IaaS providers' tiers go [free usage] -> [costed usage]. > Amazon should halt the application when it exceeds quota. There is no easy way to do this in a distributed system (which is why IaaS services don't even try; and why their billing dashboards are always these weird detached things that surface billing only in monthly statements and coarse-grained charts, with no visibility into the raw usage numbers.) There's a lot of inherent complexity of converting "usage" into "billable usage." It involves not just muxing usage credit-spend together, but also classifying spend from each system into a SKU [where the appropriate bucket for the same usage can change over time]; and then a lot of lookups into various control-plane systems to figure out whether any bounded or continuous discounts and credits should be applied to each SKU. And that means that this conversion process can't happen in the services themselves. It needs to be a separate process pushed out to some specific billing system. Usually, this means that the services that generate billable usage are just asynchronously pushing out "usage-credit spend events" into something like a log or message queue; and then a billing system is, asynchronously, sucking these up and crunching through them to emit/checkpoint "SKU billing events" against an invoice object tied to a billing account. Due to all of the extra steps involved in this pipeline, the cumulative usage that an IaaS knows about for a given billing account (i.e. can fire a webhook when one of those billing events hits an MQ topic) might be something like 5 minutes out-of-date of the actual incoming usage-credit-spend. Which means that, by the time any "trigger" to shut down your application because it exceeded a "quota" went through, your application would have already spent 5 minutes more of credits. And again, for a large, heavily-loaded application — the kind these services are designed around — that extra five minutes of usage could correspond to millions of dollars of extra spend. Which is, obviously, unacceptable from a customer perspective. No customer would accept a "quota system" that says you're in a free plan, yet charges you, because you accrued an extra 5 minutes of usage beyond the free plan's limits before the quota could "kick in." But nor would the IaaS itself just be willing to eat that bill for the actual underlying costs of serving that extra 5 minutes of traffic, because that traffic could very well have an underlying cost of "millions of dollars." So instead they just say "no, we won't implement a data-plane billable-usage-quota feature; if you want it, you can either implement it yourself [since your L7 app can observe its usage 'live' much better than our infra can] or, more idiomatically to our infra, you can ensure that any development project is configured with appropriate sandboxing + other protections to never get into a situation where any resource could exceed its the free-tier-credited usage in the first place."