8 ms·

This is an example of why cloud hosting is so scary. Yes, Amazon, and I assume Azure and Google's cloud and others, "usually" refund the money. But I don't wa

by JJMcJ 1y ago

This is an example of why cloud hosting is so scary. Yes, Amazon, and I assume Azure and Google's cloud and others, "usually" refund the money. But I don't want to be forced into bankruptcy because my five visitor a week demo project suddenly becomes the target of a DDOS for no reason at all and the hosting company decides this isn't a "usually" so please send the wire transfer.

JackSlateur 1y ago
They have billing limits https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-managing-costs.html https://docs.aws.amazon.com/cost-management/latest/userguide...
- agwa 1y ago
  Those are not in fact limits: > There can be a delay between when you incur a charge and when you receive a notification from AWS Budgets for the charge. This is due to a delay between when an AWS resource is used and when that resource usage is billed. You might incur additional costs or usage that exceed your budget notification threshold before AWS Budgets can notify you, and your actual costs or usage may continue to increase or decrease after you receive the notification.
- appreciatorBus 1y ago
  These aren’t limits though, they are just budget notifications. What would be helpful, would be if when you set up your account there was a default limit – as in an actual limit, where all projects stop working once you go over it - of some sane amount like $5 or $50 or even $500. I have a handful of toy projects on AWS and Google cloud. On both I have budgets set up at $1 and $10, with notifications at 10% 50% and 90%. It’s great, but it’s not a limit. I can still get screwed if somehow, my projects become targets, and I don’t see the emails immediately or aren’t able to act on them immediately. It blows my mind there’s no way I can just say, “there’s no conceivable outcome where I would want to spend more than $10 or more than $100 or whatever so please just cut me off as soon as I get anywhere close to that.” The only conclusion I can come to is that these services are simply not made for small experimental projects, yet I also don’t know any other way to learn the services except by setting up toy projects, and thus exposing yourself to ruinous liability.
  - sbarre 1y ago
    I feel that the likely answer here is that instrumenting real-time spending limit monitoring and cut-off at GCP/AWS scale is Complicated/Expensive to do, so they choose to not do it. I suppose you could bake the limits into each service at deploy time, but that's still a lot of code to write to provide a good experience to a customer who is trying to not pay you money. Not saying this is a good thing, but this feels about right to me.
    - appreciatorBus 1y ago
      I agree that that’s the likely explanation. It just feels infuriating that the services are sold as easy to get started and risk free with generous free tiers, inviting people and companies to try out small projects, yet each small experiment contains an element of unlimited risk with no mitigation tools.
    - Xelbair 1y ago
      I don't care if it is expensive for them. I'm not running their business, I'm their customer - it is inconvenient for me. And frankly any pay-as-you-go scheme should be regulated to have maximum spending limit setting. Not only in IT.
      - VectorLock 1y ago
        Its not expensive for them, its expensive for their customers. If you went over your spending limit and they deleted all your shit, people would be absolutely apoplectic. Instead they make you file a relatively painless ticket and explain why you accidentally went over what you wanted to spend. This is an engineering trade-off they made to make things less painful for their customers.
        wpm 1y ago
        Why would I apoplectic at Amazon if I set “turn my shit off after it has accrued $10 in charges” to TRUE and they actually followed what I asked them to do?
- Loudergood 1y ago
  "Limits" like this are how I woke up one Sunday morning in my college dorm with a $7k bill from dreamhost.
- master_crab 1y ago
  As others have said these are not limits, just notifications. You can’t actually create a limit unless you self create one using another AWS service (surprise) like lambda to read in the reports and shut things down. And as others have also mentioned, the reports have a delay. In many cases it’s several hours. But worst case, your CURs (Cost usage reports) don’t really reflect reality for up to 24 hours after the fact.
- meepmorp 1y ago
  To paraphrase Rainer Wolfcastle - the budgets do nothing! You get a warning. There's no service cutoffs or hard limits on spending.
- Foobar8568 1y ago
  As far as I know, neither Google, Amazon or Azure have a budget limit, only alerts. This is a reason why I am not only clueless of anything related to cloud infrastructure unless it's stuff I am doing on the job, nor I am willing to build anything on these stacks. And while I guess I have less than 10 products build with these techs, I am appeal by the overall reliability of the services. Oh lastly, for Azure, in different European regions you can't instance resources, you need to go through your account representative who asks authorization from the US. So much for now having to deal with infrastructure pain. It's just a joke.
  - Ciantic 1y ago
    I've used Azure with spending limits. They do work, they shut down things, and the lights go off. [1], Only some external resources you are unlikely to use don't follow spending limits, but when you create such resources, they are clearly marked as external. That's one positive side of Azure. [1]: https://learn.microsoft.com/en-us/azure/cost-management- https://learn.microsoft.com/en-us/azure/cost-management-
    - Foobar8568 1y ago
      These limits are only for subscriptions with a credit amount e.g. $200 trials, Visual Studio subscriptions etc. As soon as you are on a pay as you go, you only have access to budget limit.
- JackSlateur 1y ago
  To all of those who say "this is not limit, only notifications": yes, notifications that can trigger whatever you want, including a shutdown of whatever you have Is this a perfect solution: no Is this still a solution: yes
  - rustc 1y ago
    The notifications are not real time. You can rack up a significant bill before they are triggered.
- ldoughty 1y ago
  I work in this space regularly. There can be a delay of 2-3 days from the event to charge. Seems some services report faster than others. But this means by the time you get a billing alert it has been ongoing for hours if not days.
Bluecobra 1y ago
When I am playing around in the cloud I am super paranoid about charges, so I end up locking the ACLs to only permit traffic to my home IP. It’s too bad that they don’t have a better built in way of making sandbox labs. When I was doing cloud training with A Cloud Guru, it would generate a whole global AWS instance that would only last for 30 minutes.
- psychoslave 1y ago
  Why don't you run locally?
  - bdangubic 1y ago
    run entire aws infra locally while studying for aws certification?
    - psychoslave 1y ago
      Let’s rephrase the question then, why makes an application dependent on AWS?
      - kbart 1y ago
        AWS skills are in quite strong demand, so it totally pays off to know the platform and have some hands-on experience if you work in the related area.
      - nucleardog 1y ago
        In general that would be a good question, but you've asked it in a case where "use AWS" is the _only_ way to accomplish the goal... which is learning AWS.
      - bdangubic 1y ago
        You commented on a post that included When I was doing cloud training with A Cloud Guru which is cloud certification platform. Can’t run “locally” getting prepped for an AWS certification and AWS is absolute shit for beginners in terms of cost protections
tetromino_ 1y ago
If you sign up for electrical service for your house, and your shithead neighbor taps your line to power his array of grow lamps and crypto mining rigs, the power company will happily charge you thousands of dollars, and you will need a police report and traverse many layers of customer service hell to get a refund. If you sign up for water service and a tree root cracks your pipe, the water company will happily charge you thousands of dollars for the leaked water, and will then proceed to mandate that you to fix the broken pipe at your own expense for a couple tens of thousands more; and yes, that may well bankrupt you, water company don't care. So why do you expect different treatment from a computing utility provider?
- Touche 1y ago
  The difference is this actually happens, a lot, unlike your straw man. It happens enough that there's a website dedicated to it.
- ndsipa_pomu 1y ago
  The first instance is difficult to fix as crime can often involve substantial losses to people and often there's no route to getting a refund. The broken water pipe should be covered by buildings insurance, but I can imagine it not being covered by some policies. Luckily a broken water pipe is likely not as expensive as not having e.g. third party liability protection if part of your roof falls off and hits someone.
- psychoslave 1y ago
  I don't know in US, but having limits on how much electricity a house is able to take from the gride is absolutely something in some countries out there.
  - OJFord 1y ago
    Definitely in the US too, I'm not resident either, but your 100A or whatever supply is a hard limit on what it can cost you per time period.
- pixl97 1y ago
figassis 1y ago
They refund those that know how to demand it, and that notice. If you have complex infra and not a lot of observability, you'll just assume the costs are legitimate. Imagine how much they're making off of those oops moments. Probably a bug chunk of their revenue reports.
AtlasBarfed 1y ago
Aws, able to bill everything down to like milliseconds of usage... We can't implement a basic cost limiter policy. I think we all know why.
- tstrimple 1y ago
  It's extra frustrating I think on the Azure side because they absolutely have cost limited accounts for MSDN subscribers but won't extend that functionality to general users. Just let me set a cap on the cost I'm willing to pay per month and let me deal with the consequences of the resource being shut down unexpectedly. You can work around these things if you instrument the right metrics and create the right alerts so you can take action in time. But those are often hard learned lessons and not the happy path to using the cloud. It's entirely possible to build cloud first solutions that scale better and are cheaper than your standard reliable colo solutions. But you've got to understand the tradeoffs and when to limit scaling otherwise things can run away from you. I still reach for "cloud first" tools when building my own projects because I know how to run them extremely cheaply without the risk of expenses blowing up because some random thing I've built lands on HN or the equivalent. Many hobby projects or even small businesses can leverage free tiers of cloud services almost indefinitely. But you've got to architect your solutions differently to leverage the advantages and avoid the weaknesses of the cloud. Actually understand the strengths and limitations of the various cloud "functions as a service" offerings and understand where your needs could be solved from those tools and how to work within those cost constraints. Repeatedly I see people trying to use the cloud as if it's just another colo or datacenter and build things in the same way they did before and only think about things in terms of virtual machines tend to have a more difficult time adopting the cloud and they end up spending far more than the companies who can tear down and spin up entire environments through IaC and leverage incremental pricing to your benefit.
- znpy 1y ago
  > I think we all know why. There's no need to imply that, it's not illegal to criticise AWS. They do not want anybody to be able to set a limit on spend as that would probably hurt the business model.

by JJMcJ 1y ago

JackSlateur 1y ago

They have billing limits https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-managing-costs.html https://docs.aws.amazon.com/cost-management/latest/userguide...

agwa 1y ago
Those are not in fact limits: > There can be a delay between when you incur a charge and when you receive a notification from AWS Budgets for the charge. This is due to a delay between when an AWS resource is used and when that resource usage is billed. You might incur additional costs or usage that exceed your budget notification threshold before AWS Budgets can notify you, and your actual costs or usage may continue to increase or decrease after you receive the notification.
appreciatorBus 1y ago
These aren’t limits though, they are just budget notifications. What would be helpful, would be if when you set up your account there was a default limit – as in an actual limit, where all projects stop working once you go over it - of some sane amount like $5 or $50 or even $500. I have a handful of toy projects on AWS and Google cloud. On both I have budgets set up at $1 and $10, with notifications at 10% 50% and 90%. It’s great, but it’s not a limit. I can still get screwed if somehow, my projects become targets, and I don’t see the emails immediately or aren’t able to act on them immediately. It blows my mind there’s no way I can just say, “there’s no conceivable outcome where I would want to spend more than $10 or more than $100 or whatever so please just cut me off as soon as I get anywhere close to that.” The only conclusion I can come to is that these services are simply not made for small experimental projects, yet I also don’t know any other way to learn the services except by setting up toy projects, and thus exposing yourself to ruinous liability.
- sbarre 1y ago
  I feel that the likely answer here is that instrumenting real-time spending limit monitoring and cut-off at GCP/AWS scale is Complicated/Expensive to do, so they choose to not do it. I suppose you could bake the limits into each service at deploy time, but that's still a lot of code to write to provide a good experience to a customer who is trying to not pay you money. Not saying this is a good thing, but this feels about right to me.
  - appreciatorBus 1y ago
    I agree that that’s the likely explanation. It just feels infuriating that the services are sold as easy to get started and risk free with generous free tiers, inviting people and companies to try out small projects, yet each small experiment contains an element of unlimited risk with no mitigation tools.
  - Xelbair 1y ago
    I don't care if it is expensive for them. I'm not running their business, I'm their customer - it is inconvenient for me. And frankly any pay-as-you-go scheme should be regulated to have maximum spending limit setting. Not only in IT.
    - VectorLock 1y ago
      Its not expensive for them, its expensive for their customers. If you went over your spending limit and they deleted all your shit, people would be absolutely apoplectic. Instead they make you file a relatively painless ticket and explain why you accidentally went over what you wanted to spend. This is an engineering trade-off they made to make things less painful for their customers.
      - wpm 1y ago
        Why would I apoplectic at Amazon if I set “turn my shit off after it has accrued $10 in charges” to TRUE and they actually followed what I asked them to do?
Loudergood 1y ago
"Limits" like this are how I woke up one Sunday morning in my college dorm with a $7k bill from dreamhost.
master_crab 1y ago
As others have said these are not limits, just notifications. You can’t actually create a limit unless you self create one using another AWS service (surprise) like lambda to read in the reports and shut things down. And as others have also mentioned, the reports have a delay. In many cases it’s several hours. But worst case, your CURs (Cost usage reports) don’t really reflect reality for up to 24 hours after the fact.
meepmorp 1y ago
To paraphrase Rainer Wolfcastle - the budgets do nothing! You get a warning. There's no service cutoffs or hard limits on spending.
Foobar8568 1y ago
As far as I know, neither Google, Amazon or Azure have a budget limit, only alerts. This is a reason why I am not only clueless of anything related to cloud infrastructure unless it's stuff I am doing on the job, nor I am willing to build anything on these stacks. And while I guess I have less than 10 products build with these techs, I am appeal by the overall reliability of the services. Oh lastly, for Azure, in different European regions you can't instance resources, you need to go through your account representative who asks authorization from the US. So much for now having to deal with infrastructure pain. It's just a joke.
- Ciantic 1y ago
  I've used Azure with spending limits. They do work, they shut down things, and the lights go off. [1], Only some external resources you are unlikely to use don't follow spending limits, but when you create such resources, they are clearly marked as external. That's one positive side of Azure. [1]: https://learn.microsoft.com/en-us/azure/cost-management- https://learn.microsoft.com/en-us/azure/cost-management-
  - Foobar8568 1y ago
    These limits are only for subscriptions with a credit amount e.g. $200 trials, Visual Studio subscriptions etc. As soon as you are on a pay as you go, you only have access to budget limit.
JackSlateur 1y ago
To all of those who say "this is not limit, only notifications": yes, notifications that can trigger whatever you want, including a shutdown of whatever you have Is this a perfect solution: no Is this still a solution: yes
- rustc 1y ago
  The notifications are not real time. You can rack up a significant bill before they are triggered.
ldoughty 1y ago
I work in this space regularly. There can be a delay of 2-3 days from the event to charge. Seems some services report faster than others. But this means by the time you get a billing alert it has been ongoing for hours if not days.

Bluecobra 1y ago

When I am playing around in the cloud I am super paranoid about charges, so I end up locking the ACLs to only permit traffic to my home IP. It’s too bad that they don’t have a better built in way of making sandbox labs. When I was doing cloud training with A Cloud Guru, it would generate a whole global AWS instance that would only last for 30 minutes.

psychoslave 1y ago
Why don't you run locally?
- bdangubic 1y ago
  run entire aws infra locally while studying for aws certification?
  - psychoslave 1y ago
    Let’s rephrase the question then, why makes an application dependent on AWS?
    - kbart 1y ago
      AWS skills are in quite strong demand, so it totally pays off to know the platform and have some hands-on experience if you work in the related area.
    - nucleardog 1y ago
      In general that would be a good question, but you've asked it in a case where "use AWS" is the _only_ way to accomplish the goal... which is learning AWS.
    - bdangubic 1y ago
      You commented on a post that included When I was doing cloud training with A Cloud Guru which is cloud certification platform. Can’t run “locally” getting prepped for an AWS certification and AWS is absolute shit for beginners in terms of cost protections

tetromino_ 1y ago

If you sign up for electrical service for your house, and your shithead neighbor taps your line to power his array of grow lamps and crypto mining rigs, the power company will happily charge you thousands of dollars, and you will need a police report and traverse many layers of customer service hell to get a refund. If you sign up for water service and a tree root cracks your pipe, the water company will happily charge you thousands of dollars for the leaked water, and will then proceed to mandate that you to fix the broken pipe at your own expense for a couple tens of thousands more; and yes, that may well bankrupt you, water company don't care. So why do you expect different treatment from a computing utility provider?

Touche 1y ago
The difference is this actually happens, a lot, unlike your straw man. It happens enough that there's a website dedicated to it.
ndsipa_pomu 1y ago
The first instance is difficult to fix as crime can often involve substantial losses to people and often there's no route to getting a refund. The broken water pipe should be covered by buildings insurance, but I can imagine it not being covered by some policies. Luckily a broken water pipe is likely not as expensive as not having e.g. third party liability protection if part of your roof falls off and hits someone.
psychoslave 1y ago
I don't know in US, but having limits on how much electricity a house is able to take from the gride is absolutely something in some countries out there.
- OJFord 1y ago
  Definitely in the US too, I'm not resident either, but your 100A or whatever supply is a hard limit on what it can cost you per time period.
pixl97 1y ago

figassis 1y ago

They refund those that know how to demand it, and that notice. If you have complex infra and not a lot of observability, you'll just assume the costs are legitimate. Imagine how much they're making off of those oops moments. Probably a bug chunk of their revenue reports.

AtlasBarfed 1y ago

Aws, able to bill everything down to like milliseconds of usage... We can't implement a basic cost limiter policy. I think we all know why.

tstrimple 1y ago
It's extra frustrating I think on the Azure side because they absolutely have cost limited accounts for MSDN subscribers but won't extend that functionality to general users. Just let me set a cap on the cost I'm willing to pay per month and let me deal with the consequences of the resource being shut down unexpectedly. You can work around these things if you instrument the right metrics and create the right alerts so you can take action in time. But those are often hard learned lessons and not the happy path to using the cloud. It's entirely possible to build cloud first solutions that scale better and are cheaper than your standard reliable colo solutions. But you've got to understand the tradeoffs and when to limit scaling otherwise things can run away from you. I still reach for "cloud first" tools when building my own projects because I know how to run them extremely cheaply without the risk of expenses blowing up because some random thing I've built lands on HN or the equivalent. Many hobby projects or even small businesses can leverage free tiers of cloud services almost indefinitely. But you've got to architect your solutions differently to leverage the advantages and avoid the weaknesses of the cloud. Actually understand the strengths and limitations of the various cloud "functions as a service" offerings and understand where your needs could be solved from those tools and how to work within those cost constraints. Repeatedly I see people trying to use the cloud as if it's just another colo or datacenter and build things in the same way they did before and only think about things in terms of virtual machines tend to have a more difficult time adopting the cloud and they end up spending far more than the companies who can tear down and spin up entire environments through IaC and leverage incremental pricing to your benefit.
znpy 1y ago
> I think we all know why. There's no need to imply that, it's not illegal to criticise AWS. They do not want anybody to be able to set a limit on spend as that would probably hurt the business model.