5 ms·

Hard spend limits are an anti-feature for enterprise customers, who are the core customer of AWS. Almost no level of accidental spend is worth creating downtime

• throw2343223434 2y ago
  But the presence of a feature does not mean it has to be used?
  • spacebanana7 2y ago
    The accounting teams of many orgs would want this feature to be enabled, but the tech teams wouldn't. Asking AWS not to add this feature means the tech teams win the debate before it starts.
    • kristiandupont 2y ago
      That seems like a very hypothetical problem you are solving there..
      • sigseg1v 2y ago
        I feel pain just reading this because I know where that guy is coming from; at many orgs as soon as some stupid thing is available it gets mandated by people who have no idea how it works or what impacts it will have, all the time. If it was just as simple as not enabling the stupid option surely we could just not ena... Excuse me for a minute, I have to go reset my password that expires every 10 days and can not match any previous password and enter my enterprise mandated sms 2fa because authenticator scary -- woops my SharePoint session expired after 120 seconds of inactivity let me just redo that -- oh what's that my corporate vpn disconnected because I don't have the mandatory patches applied, let me just restart -- Woah would you look at that my network interface doesn't work anymore after that update -- yes yes I'm sorry I know my activity light on MS Teams turned yellow for 5 minutes I'm working on it, just gotta do these other 12 steps so I can reset my password to -- oh look it's time to fill out the monthly company values survey, what's that, it's due by end of day?
    • sokoloff 2y ago
      If a company chooses to be run by accounting rather than by tech, that's fine; they should get the good and the bad outcomes that come from that choice.
• rwmj 2y ago
  Cool, have an "I know what I'm doing, opt me out!" button.
• herculity275 2y ago
  > Even having the option of a hard spend limit would be hazardous, because accounting teams might push the use of such tools, and thereby risk data loss incidents when problems happen. "Hazardous" feels like the wrong word here - if your customer decides to enact a spend limit it should not be up to you to decide whether that's good for them or not.
• tw04 2y ago
  > Hard spend limits are an anti-feature for enterprise customers, who are the core customer of AWS. Almost no level of accidental spend is worth creating downtime or data loss in a critical application. Great, so they don’t have to use the feature? That excuse was a great excuse when AWS was an MVP for someone. 20+ years later… there is no excuse.
• whywhywhywhy 2y ago
  So have a “Starter” account with spend limits then, don’t understand how an individual is supposed to learn this stack and actually sleep at night without waking up panicking something has been left running.
  • vultour 2y ago
    By checking what resources you are spinning up and double checking everything has been removed after you're done. I have used AWS for small projects many times and have never been hit with a surprise bill. The platform is built for actual customers, not your hello world app.
    • ekidd 2y ago
      Being aware of everything you're running on AWS is trivial when it's all on the EC2 dashboard, and you create everything by hand or using Terraform. Many of these new AWS-provided stacks, however, seem to create stuff all over your account. The moral of the story? Don't ever use AWS tools like the one the OP describes, ones which create a bunch of different resources for you automatically.
      • dijksterhuis 2y ago
        > The moral of the story? Don't ever use AWS tools like the one the OP describes, ones which create a bunch of different resources for you automatically. i use them to create a small test stack to look at it for a day or two. then go through, delete all of the resources, put what i need into terraform etc. has worked well for me in the past. but yeah, i would never blindly use aws tools to magically put something into production.
• another-dave 2y ago
  > Almost no level of accidental spend is worth creating downtime or data loss in a critical application. That feels like a bit of a red herring — if that was their ethos, then you'd _have_ to choose burstable/autoscaling config on every service. If I can configure a service to fall over rather than scale at a hard limit, that points to them understanding different their use cases (prod vs dev) and customer types (start-up vs enterprise). Additionally, anytime I've worked for an enterprise customer, they've had a master service agreement set-up with AWS professional services rather than entering credit card info, so they could use that as a simple way to offer choices.
  • Terretta 2y ago
    A service doesn't just "fall over" at a limit. There has to be other machinery to limit it. Given that all your usage and traffic other than that at the limit request should not be gated or limited, why would you want someone else injecting additional complexity and bottleneck risk inline? Determine your own graceful envelope, implement accordingly.
    • another-dave 2y ago
      The OP was saying that having any spending limit would be antithetical to enterprise usage because it could bring critical services to a halt. My point was, if that's a reason to have unbounded spending, why allow me to spin up a service that can get CPU or RAM bound? > Determine your own graceful envelope, implement accordingly. Which most people do have, but we then also want an "ungraceful" backstop — trains have both a brake and a dead man's switch.
• oliwarner 2y ago
  Is every problem you see so insurmountable? In the olden days if we spotted a customer ringing up a colossal bill, we would tell them. These huge Amazon bills are fast but still multiple days. They can trivially use rolling-projection windows to know when an account is having a massive spike. They could use this foresight to call the customer, ensure they're informed, give them the choice about how to continue. This isn't atomic rocket surgery. "Oh but profit" isn't an argument. They are thousands of dollars up before a problem occurs. The only money they lose is money gained through customer accident. Much of it forgiven to customers who cannot afford it. It's not happily spent. They can do better business.
  • cdchn 2y ago
    Cost Anomaly alerts do exactly this. You can't even setup Cloudwatch alerts on your actual spend, the only metric available to you is "EstimatedCharges".
    • oliwarner 2y ago
      Which is great if you understand them and set them up. The post here is an experience following an Amazon tutorial and without expecting it, ending up $$$ in debt. Again, Amazon isn't stepping up to protect users from themselves. They could do a lot more.
      • cdchn 2y ago
        They refunded him the money, without a lot of hassle. He toddled into a rough edge, bumped his head, Amazon made it better and apparently gave him a lollipop as well and sent him on his way. I think Amazon should get some credit here.
  • marcinzm 2y ago
    AWS provides forecasted spend and other alerts. People just don't use them and then complain.
• concerndc1tizen 2y ago
  This guy is a straight shooter with upper management written all over him. (Office Space)
• jjallen 2y ago
  So make it an option. That’s not hard.
• dkdbejwi383 2y ago
  Make "unlimited spend" an opt-in. That way users who have explicitly chosen this and agreed to the terms can't then complain to support to try and get the bill waived.
• dogleash 2y ago
  > Even having the option of a hard spend limit would be hazardous, because accounting teams might push the use of such tools Tell me you're Shadow IT without telling me you're Shadow IT. I know legitimizing shadow IT is still the value proposition of AWS to a lot of organizations. But it sucks if that's the reason the rest of us can't get an optional feature.
• scott_w 2y ago
  > Even having the option of a hard spend limit would be hazardous, because accounting teams might push the use of such tools, and thereby risk data loss incidents when problems happen. This is an interesting point and something I can totally imagine happening! I guess if you have fixed spending limits in a large enough organisation, you lose some of the benefit of cloud infrastructure. Convincing a (metaphorically) remote Finance department to increase your fixed spending limit is probably a tougher task than ordering a load of new hardware!
• traceroute66 2y ago
  > Hard spend limits are an anti-feature for enterprise customers Yada yada yada, that's the same old excuse the cloud providers trot out. Now, forgive me for my clearly Nobel Prize winning levels of intellect when I point out the following... Number one: You would not have to turn on the hard spend limit if such functionality were to be provided. Number two: You could enable customers to set up hard limits IN CONJUNCTION WITH alerts and soft limits, i.e. hitting the hard limit would be the last resort. A bit like trains hitting the buffers at a station ... that is preferable to killing people at the end of the platform. The same with hard spend limits, hitting the limit is better than waking up in the morning to a $1m cloud bill.
  • Terretta 2y ago
    There's no way to implement a hard limit without getting in the middle of your system in ways that (a) alter the system design, (b) in ways you cannot correct for, and (c) not for the better.
    • sigseg1v 2y ago
      In my experience every AWS service I've worked with has an API to destroy the resource, eg destroy RDS instance without backup, terminate EC2, etc. I want it to irrecoverably destroy everything in my account when it hits the limit, no questions asked.
    • fallingsquirrel 2y ago
      Of course there is. If someone hits their spending limit, asynchronously shut off the services (using the same API call that your customers can use, so no need to alter the system). Then apply the hard limit in the billing code. If it took a minute or two to shut off all the instances, maybe the customer's bill should have been $1.001M instead of $1M, but cap the bill to $1M anyway. Given their profit margins of x,000% I think they can afford the lost pennies.
    • traceroute66 2y ago
      > There's no way to implement a hard limit without ... I don't believe that for a minute. You know why ? Let's turn it on its head. What happens if the credit card on your AWS account expires or is frozen ? You think AWS are going to continue letting you rack up money with no card to charge against ? I betcha they'll freeze your AWS assets the nanosecond they can't get a charge against the card. The mechanism for customer-defined hard limits is basically no different.
• GTP 2y ago
  > Almost no level of accidental spend is worth creating downtime or data loss in a critical application But not all applications are critical, and the company deploying those applications should be able to differentiate between what's critical and what's not. If they're unable to, that's their fault. If there's no option to set hard limits, that's AWS' fault.
• moribunda 2y ago
  Well, then enterprise won't use it. OTOH for example the default quota of 55PB (yes, go check it) for your daily limit of data extraction from GBQ is funny, until you make a costly mistake, or some forked process turns zombie. This is predatory practice, that I can't set up MONEY limits for cloud services.