7 ms·

They should give you the option to set a hard limit across your entire account, to prevent you from accidentally spending more money than you have. "If I try to

• umvi 7y ago
  Seems like circuit breakers should be a standard safety feature for automatically infinitely scaling computers. I would rather my whole system shut down and be unusable while I investigate vs. auto-scale and charge me a bill I can't cover. However, searching around it seems like I can only get alerts when a $$$ threshold is passed, but AWS won't take any action to stop computing or anything. Please prove me wrong.
  • samstave 7y ago
    they already are.... but you also need to put a bit more effort into it. For example, you shut down the ability to launch anything in a particular region easily - but assume you specifically want to exceed a default limit - you can call speak with your rep and have any of your limits set to whatever you want
  • ghaff 7y ago
    >I would rather my whole system shut down and be unusable while I investigate vs. auto-scale and charge me a bill I can't cover. The counterargument is that you get a usage spike (which is often a good thing for a company), and AWS shuts down everything connected to your AWS account without warning. I'm not necessarily sure that optional/non-default hard circuit breakers would be a bad thing. But it certainly appears not to be a heavily demanded customer feature and, honestly, if it's not the default--which is shouldn't be--I wonder how many customers, or at least customers the cloud providers really care about, would use them.
    • _bxg1 7y ago
      You'd factor that in to the ceiling you set. Maybe your ceiling is 2x or 3x your expected usage. That could still be low enough not to bankrupt your company.
    • kortilla 7y ago
      The usage spike is very very rarely worth the cost. That’s a pipe dream the cloud providers sell to cover up the fact that these scenarios are sweet sweet profit for them and nothing more. There are very few businesses where making more money is just a matter of throwing some more compute at it. Nearly every customer (i.e. all of them with a budget) would make use of circuit breakers and it would make Amazon absolutely $0 while costing them untold amounts. Are you really surprised Amazon hasn’t implemented them?
      • _bxg1 7y ago
        If enough people demand them it could become a competitive advantage in the already-cutthroat cloud hosting market
      • maerF0x0 7y ago
        I imagine if usage spikes are not valuable and uncommon then static resources could be less expensive to provision, right? For example Vultr can give you a "bare metal" 8vcpu-32GB box for $120 a month (Not sure if this is contract or on-demand) vs amazons M5.2xlarge for $205 reserved. $80 might not sound like much, but that's 70% more. Who would love to save ~42% on their cloud costs?
        • deleted 7y ago
          [deleted]
      • scarface74 7y ago
        How much more does it costs AWS to allow you to spin up resources and then liberally offer refunds when you contact them and tell them you made a mistake?
        • ProZsolt 7y ago
          Not much. Some electricity. Servers are there either way.
          • scarface74 7y ago
    • danmg 7y ago
      Most cellphone providers provide you with a text message when you're over 90% of your hard cap, and you can login and buy more bandwidth if you really need it. The same could be done with cloud doo-dads.
      • threentaway 7y ago
        You can absolutely set this up in AWS and GCP.
      • rhizome 7y ago
        Amazon isn't going to put much effort into automating reminders for you to keep your bill low.
        • WrtCdEvrydy 7y ago
          Actually, Budgets allow you to feed an SNS that will dispatch reminders at any point (10%, 50%, 100%) of a total spend amount. Sadly, some services take as long as 24 hours to report billing.
  • scarface74 7y ago
    There are circuit breakers. AWS has plenty of “soft limits” that you have to request to increase including the number of EC2 instances you can spin up.
    • beagle3 7y ago
      And yet, the simplest circuit breaker, total spend, is well hidden if it exists at all, judging from all the horror stories.
      • threentaway 7y ago
        It's not hidden, it's well documented. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...
        • ghaff 7y ago
          Those are alerts though, not circuit breakers. (Although alerts can be used to trigger certain types of circuit breakers.) What some people are asking for--and it's a reasonable use case but one that AWS, somewhat understandably, isn't really focused on--is: "Burn it all down if you have to, including deleting databases, but no way no how charge me a penny more than $100/month (or whatever my set limit is)."
          • beagle3 7y ago
            The circuitbreaker everyone would want is probably: "Drop all compute with its ephemeral storage, when the cost of every resource already used and the projected cost of my stable storage (EBS+S3+Glacier+Aurora+Whatnot) is greater than my preset $MONTHLY_MAXIMUM. That means any data getting lost as a result of that limit is data that they weren't guaranteeing in the first place. You might not be able to actually read your EBS volumes, S3 buckets or Aurora tables without increasing the spending limit or otherwise committing more funds, but it won't go away that second, and you would have enough time to fix it (worst case - wait until next month; you did budget that already). Alternatively: assign each resource to a pool, and monthly spending limits to each pool. Give your EBS/S3 $1000/month, and your R&D-pet-project-that-may-accidentally-spawn-a-billion-machines $50/month.
  • txcwpalpha 7y ago
    >I would rather my whole system shut down and be unusable while I investigate vs. auto-scale and charge me a bill I can't cover. Sure, but most large companies (the kind which AWS gets a lot more revenue from and cares about a lot more) want the exact opposite. Most large companies have the extra cash to spend in the case that a runwaway auto-scale was in error, but on the other hand, completely shutting down operations whenever a traffic spike happens could resort in millions of lost revenue. >However, searching around it seems like I can only get alerts when a $$$ threshold is passed, but AWS won't take any action to stop computing or anything. Please prove me wrong. The general advice is to use the various kinds of usage alerts (billing alerts, forecasts, etc) to trigger Lambda functions that shut down your instances as you desire. It takes a little configuration on your part, but again, AWS intentionally errs on the side of not automatically turning off your instances unless you specifically tell it to.
    • ISL 7y ago
      If the customer dies, the lifetime value of the customer is almost-certainly lower. Great companies find ways to help their customers thrive.
      • perl4ever 7y ago
        I don't know what Amazon actually does, but what I think of as normal is the customer calls the help desk and they reverse the charge. This seems simpler than worrying about how to code algorithms that will deal with all possibilities. People make mistakes with transferring money in the millions of dollars all the time, and it's not uncommon for people to be just like "oops, back that out". It's obviously going to be in the news when that doesn't happen though.
    • gray_-_wolf 7y ago
      > Sure, but most large companies (the kind which AWS gets a lot more revenue from and cares about a lot more) want the exact opposite. It does not have to be all or nothing. You could for example setup separate account per department and/or purpose and impose hard cap on spending for experimentation, but not on production.
• thanksforfish 7y ago
  The argument I've heard against this is that for certain types of spending, cutting things off causes data loss. Automatically doing that is also a foot gun. The proposed solution is usually to setup billing alerts so you can detect the issue early and fix the problem in a way that makes sense. I'd suggest further: new AWS (azure, etc) accounts should have billing alarms at $0, $10, $100, $1000, etc. by default on account creation. Users can delete if they don't want them or want something different. Getting an alert at $100 as it happens instead of getting a >$1k bill at the end of the month is a much better customer experience.
  • mandelbrotwurst 7y ago
    They could include it as an option with a big red warning advising against it. Depending on what you're doing, data loss might not be nearly as big of a threat as a massive bill. I could also imagine it being configurable on a service by service basis to mitigate against the data loss downside - e.g. maybe you have a hard cap on your lambdas but not your database snapshots.
  • silentsea90 7y ago
    the solution is "you do you". Allow cloud customers to choose the behavior of whether they're good with just alerting or if they want a circuit breaker to act, or do both. I am sure there is an upper bound of spending where if you were asleep and not reacting to notifications, you'd be better off having the system die instead of kill your company or make you bankrupt.
    • thanksforfish 7y ago
      Isn't that where we are now? Choose your desired behavior seems equivalent to "here's billing alerts and AWS API, good luck". Someone could build a cost circuit break lambda function fairly easily. Wire a billing alert to the lambda, use AWS API to terminate all EC2 instances (or other resources). Someone could open source that fairly easily. I think it's about reasonable defaults. You can recommend that customers configure their accounts with the behaviors they choose, but clearly many aren't using the tools they currently have. So we have these horror stories. I will mention that I don't think deleting resources is a good default behavior as that can cause data loss (itself a potentially business ending event). But people are certainly able to evaluate their own risk and can implement that with the tools that exist today.
• Thaxll 7y ago
  So you set a hard limits then suddently storage doesn't work anymore, queries don't get routed etc ... really bad and weird behaviour ensure.
  • silentsea90 7y ago
    sure, but you'd imagine basic alerting to tell you the circuit breaker in action right now. Depending on your application, bad behavior is better than a dead company.
  • ceejayoz 7y ago
    "We went bankrupt overnight due to a runaway script" is "really bad and weird behavior"...
    • Thaxll 7y ago
      There is no such a thing xyz went bankrupt because of cloud over provisioning. First of all we don't have the whole story from OP and I suspect he's not telling everything, second AWS will cover those accidents, third by default you can't create many resources on an account with default quota.
      • ceejayoz 7y ago
        AWS may cover those accidents, and you can get the default quotas raised quite a bit. I agree that some info is probably missing here, but it's not entirely implausible.
• silentsea90 7y ago
  I was typing the exact same solution when I saw your response. Completely agree that this should exist.
• samstave 7y ago
  FYI all limis are maleable - and you CAN request limits increase/decrease as you desire - you just typically cant do it in the console - but you can speak to your rep and have it done
• blargmaster33 7y ago
  Azure has this, budgets.
• 6gvONxR4sf7o 7y ago
  Too bad you can't segment it to have a hard limit on the non-essential stuff. So at least you can serve the webpage saying what you are or something.