5 ms·

Seems like circuit breakers should be a standard safety feature for automatically infinitely scaling computers. I would rather my whole system shut down and be

• samstave 7y ago
  they already are.... but you also need to put a bit more effort into it. For example, you shut down the ability to launch anything in a particular region easily - but assume you specifically want to exceed a default limit - you can call speak with your rep and have any of your limits set to whatever you want
• ghaff 7y ago
  >I would rather my whole system shut down and be unusable while I investigate vs. auto-scale and charge me a bill I can't cover. The counterargument is that you get a usage spike (which is often a good thing for a company), and AWS shuts down everything connected to your AWS account without warning. I'm not necessarily sure that optional/non-default hard circuit breakers would be a bad thing. But it certainly appears not to be a heavily demanded customer feature and, honestly, if it's not the default--which is shouldn't be--I wonder how many customers, or at least customers the cloud providers really care about, would use them.
  • _bxg1 7y ago
    You'd factor that in to the ceiling you set. Maybe your ceiling is 2x or 3x your expected usage. That could still be low enough not to bankrupt your company.
  • kortilla 7y ago
    The usage spike is very very rarely worth the cost. That’s a pipe dream the cloud providers sell to cover up the fact that these scenarios are sweet sweet profit for them and nothing more. There are very few businesses where making more money is just a matter of throwing some more compute at it. Nearly every customer (i.e. all of them with a budget) would make use of circuit breakers and it would make Amazon absolutely $0 while costing them untold amounts. Are you really surprised Amazon hasn’t implemented them?
    • _bxg1 7y ago
      If enough people demand them it could become a competitive advantage in the already-cutthroat cloud hosting market
    • maerF0x0 7y ago
      I imagine if usage spikes are not valuable and uncommon then static resources could be less expensive to provision, right? For example Vultr can give you a "bare metal" 8vcpu-32GB box for $120 a month (Not sure if this is contract or on-demand) vs amazons M5.2xlarge for $205 reserved. $80 might not sound like much, but that's 70% more. Who would love to save ~42% on their cloud costs?
      • deleted 7y ago
        [deleted]
    • scarface74 7y ago
      How much more does it costs AWS to allow you to spin up resources and then liberally offer refunds when you contact them and tell them you made a mistake?
      • ProZsolt 7y ago
        Not much. Some electricity. Servers are there either way.
        • scarface74 7y ago
          Exactly. They decided it was cheaper for them to let you make mistakes and then grant refunds.
    • lostlogin 7y ago
      > Are you really surprised Amazon hasn’t implemented them? It become harmful to them though. At a certain point people feel the hit and avoid the service. Having people spent a little more accidentally and go ‘oh well, oops’ is the sweet spot. An unexpected $80k which kills the company is bad for everyone.
  • danmg 7y ago
    Most cellphone providers provide you with a text message when you're over 90% of your hard cap, and you can login and buy more bandwidth if you really need it. The same could be done with cloud doo-dads.
    • threentaway 7y ago
      You can absolutely set this up in AWS and GCP.
    • rhizome 7y ago
      Amazon isn't going to put much effort into automating reminders for you to keep your bill low.
      • WrtCdEvrydy 7y ago
        Actually, Budgets allow you to feed an SNS that will dispatch reminders at any point (10%, 50%, 100%) of a total spend amount. Sadly, some services take as long as 24 hours to report billing.
• scarface74 7y ago
  There are circuit breakers. AWS has plenty of “soft limits” that you have to request to increase including the number of EC2 instances you can spin up.
  • beagle3 7y ago
    And yet, the simplest circuit breaker, total spend, is well hidden if it exists at all, judging from all the horror stories.
    • threentaway 7y ago
      It's not hidden, it's well documented. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...
      • ghaff 7y ago
        Those are alerts though, not circuit breakers. (Although alerts can be used to trigger certain types of circuit breakers.) What some people are asking for--and it's a reasonable use case but one that AWS, somewhat understandably, isn't really focused on--is: "Burn it all down if you have to, including deleting databases, but no way no how charge me a penny more than $100/month (or whatever my set limit is)."
        • beagle3 7y ago
          The circuitbreaker everyone would want is probably: "Drop all compute with its ephemeral storage, when the cost of every resource already used and the projected cost of my stable storage (EBS+S3+Glacier+Aurora+Whatnot) is greater than my preset $MONTHLY_MAXIMUM. That means any data getting lost as a result of that limit is data that they weren't guaranteeing in the first place. You might not be able to actually read your EBS volumes, S3 buckets or Aurora tables without increasing the spending limit or otherwise committing more funds, but it won't go away that second, and you would have enough time to fix it (worst case - wait until next month; you did budget that already). Alternatively: assign each resource to a pool, and monthly spending limits to each pool. Give your EBS/S3 $1000/month, and your R&D-pet-project-that-may-accidentally-spawn-a-billion-machines $50/month.
• txcwpalpha 7y ago
  >I would rather my whole system shut down and be unusable while I investigate vs. auto-scale and charge me a bill I can't cover. Sure, but most large companies (the kind which AWS gets a lot more revenue from and cares about a lot more) want the exact opposite. Most large companies have the extra cash to spend in the case that a runwaway auto-scale was in error, but on the other hand, completely shutting down operations whenever a traffic spike happens could resort in millions of lost revenue. >However, searching around it seems like I can only get alerts when a $$$ threshold is passed, but AWS won't take any action to stop computing or anything. Please prove me wrong. The general advice is to use the various kinds of usage alerts (billing alerts, forecasts, etc) to trigger Lambda functions that shut down your instances as you desire. It takes a little configuration on your part, but again, AWS intentionally errs on the side of not automatically turning off your instances unless you specifically tell it to.
  • ISL 7y ago
    If the customer dies, the lifetime value of the customer is almost-certainly lower. Great companies find ways to help their customers thrive.
    • perl4ever 7y ago
      I don't know what Amazon actually does, but what I think of as normal is the customer calls the help desk and they reverse the charge. This seems simpler than worrying about how to code algorithms that will deal with all possibilities. People make mistakes with transferring money in the millions of dollars all the time, and it's not uncommon for people to be just like "oops, back that out". It's obviously going to be in the news when that doesn't happen though.
  • gray_-_wolf 7y ago
    > Sure, but most large companies (the kind which AWS gets a lot more revenue from and cares about a lot more) want the exact opposite. It does not have to be all or nothing. You could for example setup separate account per department and/or purpose and impose hard cap on spending for experimentation, but not on production.