5 ms·

Planning for an AWS outage is a complete waste of time and energy for most companies. Yes it does happen but very rarely to the tune of a few hours every 5-10 y

• jacquesm 1y ago
  Thank you for illustrating my point. You didn't even bother to read the second paragraph.
  • shawabawa3 1y ago
    > Now imagine for a bit that it will never come back up. See where that leads you. The internet got its main strengths from the fact that it was completely decentralized. We've been systematically eroding that strength. my business contingency plan for "AWS shuts down and never comes back up" is to go bankrupt
    • jacquesm 1y ago
      Is that also your contingency plan for 'user uploads objectionable content and alerts Amazon to get your account shut down'? Make sure you let your investors know.
      • padjo 1y ago
        If your mitigation for that risk is to have an elaborate plan to move to a different cloud provider, where the same problem can just happen again, then you’re doing an awful job of risk management.
        • jacquesm 1y ago
          > If your mitigation for that risk is to have an elaborate plan to move to a different cloud provider, where the same problem can just happen again, then you’re doing an awful job of risk management. Where did I say that? If I didn't say it: could you please argue in good faith. Thank you.
          • matsemann 1y ago
            "Is that also your contingency plan if unrelated X happens", and "make sure your investors know" are also not exactly good faith or without snark, mind you. I get your point, but most companies don't need Y nines of uptime, heck, many should probably not even use AWS, k8s, serverless or whatever complicated tech gives them all these problems at all, and could do with something far simpler.
            • jacquesm 1y ago
              The point is, many companies do need those nines and they count on AWS to deliver and there is no backup plan if they don't. And that's the thing I take issue with, AWS is not so reliable that you no longer need backups.
              • padjo 1y ago
                My experience is that very few companies actually need those 9s. A company might say they need them, but if you dig in it turns out the impact on the business of dropping a 9 (or two) is far less than the cost of developing and maintaining an elaborate multi-cloud backup plan that will both actually work when needed and be fast enough to maintain the desired availability. Again, of course there are exceptions, but advising people in general that they should think about what happens if AWS goes offline for good seems like poor engineering to me. It’s like designing every bridge in your country to handle a tomahawk missile strike.
                • jacquesm 1y ago
                  HN denizens are more often than not founders of exactly those companies that do need those 9's. As I wrote in my original comment: the founders are usually shocked at the thought that such a thing could happen and it definitely isn't a conscious decision that they do not have a fall-back plan. And if it was a conscious decision I'd be fine with that, but it rarely is. About as rare as companies that have in fact thought about this and whose incident recovery plans go further than 'call George to reboot the server'. You'd be surprised how many companies have not even performed the most basic risk assessment.
  • chanux 1y ago
    I get you. I am with you. But isn't money/resources always a constraint to have a solid backup solution? I guess the reason why people are not doing it is because it hasn't been demonstrated it's worth it, yet! I've got to admit though, whenever I hear about having a backup plan I think having apples to apples copy elsewhere which is probably not wise/viable anyway. Perhaps having just enough to reach out to the service users/customers suffice. Also I must add I am heavily influenced by a comment by Adrian Cockroft on why going multi cloud isn't worth it. He worked for AWS (at the time at least) so I should have probably reached to the salt dispenser.
  • mlrtime 1y ago
    We all read it.. AWS not coming back up is your point on nat having a backup plan? You might as well say the entire NY + DC metro losses power and "never comes back up" What is the plan around that? The person replying is correct, most companies do not have a actionable plan for AWS never coming back up. I worked at a medium-large company and was responsible for reviewing the infrastructure BCP plan. It stated that AWS going down was a risk, and if it happens we wait for it to come back up. (In a lot more words than that).
• antihero 1y ago
  My website running on an old laptop in my cupboard is doing just fine.
  • whatevaa 1y ago
    When your laptop dies it's gonna be a pretty long outage too.
    • deleted 1y ago
      [deleted]
    • antihero 1y ago
      I will find another one
  • api 1y ago
    I have this theory of something I call “importance radiation.” An old mini PC running a home media server and a Minecraft server will run flawlessly forever. Put something of any business importance on it and it will fail tomorrow. Related, I’m sure, is the fact that things like furnaces and water heaters will die on holidays.
    • jacquesm 1y ago
      That's a great concept. It explains a lot, actually!
• davedx 1y ago
  > I can almost guarantee that whatever plans you have won’t get you fully operational faster than just waiting for AWS to fix it. Absurd claim. Multi-region and multi-cloud are strategies that have existed almost as long as AWS. If your company is in anything finance-adjacent or critical infrastructure then you need to be able to cope with a single region of AWS failing.
  • afro88 1y ago
    > If your company is in anything finance-adjacent or critical infrastructure GP said: > most companies Most companies aren't finance-adjacent or critical infrastructure
  • padjo 1y ago
    It’s not absurd, I’ve seen it happen. Company executes on their DR plan due to AWS outage, AWS is back before DR is complete, DR has to be aborted, service is down longer than if they’d just waited. Of course there are cases when multi cloud makes sense, but it is in the minority. The absurd claim is that most companies should worry cloud outages and plan for AWS to go offline for ever.
  • philipallstar 1y ago
    > If your company is in anything finance-adjacent or critical infrastructure then you need to be able to cope with a single region of AWS failing That still fits in with "almost guarantee". It's not as though it's true for everyone, e.g. people who might trigger DR after 10 minutes of downtime, and have it up and running within 30 more minutes. But it is true for almost everyone, as most people will trigger it after 30 minutes or more, which, plus the time to execute DR, is often going to be far less than the AWS resolution time. Best of all would be just multi-everything services from the start, and us-east-1 is just another node, but that's expensive and tricky with state.
  • ants_everywhere 1y ago
    I thought we were talking about an AWS outage, not just the outage of a single region? A single region can go out for many reasons, including but not limited to war.
• mlrtime 1y ago
  >Yes it does happen but very rarely to the tune of a few hours every 5-10 years. It is rare, but it happens at LEAST 2-3x a year. AWS us-east-1 has a major incident affecting multiple services (that affect most downstream aws services) multiple times a year. Usually never the same root cause. Not very many people realize that there are some services that still run only in us-east-1.
  • energy123 1y ago
    It happens 2-3x a year during peacetime. Tail events are not homogeneously distributed across time.
    • mlrtime 1y ago
      I don't understand, peacetime?
      • mbreese 1y ago
        Peacetime = When not actively under a sustained attack by a nation-state actor. The implication being, if you expect there to be a “wartime”, you should also expect AWS cloud outages to be more frequent during a wartime.
        • mrits 1y ago
          It makes a lot more sense if they had a typo of peak
        • smaudet 1y ago
          Don't forget stuff like natural disasters and power failures...or just a very adventurous squirrel. AWS (over-)reliance is insane...
        • lII1lIlI11ll 1y ago
          What about being actively attacked by multinational state or an empire? Does it count or not? Why people keep using "nation-state" term incorrectly in HN comments is beyond me...
• lucideer 1y ago
  > to the tune of a few hours every 5-10 years I presume this means you must not be working for a company running anything at scale on AWS.
  • skywhopper 1y ago
    That is the vast majority of customers on AWS.
    • lucideer 1y ago
      Ha ha, fair, fair.
• sreekanth850 1y ago
  Depends on how serious you are with SLA's.
• kelseydh 1y ago
  It seems like this can be mostly avoided by not using us-east-1.
• psychoslave 1y ago
  This is planning future based on best things in the past. Not completely irrational and if you can't afford plan B, okayish. But thinking AWS SLA is granted forever, and everyone should put all its eggs in it because "everyone do it" is neither wise not safe. Those who can afford it, and there are many businesses like that out there, should have a plan B. And actually AWS should not necessarily be plan A. Nothing is forever. Not the Roman empire, not Inca empire, not china dynasties, not USA geological supremacy. That's not a question of if but when. It doesn't need to be through a lot of suffering, but if we don't systematically organise for a humanity which spread well being for everyone in a systematically resilient way, we will make it through more lot of tragic consequences when this or that single point of failure finally falls.
• Waterluvian 1y ago
  Without having a well-defined risk profile that they’re designing to satisfy, everyone’s just kind of shooting from the hip with their opinions on what’s too much or too little.
  • AndrewThrowaway 1y ago
    It is like discussing zombie apocalypse. People who are invested in bunkers will hardly understand those who are just choosing death over living in those bunkers for a month longer.
  • hnlmorg 1y ago
    Exactly this! One of my projects is entirely hosted on S3. I don't care enough if it becomes unavailable for a few hours to justify paying to distribute it to GCP et al. And actually for most companies, the cost of multi-cloud is greater than the benefits. Particularly when those larger entities can just bitch to their AWS account manager to get a few grand refunded as credits.
• DiffEq 1y ago
  Maybe; but Parlar had no plan and are now nothing....because AWS decided to shut them off. Always have a good plan...
• nucleardog 1y ago
  > Planning for an AWS outage is a complete waste of time and energy for most companies. Yes it does happen but very rarely to the tune of a few hours every 5-10 years. Not only that, but as you're seeing with this and the last few dozen outages... when us-east-1 goes down, a solid chunk of what many consumers consider the "internet" goes down. It's perceived less as "app C is down" and more is "the internet is broken today".
  • jgwil2 1y ago
    Isn't the endpoint of that kind of thinking an even more centralized and fragile internet?
    • nucleardog 1y ago
      To be clear, I'm not advocating for this or trying to suggest it's a good thing. That's just reality as I see it. If my site's offline at the same time as the BBC has front page articles about how AWS is down and it's broken half the internet... it makes it _really_ easy for me to avoid blame without actually addressing the problem. I don't need to deflect blame from my customers. Chances are they've already run into several other broken services today, they've seen news articles about it, and all from third parties. By the time they notice my service is down, they probably won't even bother asking me about it. I can definitely see this encouraging more centralization, yes.
• YouAreWRONGtoo 1y ago
  More like 2-3 times per year and this is not counting smaller outages or simply APIs that don't do what they document.
  • sgarland 1y ago
    > APIs that don’t do what they document Oh god, this. At my company, we found a bug recently with rds.describe_events, which we needed to read binlog information after a B/G cutover. The bug, which AWS support “could not see the details of,” was that events would non-deterministically not show up if you were filtering by instance name. Their recommended fix was to pull in all events for the past N minutes, and do client-side filtering. This was on top of the other bug I had found earlier, which was that despite the docs stating that you can use a B/G as a filter - a logical choice when querying for information directly related to the B/G you just cut over - doing so returns an empty set. Also, you can’t use a cluster (again, despite docs stating otherwise), you have to use the new cluster’s writer instance.
    • YouAreWRONGtoo 1y ago
      While I don't know your specific case, I have seen it happen often enough that there are only two possibilities left: 1. they are idiots 2. they do it on purpose and they think you are an idiot For me, it just means that the moment you integrate with any API, you are basically their bitch (unless you implement one from every competitor in the market, at which point you can just as well do it yourself).
• kxrm 1y ago
  Completely agree, but I think companies need to be aware of the AWS risks with third parties as well. Many services were unable to communicate with customers. Hosting your services on AWS while having a status page on AWS during an AWS outage is an easily avoidable problem.
• snowwrestler 1y ago
  I would take the opposite view, the little AWS outages are an opportunity to test your disaster recovery plan, which is worth doing even if it takes a little time. It’s not hard to imagine events that would keep AWS dark for a long period of time, especially if you’re just in one region. The outage today was in us-east-1. Companies affected by it might want to consider at least geographic diversity, if not supplier diversity.
  • SteveNuts 1y ago
    > Companies affected by it might want to consider at least geographic diversity, if not supplier diversity. Sure, it's worth considering, but for most companies it's not going to be worth the engineering effort to architect cross-cloud services. The complexity is NOT linear. IMO most shops should focus on testing backups (which should be at least cross-cloud, potentially on-prem of some sort) to make sure their data integrity is solid. Your data can't be recreated, everything else can be rebuilt even if it takes a long time.
• zaphirplane 1y ago
  > tune of a few hours every 5-10 years You know that’s not true, is-east-1 last one was 2 years ago? But other services have bad days and foundational one drag others a long
  • Esophagus4 1y ago
    It’s even worse than that - us-east-1 is so overloaded, and they have roughly 5+ outages per year on different services. They don’t publish outage numbers so it’s hard to tell. At this point, being in any other region cuts your disaster exposure dramatically
  • coffeebeqn 1y ago
    We don’t deploy to us-east but still so many of our API partners and 3rd party services were down a large chunk of the service was effectively down. Including stuff like many dev tools
• Spooky23 1y ago
  Sure, if your blog or whatever goes down who cares. But otherwise you should thinking about disaster planning and resilience. AWS US-East 1 has many outages. Anything significant should account for that.
• lumost 1y ago
  This depends on the scale of company. A fully functional DR plan probably costs 10% of the infra spend + people time for operationalization. For most small/medium businesses its a waste to plan for a once per 3-10 year event. If you’re a large or legacy firm the above costs are trivial and in some cases it may become a fiduciary risk not to take it seriously.
  • jacquesm 1y ago
    And if you're in a regulated industry it might even be a hard requirement.
• pyrale 1y ago
  What if AWS dumps you because your country/company didn't please the commander in chief enough? If your resilience plan is to trust a third party, that means you don't really care about going down, does it? Besides that, as the above poster said, the issue with top tier cloud providers (or cloudflare, or google, etc) is not just that you rely on them, it is that enough people rely on them that you may suffer even if you don't.
• temperceve 1y ago
  Depends on the business. For 99% of them this is for sure the right answer.
• throw0101d 1y ago
  > Planning for an AWS outage […] What about if your account gets deleted? Or compromised and all your instances/services deleted? I think the idea is to be able to have things continue running on not-AWS.
  • rjmunro 1y ago
    This. I wouldn't try to instantly failover to another service if AWS had a short outage, but I would plan to be able to recover from a permanent AWS outage by ensuring all your important data and knowledge is backed up off-AWS, preferably to your own physical hardware and having a vague plan of how to restore and bring things up again if you need to. "Permanent AWS outage" includes someone pressing the wrong button in the AWS console and deleting something important or things like a hack or ransomware attack corrupting your data, as well as your account being banned or whatever. While it does include AWS itself going down in a big way, it's extremely unlikely that it won't come back, but if you cover other possibilities, that will probably be covered too.
• maerF0x0 1y ago
  Using AWS instead of a server in the closet is step 1. Step 2 is multi-AZ Step 3 is multi-region Step 4 is multi-cloud. Each company can work on it's next step, but most will not have positive EROI going from 2 to 3+
  • joshuat 1y ago
    Multi-cloud is a hole in which you can burn money and not much more
  • deleted 1y ago
    [deleted]
• dangoldin 1y ago
  I worked at an adtech company where we invested a bit in HA across AZ + regions. Lo and behold there was an AWS outage and we stayed up. Too bad our customers didn't and we still took the revenue hit. Lesson here is that your approach will depend on your industry and peers. Every market will have their won philosophy and requirements here.
• delfinom 1y ago
  In before meteor strike takes a AWS region and they cant restore data.
• coffeebeqn 1y ago
  We started that planning process at my previous company after one such outage but it became clear very quickly that the costs of such resilience would be 2-3x hosting costs in perpetuity and who knows how many manhours. Being down for an hour was a lot more palatable to everyone
• indoordin0saur 1y ago
  Been doing this for about 8 years and I've worked through a serious AWS disruption at least 5 times in that time.
• croes 1y ago
  Telefonica is moving it 5G core network to AWS https://aws.amazon.com/blogs/industries/o2-telefonica-moves-its-5g-core-network-to-the-cloud-with-aws-and-nokia/ https://aws.amazon.com/blogs/industries/o2-telefonica-moves-... A few hours could be a problem. Not to mention it creates valuable a single point of failure for a hostile attack.