10 ms·

If you haven't seen yet, news is it was a power loss: > 5:01 AM PST We can confirm a loss of power within a single data center within a single Availability Zon

• codeduck 5y ago
  another example of a single dc in a single AZ rendering an entire region almost unusable. This has shades of eu-central-1 all over again.
  • nightpool 5y ago
    Amazon is claiming the failure is limited to a single AZ. Are you seeing failures for instances outside of that AZ? If not, how has this rendered "the entire region almost unusable"?
    • londons_explore 5y ago
      A lot of people will automatically fail over jobs to other AZ's. That often involves spinning up lots more EC2 instances and moving PB's of data. The end result is all capacity on other AZ's gets used up, and networks get full to capacity, and even if those other zones are technically working, practically they aren't really usable.
      • Godel_unicode 5y ago
        That doesn't appear to have happened though, I haven't seen issues outside az4
      • reilly3000 5y ago
        While there may be more machines provisioned, many orgs run active setups for failover so they aren’t as affected. In terms of data transfer, it should already be there. Where would it come from? Certainly not the dead AZ.
        • manquer 5y ago
          It is Amazon's services themselves which are advertised multi-AZ that would do bulk of this thundering hurd kind of requests.
    • tyingq 5y ago
      Perspective, I would guess. Unless you spend a lot of time on retry/timeout/fail logic around AWS apis, your app could be stuck/blocked in the RunInstances() api, for example.
    • matharmin 5y ago
      Yes, I've seen issues that affected the entire region. In my specific case, I happened to have an ElastiCache cluster in the affected AZ that became unreachable (my fault for single AZ). But even now, I'm unable to create any new ElastiCache clusters in different AZs (which I wanted to use for manual failover). And there were a lot of errors on the AWS console during the outage. "almost unusable" is maybe exaggerating, but there were definitely issues affecting more than just the single AZ.
      • deleted 5y ago
        [deleted]
      • wizwit999 5y ago
        That seems acceptable. The Data plane failure is contained to an AZ. Control plane is often not.
      • jedberg 5y ago
        Probably because you aren’t the only one trying to do that. The folks who successfully fail over a zone are the ones who have already automated the process and are running active/active configurations so everything is set up and ready to go.
        • bobby_the_whale 5y ago
    • codeduck 5y ago
      We've had alerts for packet loss and had issues in recovering region-spanning services (both AWS and 3rd party). Yes, some of these we should be better at handling ourselves, but... it's all very well to say "expect to lose an AZ" but during this outage it's not been physically possible to remove the broken AZ instances from multi-AZ services because we cannot physically get them to respond to or acknowledge commands. edit: just to short circuit any "well, why aren't you running redundant regions" - we run redundant regions at all times. But for reasons of latency, many customers will bind to their closest region, and the nature of our technology is highly location-bound It is not possible for us to move active sessions to an alternate region. So something like this is... unpleasant.
      • bobby_the_whale 5y ago
      • mentat 5y ago
        You don't have health checks?
        • codeduck 5y ago
          How are health checks supposed to help when you can't do anything?
          • buchanmilne 5y ago
            You said: > it's all very well to say "expect to lose an AZ" but during this outage it's not been physically possible to remove the broken AZ instances from multi-AZ services because we cannot physically get them to respond to or acknowledge commands "Expect to lose an AZ" includes not being able to make any changes to existing instances in the affected AZ. If you had instances across multiple AZs behind an ELB with health checks, then the ELB should automatically remove the affected instances. If you have a different architecture, you would want to: * Have another mechanism that automatically stops sending traffic to impaired instances (ideal), or * Have a means to manually remove the instances from service without being able to interact with or modify those instances in any way Does that help, or have I misunderstood your problem?
• GrumpyNl 5y ago
  How come they dont have power backups?
  • trelane 5y ago
    Anything can fail, even your backup, and especially if it's mechanical.
    • rdines 5y ago
      The battery backups (called uninterruptible power supplies) are only meant to bridge the gap between the power going out and the generator turning on, which is a few minutes. Did they say power was the issue this time? I suspect it’s actually something else (ahem network)
  • chkhd 5y ago
    "When a fail-safe system fails, it fails by failing to fail-safe." - https://en.wikipedia.org/wiki/Systemantics https://en.wikipedia.org/wiki/Systemantics
    • 2-718-281-828 5y ago
      is that just playing with words?
      • itsoktocry 5y ago
        >is that just playing with words? It conveys reality, that "fail-safe" isn't literal, as if anyone believed that.
        • 2-718-281-828 5y ago
          I mean it has to be play with words or tongue in cheek simply b/c the assumption of a fail-safe system failing is already contradictory. So you cannot say anything smart about that beyond - there are no fail-safe systems that fail.
• vinay_ys 5y ago
  This is quite interesting as they claim their datacenter design does better than Uptime's Tier3+ design requirements which require redundant power supply paths. [https://aws.amazon.com/compliance/uptimeinstitute/ https://aws.amazon.com/compliance/uptimeinstitute/]. I really hope they publish a thorough RCA for this incident.
  • tyingq 5y ago
    "Electrical power systems are designed to be fully redundant so that in the event of a disruption, uninterruptible power supply units can be engaged for certain functions, while generators can provide backup power for the entire facility." https://aws.amazon.com/compliance/data-center/infrastructure-layer/ https://aws.amazon.com/compliance/data-center/infrastructure... So they have 2 different sources of power coming in. And generators. They do mention the UPS is only for "certain functions", so I guess it's not enough to handle full load while generators spin up if the 2 primaries go out. Or perhaps some failure in the source switching equipment (typically called a "static transfer switch"). Some detail on different approaches: https://www.donwil.com/wp-content/uploads/white-papers/Using-Static-Transfer-Switches-to-Enhance-Data-Center-Availability-and-Maintainability-Technical-Note-TN-00011.pdf https://www.donwil.com/wp-content/uploads/white-papers/Using...
    • vinay_ys 5y ago
      Usually when someone claims T3+ they mean they have UPS clusters in 3+1 (or such) configuration and two different such UPS clusters power two power-strips in a rack. Then, would also have incoming grid power supply from two different HV sub-stations with non-intersecting cable paths. They would also have diesel power generators in 3+1 or 5+2 configurations with automatic startup time in seconds. The UPS's power storage (chemical or potential energy based devices) can hold enough energy to handle full load for several minutes. If these are design and maintained correctly, even while concurrent scheduled maintenance is ongoing, an unexpected component failure should not cause catastrophic outage. At each layer (grid incomers, generator incomers, UPS power incomers) there are switches to switch over whenever there's a need (maintenance or failure). If they claim tier4, then they basically have everything in n+n configuration.
• SCdF 5y ago
  So dumb question from someone who hasn't maintained large public infrastructure: Isn't the whole point of availability zones is that you deploy to more than one and support failing over if one fails? IE why are we (consumers) hearing about this or being obviously impacted (eg Epic Games Store is very broken right now)? Is my assessment wrong, or are all these apps that are failing built wrong? Or something in between?
  • robjan 5y ago
    That's the theory but in practice very few companies bother because it's expensive, complicated and most workloads or customers can tolerate less than 100% uptime.
  • gpm 5y ago
    > or are all these apps that are failing built wrong Deploying to multiple places is more expensive, it's not wrong to choose not to, it's trading off reliability for cost. It's also unclear to me how often things fail in a way that actually only affect one AZ, but I haven't seen any good statistics either way on that one.
  • _joel 5y ago
    You're supposed to build your app across multiple AZ's but I know a lot of companies that don't do this and shove everything in a single AZ. It's not just about deploying and instance there but ensuring the consistency of data and state across the az's
  • peeters 5y ago
    As I understand it for something like SQS, Lambda etc, AWS should automatically tolerate an AZ going down. They're responsible for making the service highly available. For something like EC2 though, where a customer is just running a node on AWS, there's no automatic failover. It's a lot more complicated to replicate a running, stateful virtual machine and have it seamlessly failover to a different host. So typically it's up to the developers to use EC2 in a way that makes it easy to relaunch the nodes on a different AZ.
    • 5y ago
• xyst 5y ago
  This region in general is a clusterfuck. If companies by now do not have a disaster recovery and resiliency strategy in place, you are just shooting yourself in the foot.
  • philsnow 5y ago
    In today's world of stitching together dozens of services, who each probably do the same thing, how is one to avoid a dependency on us-east-1? Add yet another bullet to the vendor questionnaire (ugh) about whether they are singly-homed / have a failover plan? It's turtles all the way down, and underneath all the turtles is us-east-1.
    • deleted 5y ago
      [deleted]
• alostpuppy 5y ago
  Why do folks host their stuff in us-East? Is there a draw other than organizational momentum?
  • dragonwriter 5y ago
    > Why do folks host their stuff in us-East? Off the top of my head, US-EAST-1 is: (1) topologically closer to certain customers than other regions (this applies to all regions for different customers), (2) consistently in the first set of regions to get new features, (3) usually in the lowest price tier for features whose pricing varies by region, (4) where certain global (notionally region agnostic) services are effectively hosted and certain interactions with them in region-specific services need to be done. #4 is a unique feature of US-East-1, #2-#3 are factors in region selection that can also favor other regions, e.g., for users in the West US, US-West-2 beats US-West-1 on them, and is why some users topologically closer to US-West-1 favor US-West-2.
    • alostpuppy 5y ago
      Thank you! This one is why I love HN.
  • superdug 5y ago
    It's the cheapest.
    • deanCommie 5y ago
      us-east-2 has exactly the same prices as us-east-1.
      • res0nat0r 5y ago
        Most likely inertia. us-east-1 was the first AWS region, gets new features released there first and is the largest in the USA, so many companies have been running their for many years, and the cost of moving to us-east-2 > the cost of occasional AWS created downtime.
• notyourday 5y ago
  We are being told that the are still issues in the USE1-AZ4 and some of the instances are stuck in the wrong state as of 16:15 PM EST. There's no ET for resolution.