8 ms·

AWS power failure in US-EAST-1 region killed some hardware and instances

129 points by bratao 5y ago

1cvmask 5y ago
A lot more companies will go to a multi-cloud active active architecture with maybe even bare metal redundancies.
- psanford 5y ago
  I would stay away from any company that thinks "we need to go multicloud" as a response to this outage. This affected a single az in a single region. If it caused you downtime or a partial outage, it means you are not fully resilient to single az failures. The correct thing is to fix your application to handle that. If you can't handle a single az failure there is no way you are going to handle failing over across different cloud providers correctly.
  - CubsFan1060 5y ago
    There will, however, be a lot of executives _talking_ about going multi cloud.
  - luhn 5y ago
    To be fair though, from what I heard AZ outage caused an EC2 API brownout, so people couldn't launch new instances in the other AZs. That put a wrench in a lot of multi-AZ architectures. Not advocating for multi-cloud though...
  - dragonwriter 5y ago
    > If it caused you downtime or a partial outage, it means you are not fully resilient to single az failures. Given the number of AWS global services that have dependencies on infra in US-EAST-1 (and, from the impacts of this and other past outages, seen vulnerable to single-AZ failures in US-EAST-1) that's...less avoidable for certain regions/AZs than one might naively expect. Most clouds seem to have at least some degree of this kind of vulnerability.
  - nonane 5y ago
    > This affected a single az in a single region. If it caused you downtime or a partial outage, it means you are not fully resilient to single az failures. This is not true. Amazon is not being upfront about what happened here. It was simply not a single AZ failure. Our us-east-1 ELB load balancers were hosed and were unable to direct traffic to other AZs - they simply stopped working an were dropping traffic. We tried creating load balancers in different AZs and that didn't work either. How can you be resilient to single AZ failures if load balancers stop working region wide during a single AZ outage?
    - acdha 5y ago
      Did your TAM go into any details on that? Over a couple hundred load-balancers, the only issue we had was taking longer to register new instances and that affected only a couple of them. Running services weren't interrupted, latency remained, etc. which is what I'd expect for a single AZ failure.
- deleted 5y ago
  [deleted]
- tyingq 5y ago
  Multi-cloud is odd to me, unless you're a company selling a service to cloud customers. By definition, you would have to either go lowest-common-denominator, or build complicated facades in front of like services. If you're going lowest-common-denominator, then multi-old-school-hosting would be far cheaper.
  - isbvhodnvemrwvn 5y ago
    You'd have to go worse than lowest common denominator. To have active-active replication you need to take the cost of latency between the different DCs of the various providers, and this thing will kill performance.
- ransom1538 5y ago
  me: "Yeah! AWS went down a few times, I think we should pay double!" vp: no.
  - emodendroket 5y ago
    More than double realistically. But you could achieve most of the benefit at much lower cost by going multi-AZ.
  - karmasimida 5y ago
    Infra is pure cost center. A few hours downtime is not going to justify double cost, and worse whose benefits can only be demonstrated during those few hours. It will get shutdown immediately. And what is more important, corporate world really don't care that much about downtime, not as much as they care about who to assign the blame, in this case AWS is a perfect irresistible externality much like a natural disaster. Disclaimer: Ex-AWS employee
- profmonocle 5y ago
  Egress fees can make replicating databases, storage buckets, etc. between clouds very expensive. Multi-region is a much more affordable option. Multi-region outages aren't unheard of among the major cloud operators, but they're less common than single-AZ or single-region outages. IMO, most companies just aren't sensitive enough to downtime that multi-AZ + multi-region deployment within a single cloud provider isn't good enough.
- beermonster 5y ago
  As large cloud outages become more frequent and the impact greater each time, I feel it’s more likely people will reconsider moving workloads off-prem.
CoastalCoder 5y ago
Any idea why a power failure would cause (or reveal) hardware damage? Leading up to Y2K, I remember concerns about spinning hard disks not being able to start up again. And if the power is flaky with spikes and brown-outs, I understand that's a problem. But is either of those relevant to AWS?
- kube-system 5y ago
  If I were to make some WAGs about things that could generally be described this way: * increased failure rates of drives (or other hardware) due to thermal cycling * cache power failures leading to corrupted data * previously unknown failures in recovery processes (like firmware bugs that might be described as a "hardware problem") * cooling failures leading to hardware failure Most of these are pretty rare issues, but it becomes more probable to happen at least once when you're running freekin' us-east-1
- SteveNuts 5y ago
  In my experience it's common when there's a full power down of a large data center like this. In 2013 the company I was working at had an unplanned power outage which took out 200 spinning drives, and 18 servers. I think the motors in the drives are fine running at constant speed but don't have the juice to spin up from a dead stop. I remember IBM was renting private planes to get us replacement hardware from around the country (we were a very large customer).
  - jbergens 5y ago
    A very long time ago I worked for a medium sized company and there was a power outage. The servers were in a room in our building but they had ups power. The problem was that the ups's were broken/malfunctioning and the servers needed to do a disk check after an unplanned reboot. It took hours before we could work again even though most drives survived. This was also before git so we could not reach the central version control system and could not commit any code. I think all code and workspaces were on servers that was not up yet.
  - lamontcg 5y ago
daneel_w 5y ago
"As is often the case with a loss of power, there may be some hardware that is not recoverable..." No. Not even rarely. If they lost hardware because of this something much different than just loss of power happened on their servers' mains rails.
- Johnny555 5y ago
  If my home UPS failed, I'd be awfully surprised if my home fileserver that was plugged into it failed too. But if I lost power to thousands of servers, I'd expect some number of them to fail. I've even lost servers when losing power to a single rack.
- jdsully 5y ago
  Starting up is hard on electronics due to the inrush current. There may be marginal devices just barely operating that will not survive a reboot. Things like bad capacitors aren’t always detectable during steady state. At AWS scale it’s highly likely there are more than a few of these.
  - daneel_w 5y ago
    Did you think this over before hitting the reply button? Can you imagine any server or networking brand surviving if its products were at substantial risk of dying when powering up or cycling? Have you ever heard of common consumer grade computer equipment regularly dying from it? No? Not me either. It's really common to find thyristors in devices to limit in-rush current, even in cheap electronics. You can bet that PSUs in data center equipment use them.
    - emodendroket 5y ago
      > Have you ever heard of common consumer grade computer equipment regularly dying from powering up? Yes, absolutely.
      - daneel_w 5y ago
        You're romanticizing here. Electronics eventually dying is not the same as being at substantial risk from dying due to power-cycling itself.
iJohnDoe 5y ago
Not piling on AWS. These things happen and I’m sure everyone involved is working to improve things. Yes, everyone should deploy in multiple availability zones. My 2 cents. Outages happen. Network glitches happen. Bad configs and bad updates happen. However, power issues should not really happen. One of the primary cost saving areas of going to the cloud is not having to do on-prem power, such as UPS, generators, maintenance etc. Not having to do on-prem cooling is another thing. These should be solved from the customer’s perspective when going into a professional data center and are the things you don’t want to worry about anymore.
- kazen44 5y ago
  Getting power right in a DC takes a ton of preventative maintenance. (doing black building tests, make sure there are working load tests which occur frequently, managing UPS battery health etc).
  - spenczar5 5y ago
    Yes, it's hard work... but it's work AWS should be great at. Like, really great at. I used to work at AWS. The most worked-up I ever saw Charlie Bell (the de facto head of AWS engineering) was in the weekly operations meeting, going over a postmortem which described a near-miss as several generators failed to start properly in a power interruption. In this meeting - with nearly 1,000 participants! - Charlie got increasingly irate about the fact that this had happened. For the next few weeks, details of some sort of flushing subsystem for backup generators was an every-time topic. Sadly, Charlie left AWS and now works at Microsoft. I can't help but wonder what that operations meeting is like now without him.
  - Justin_K 5y ago
    This is what we've been paying for and they've failed us this time.
- ksec 5y ago
  Yes. I expect software and network glitches because they are complicated and error prone. But I fully expect hardware and power redundancy. Even if they were down it shouldn't "kill" them. At least not on AWS, the biggest and arguably best DC running in the world. But it did. This suggest to me DataCenter, as an infrastructure and building on itself still have plenty of room for improvements.
gundmc 5y ago
Rare to see power issues at a modern data center cause downtime. All of those racks should have UPS and batteries to sustain during an outage until the automatic transfer switch can fail over to a redundant system or generator. Would be interested in reading more about what happened here.
- electroly 5y ago
  In previous postmortems[0] they've mentioned their UPSes. They definitely have them. They don't seem to write a lot of postmortems, though. I'm not sure whether we should expect one for this event. [0] https://aws.amazon.com/premiumsupport/technology/pes/ https://aws.amazon.com/premiumsupport/technology/pes/
- deleted 5y ago
  [deleted]
zkirill 5y ago
Have there been any incidents that affected more than one AZ? AWS RDS in multi-AZ deployment gives you two availability zones. Aurora gives you three. What kind of scenario would be used to justify three AZ’s for the purposes of high availability?
- Jweb_Guru 5y ago
  Aurora isn't really designed to handle the total failure of two availability zones (not short term, anyway). It's designed to handle one AZ + one additional node failure (which is reasonably likely to happen on large instances due to data from a single database being striped across up to 12,800 machines per AZ). Due to how quorum systems work the "simplest" way they decided to handle that was six replicas per 10 GiB segment across 3 AZs, with three of the replicas being log-only and three designated as full replicas (which is the lowest number of full replicas you can have given their failure model, and hitting that lower bound does require you to deploy across 3 AZs). If any three of the nodes are dead (log-only or otherwise), write traffic is stopped until they can bring up a fourth node, though there is some support for backup 3-of-4 quorums for very long-term AZ outages.
  - zkirill 5y ago
    Amazingly insightful answer. Thank you very much!
- wmf 5y ago
  Modern Paxos/Raft HA requires an odd number of nodes so that's how you end up with three AZs.
testemailfordg2 5y ago
Not an expert, but a more frequent and controlled power cycle of servers with HDD can make sure not a lot of servers go down in one go, when events like these happen. Also, gives you chance to identify patterns...
joelbondurant0 5y ago

129 points by bratao 5y ago

1cvmask 5y ago

A lot more companies will go to a multi-cloud active active architecture with maybe even bare metal redundancies.

psanford 5y ago
I would stay away from any company that thinks "we need to go multicloud" as a response to this outage. This affected a single az in a single region. If it caused you downtime or a partial outage, it means you are not fully resilient to single az failures. The correct thing is to fix your application to handle that. If you can't handle a single az failure there is no way you are going to handle failing over across different cloud providers correctly.
- CubsFan1060 5y ago
  There will, however, be a lot of executives _talking_ about going multi cloud.
- luhn 5y ago
  To be fair though, from what I heard AZ outage caused an EC2 API brownout, so people couldn't launch new instances in the other AZs. That put a wrench in a lot of multi-AZ architectures. Not advocating for multi-cloud though...
- dragonwriter 5y ago
  > If it caused you downtime or a partial outage, it means you are not fully resilient to single az failures. Given the number of AWS global services that have dependencies on infra in US-EAST-1 (and, from the impacts of this and other past outages, seen vulnerable to single-AZ failures in US-EAST-1) that's...less avoidable for certain regions/AZs than one might naively expect. Most clouds seem to have at least some degree of this kind of vulnerability.
- nonane 5y ago
  > This affected a single az in a single region. If it caused you downtime or a partial outage, it means you are not fully resilient to single az failures. This is not true. Amazon is not being upfront about what happened here. It was simply not a single AZ failure. Our us-east-1 ELB load balancers were hosed and were unable to direct traffic to other AZs - they simply stopped working an were dropping traffic. We tried creating load balancers in different AZs and that didn't work either. How can you be resilient to single AZ failures if load balancers stop working region wide during a single AZ outage?
  - acdha 5y ago
    Did your TAM go into any details on that? Over a couple hundred load-balancers, the only issue we had was taking longer to register new instances and that affected only a couple of them. Running services weren't interrupted, latency remained, etc. which is what I'd expect for a single AZ failure.
deleted 5y ago
[deleted]
tyingq 5y ago
Multi-cloud is odd to me, unless you're a company selling a service to cloud customers. By definition, you would have to either go lowest-common-denominator, or build complicated facades in front of like services. If you're going lowest-common-denominator, then multi-old-school-hosting would be far cheaper.
- isbvhodnvemrwvn 5y ago
  You'd have to go worse than lowest common denominator. To have active-active replication you need to take the cost of latency between the different DCs of the various providers, and this thing will kill performance.
ransom1538 5y ago
me: "Yeah! AWS went down a few times, I think we should pay double!" vp: no.
- emodendroket 5y ago
  More than double realistically. But you could achieve most of the benefit at much lower cost by going multi-AZ.
- karmasimida 5y ago
  Infra is pure cost center. A few hours downtime is not going to justify double cost, and worse whose benefits can only be demonstrated during those few hours. It will get shutdown immediately. And what is more important, corporate world really don't care that much about downtime, not as much as they care about who to assign the blame, in this case AWS is a perfect irresistible externality much like a natural disaster. Disclaimer: Ex-AWS employee
profmonocle 5y ago
Egress fees can make replicating databases, storage buckets, etc. between clouds very expensive. Multi-region is a much more affordable option. Multi-region outages aren't unheard of among the major cloud operators, but they're less common than single-AZ or single-region outages. IMO, most companies just aren't sensitive enough to downtime that multi-AZ + multi-region deployment within a single cloud provider isn't good enough.
beermonster 5y ago
As large cloud outages become more frequent and the impact greater each time, I feel it’s more likely people will reconsider moving workloads off-prem.

CoastalCoder 5y ago

Any idea why a power failure would cause (or reveal) hardware damage? Leading up to Y2K, I remember concerns about spinning hard disks not being able to start up again. And if the power is flaky with spikes and brown-outs, I understand that's a problem. But is either of those relevant to AWS?

kube-system 5y ago
If I were to make some WAGs about things that could generally be described this way: * increased failure rates of drives (or other hardware) due to thermal cycling * cache power failures leading to corrupted data * previously unknown failures in recovery processes (like firmware bugs that might be described as a "hardware problem") * cooling failures leading to hardware failure Most of these are pretty rare issues, but it becomes more probable to happen at least once when you're running freekin' us-east-1
SteveNuts 5y ago
In my experience it's common when there's a full power down of a large data center like this. In 2013 the company I was working at had an unplanned power outage which took out 200 spinning drives, and 18 servers. I think the motors in the drives are fine running at constant speed but don't have the juice to spin up from a dead stop. I remember IBM was renting private planes to get us replacement hardware from around the country (we were a very large customer).
- jbergens 5y ago
  A very long time ago I worked for a medium sized company and there was a power outage. The servers were in a room in our building but they had ups power. The problem was that the ups's were broken/malfunctioning and the servers needed to do a disk check after an unplanned reboot. It took hours before we could work again even though most drives survived. This was also before git so we could not reach the central version control system and could not commit any code. I think all code and workspaces were on servers that was not up yet.
- lamontcg 5y ago

daneel_w 5y ago

"As is often the case with a loss of power, there may be some hardware that is not recoverable..." No. Not even rarely. If they lost hardware because of this something much different than just loss of power happened on their servers' mains rails.

Johnny555 5y ago
If my home UPS failed, I'd be awfully surprised if my home fileserver that was plugged into it failed too. But if I lost power to thousands of servers, I'd expect some number of them to fail. I've even lost servers when losing power to a single rack.
jdsully 5y ago
Starting up is hard on electronics due to the inrush current. There may be marginal devices just barely operating that will not survive a reboot. Things like bad capacitors aren’t always detectable during steady state. At AWS scale it’s highly likely there are more than a few of these.
- daneel_w 5y ago
  Did you think this over before hitting the reply button? Can you imagine any server or networking brand surviving if its products were at substantial risk of dying when powering up or cycling? Have you ever heard of common consumer grade computer equipment regularly dying from it? No? Not me either. It's really common to find thyristors in devices to limit in-rush current, even in cheap electronics. You can bet that PSUs in data center equipment use them.
  - emodendroket 5y ago
    > Have you ever heard of common consumer grade computer equipment regularly dying from powering up? Yes, absolutely.
    - daneel_w 5y ago
      You're romanticizing here. Electronics eventually dying is not the same as being at substantial risk from dying due to power-cycling itself.

iJohnDoe 5y ago

Not piling on AWS. These things happen and I’m sure everyone involved is working to improve things. Yes, everyone should deploy in multiple availability zones. My 2 cents. Outages happen. Network glitches happen. Bad configs and bad updates happen. However, power issues should not really happen. One of the primary cost saving areas of going to the cloud is not having to do on-prem power, such as UPS, generators, maintenance etc. Not having to do on-prem cooling is another thing. These should be solved from the customer’s perspective when going into a professional data center and are the things you don’t want to worry about anymore.

kazen44 5y ago
Getting power right in a DC takes a ton of preventative maintenance. (doing black building tests, make sure there are working load tests which occur frequently, managing UPS battery health etc).
- spenczar5 5y ago
  Yes, it's hard work... but it's work AWS should be great at. Like, really great at. I used to work at AWS. The most worked-up I ever saw Charlie Bell (the de facto head of AWS engineering) was in the weekly operations meeting, going over a postmortem which described a near-miss as several generators failed to start properly in a power interruption. In this meeting - with nearly 1,000 participants! - Charlie got increasingly irate about the fact that this had happened. For the next few weeks, details of some sort of flushing subsystem for backup generators was an every-time topic. Sadly, Charlie left AWS and now works at Microsoft. I can't help but wonder what that operations meeting is like now without him.
- Justin_K 5y ago
  This is what we've been paying for and they've failed us this time.
ksec 5y ago
Yes. I expect software and network glitches because they are complicated and error prone. But I fully expect hardware and power redundancy. Even if they were down it shouldn't "kill" them. At least not on AWS, the biggest and arguably best DC running in the world. But it did. This suggest to me DataCenter, as an infrastructure and building on itself still have plenty of room for improvements.

gundmc 5y ago

Rare to see power issues at a modern data center cause downtime. All of those racks should have UPS and batteries to sustain during an outage until the automatic transfer switch can fail over to a redundant system or generator. Would be interested in reading more about what happened here.

electroly 5y ago
In previous postmortems[0] they've mentioned their UPSes. They definitely have them. They don't seem to write a lot of postmortems, though. I'm not sure whether we should expect one for this event. [0] https://aws.amazon.com/premiumsupport/technology/pes/ https://aws.amazon.com/premiumsupport/technology/pes/
deleted 5y ago
[deleted]

zkirill 5y ago

Have there been any incidents that affected more than one AZ? AWS RDS in multi-AZ deployment gives you two availability zones. Aurora gives you three. What kind of scenario would be used to justify three AZ’s for the purposes of high availability?

Jweb_Guru 5y ago
Aurora isn't really designed to handle the total failure of two availability zones (not short term, anyway). It's designed to handle one AZ + one additional node failure (which is reasonably likely to happen on large instances due to data from a single database being striped across up to 12,800 machines per AZ). Due to how quorum systems work the "simplest" way they decided to handle that was six replicas per 10 GiB segment across 3 AZs, with three of the replicas being log-only and three designated as full replicas (which is the lowest number of full replicas you can have given their failure model, and hitting that lower bound does require you to deploy across 3 AZs). If any three of the nodes are dead (log-only or otherwise), write traffic is stopped until they can bring up a fourth node, though there is some support for backup 3-of-4 quorums for very long-term AZ outages.
- zkirill 5y ago
  Amazingly insightful answer. Thank you very much!
wmf 5y ago
Modern Paxos/Raft HA requires an odd number of nodes so that's how you end up with three AZs.

testemailfordg2 5y ago

Not an expert, but a more frequent and controlled power cycle of servers with HDD can make sure not a lot of servers go down in one go, when events like these happen. Also, gives you chance to identify patterns...

joelbondurant0 5y ago