8 ms·

This seems to be getting slightly overblown in that thread. To be clear, this impacted one datacenter out of ten that make up one availability zone out of six i

This seems to be getting slightly overblown in that thread. To be clear, this impacted one datacenter out of ten that make up one availability zone out of six in AWS’s us-east-1 region. So we are talking 2-3% at most of that region’s capacity was impacted. I haven’t seen a report yet on exactly why their generator failed, but from what I’ve heard, the power failed, and the backup generator kicked in and ran fine for over an hour, but then it failed. This sort of thing sucks to deal with but it’s also inevitable. Of their hundreds of datacenters, a mechanical failure is going to happen occasionally no matter how good their maintenance plans are. So the key when using cloud services like AWS is to plan for the possibility of failure. EBS expects an annual failure rate of 0.1%. So one out of a thousand EBS volumes will fail in a given year. If you operate at the scale of thousands of servers in AWS, you see this sort of thing all the time. Luckily, EBS also makes it trivial to take volume snapshots which are stored in S3, which has much much higher reliability and durability. So if you have data in EBS that needs to be kept safe, take regular snapshots. Here’s a doc that explains how you can set up scheduled, auto-rotated snapshots: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/snapshot-lifecycle.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/snapshot...

staticvar 7y ago
Do you have a source detailing this issue in us-east-1? I don't see a recent post-event summary from AWS mentioning us-east-1 https://aws.amazon.com/premiumsupport/technology/pes/ https://aws.amazon.com/premiumsupport/technology/pes/
- j2bax 7y ago
  Here is what they said in my support panel. This event made for a fairly tense Saturday morning for my development team and I. [01:30 PM PDT] At 4:33 AM PDT one of ten data centers in one of the six Availability Zones in the US-EAST-1 Region saw a failure of utility power. Our backup generators came online immediately but began failing at around 6:00 AM PDT. This impacted 7.5% of EC2 instances and EBS volumes in the Availability Zone. Power was fully restored to the impacted data center at 7:45 AM PDT. By 10:45 AM PDT, all but 1% of instances had been recovered, and by 12:30 PM PDT only 0.5% of instances remained impaired. Since the beginning of the impact, we have been working to recover the remaining instances and volumes. A small number of remaining instances and volumes are hosted on hardware which was adversely affected by the loss of power. We continue to work to recover all affected instances and volumes and will be communicating to the remaining impacted customers via the Personal Health Dashboard. For immediate recovery, we recommend replacing any remaining affected instances or volumes if possible.
  - Terretta 7y ago
    Serious question, for market strategy understanding: HA would normally use two AZs, 21C architectures would use three AZs, and other patterns such as pilot light let you use additional AZs without a significant cost hit. Further, when you can make workload-handling instances so much smaller (even down into the T sizes, and with cattle patterns you can start to leverage spot pricing), each additional AZ you add to the 21C mix represents a smaller percentage of your capacity lost in an AZ outage. What factors have you choosing an automation powered CSP such as AWS but using only a single AZ out of a half dozen? If using a single AZ and tense during outages, why not use Hetzner, Softlayer (now IBM Cloud), etc.?
    - notyourday 7y ago
      If you are using anything other than VMs, the APIs are too different.
    - darkwater 7y ago
      Can you detail/link resources about "21C architectures" and "pilot light pattern"? Thanks!
      - jsjohnst 7y ago
        My guess on “21C architectures” is “21st century architectures”. I can’t seem to find anything on “pilot light pattern” in quick searches. Either way, GP is using obscure terminology at best.
        lt 7y ago
        https://d1.awsstatic.com/whitepapers/aws-disaster-recovery.pdf https://d1.awsstatic.com/whitepapers/aws-disaster-recovery.p... > The term pilot light is often used to describe a DR scenario in which a minimal version of an environment is always running in the cloud. The idea of the pilot light is an analogy that comes from the gas heater. In a gas heater, a small flame that’s always on can quickly ignite the entire furnace to heat up a house
vectorEQ 7y ago
agree. like you imply, too many people rely heavily on their providers for business critical things like backups and redundancy. while generally big providers to a good job at this (and aws certainly does a good job at this), it does not mean there is a guarantee of any kind failures won't ever occur. thus the need to heavily invest in failure resistant technologies upon this borrowed infrastructure is arguably more important than picking 'the best' provider for the job. I would like to note that the response time suggested by the tweet is a bit bad, 4 days to realise something and send out response / alerts to customers is a bit slow even for amazon. But then again, everything goes slower for bigger things, and amazon is quite big i'd say. not sure what the SLA response time to such an incident is, so it might be within agreed times...
- romaaeterna 7y ago
  But the reason I pay AWS is so that I don't have to hire a team to take care of backups and redundancy on my side. If they can't be relied on, a lot of the justification for their cost markup goes out the window.
  - cloakandswagger 7y ago
    It can be relied on, but it's up to you to configure it properly. AWS has no way of knowing how critical your application is and what level of redundancy it needs, and this has cost implications so they can't do it automatically.
    - romaaeterna 7y ago
      I'm pretty sure that RDS is EBS-backed.
      - acdha 7y ago
        … and RDS has a multi-AZ checkbox which does exactly what it claims. Anyone who used it did not have a problem with this outage.
  - 7y ago
amykhar 7y ago
We were hit by it over the weekend, and because of our failsafes, our customers didn't notice a thing. Stuff happens. It's our job to prepare for these things as well.
- blantonl 7y ago
  Exactly. The benefits of the cloud allow you almost unlimited computing resources spread across enormous swaths of geography, power, and internet access. Therefore, if you were impacted, its your fault, not AWS. Sorry.
  - nwsm 7y ago
    * If it's within your SLA
    - bdcravens 7y ago
      I don’t understand this statement. If your application is developed with fault tolerance, this is independent of any SLA.
- victor106 7y ago
  Curious to know what kind of failsafes you have in place so your customers were not impacted?
  - smn1234 7y ago
    probably multi-az, multi-tier, stateless apps https://12factor.net/ https://12factor.net/ can help guide
  - cloakandswagger 7y ago
    I'm not the original commenter but for an EC2 hosted application, any architecture that uses a VPC with proper cross-AZ subnetting and a load balancer would not have been impacted. EBS snapshots would minimize any data loss.
koolba 7y ago
EBS Snapshots don't just protect you in case of a failure to recover to something, they actually reduce the chance of failure[1]. Assuming that the S3 storage has no data loss[2], you lower the data loss possibility for your EBS volume to only those bits that have been changed since the last snapshot. [1]: https://stackoverflow.com/questions/13576363/does-taking-a-snapshot-of-an-ebs-volume-increase-reliability https://stackoverflow.com/questions/13576363/does-taking-a-s... [2]: Yes it's a big if but S3 for durability against data loss is like the US Treasury for risk free returns on T-Bills.
- Narkov 7y ago
  I actually posted that Stack Overflow question and I don't think any of the answers support my assumption and your statement that it reduces the chance of failure. The EBS volume will still fail but you can restore it from the higher reliability S3-backed EBS snapshot.
alienreborn 7y ago
Some AWS services can't span across multiple availability zones like EMR.
- skywhopper 7y ago
  Yes, and also for many managed services (like Kinesis or Lambda) you don't have any control over which AZs are in use, and so you have no choice but to wait for the AWS teams to address any problems caused on their own backends. Regardless, it's important to be aware of the risks of whatever tool you're using. It's unrealistic to expect any provider to be able to avoid failure entirely. You have to be aware of possible failure scenarios and have your own plan to address them.
- eropple 7y ago
  EMR seems kind of legacy from AWS's perspective these days. Probably very profitable, but they're seeming to try to push you towards things like Glue and Athena.
- bshacklett 7y ago
  Even in these cases, however, there are usually architectural methods for mitigating this kind of failure. In the case of EMR, using transient clusters with state stored in durable file stores such as EMRFS and vanilla S3 is one such option. It is inevitable that systems will fail. The best the industry can do is work to reduce the number of failures and understand failure modes well so that they can be planned for. AWS does a very good job of this in my experience. Regardless of whether applications are hosted in the cloud, on premises, co-located or in some hybrid configuration, it's important to design for that inevitable failure and keep business decision makers in the loop while doing so. Understanding requirements around RPO and RTO are extremely important in developing an architecture which meets the needs of the business, yet is still cost effective.
philwelch 7y ago
When people talk about data backups, they always say you should rehearse restoring your backups just so you're confident that the backups are complete. If you don't do that, you lose your data and your recovery fails because you never tested it. Likewise, I guess it's easy to speculate from the peanut gallery, but I wouldn't be surprised if the backup generators just hadn't been sufficiently tested and maintained because, well, they're backup generators.
skywhopper 7y ago
More info: Sounds like the backup generators operated fine, but only have about an hour of fuel in their individual storage tanks. There's an automated fuel delivery system to keep them topped up which is the piece that failed, meaning the individual generators didn't get their tanks refilled and eventually ran out of fuel. This was compounded by the operations staff being busy with trying to restore the main power feed and somehow (not explained) didn't get a timely notification that the fuel delivery system itself had failed. AWS says they plan to install a backup fuel delivery system at this particular datacenter (it wasn't detailed whether this SPOF is common at other datacenters or if this was an outlier) and that they have already updated their notifications to be more aggressive.
hamiltont 7y ago
Don't disagree with your point that this is overblown, but here's an important related point "Your nines are not my nines" - https://rachelbythebay.com/w/2019/07/15/giant/ https://rachelbythebay.com/w/2019/07/15/giant/
- mk89 7y ago
  If I recall correctly this is nicely covered in the book "Release it!" which BTW I recommend.
- dymk 7y ago
  But for a large number of 9s of people, AWS's 9s are their 9s.
- joehandzik 7y ago
  Reminds me of the hover text on this xkcd: https://xkcd.com/325/ https://xkcd.com/325/
kerkeslager 7y ago
> This seems to be getting slightly overblown in that thread. To be clear, this impacted one datacenter out of ten that make up one availability zone out of six in AWS’s us-east-1 region. So we are talking 2-3% at most of that region’s capacity was impacted. For how many users was this 100% of their business? Single-digit outage percentages for cloud services like AWS look like no big deal from the big perspective, which means that they often aren't a high priority. But when you're one of the customers in the 2-3%, it is a big deal, and you want it to be a high priority. Small hosting providers may have only 3 9s of availability, but when your site goes down their world stops until it's fixed. I've seen reports of 5 9s of availability from Amazon, but when your site goes down, their alerts are still all green and they'll call you back at their leisure.
- gnur 7y ago
  Honestly, if you are running on a single AZ in aws you are setting yourself up for failure..
- acdha 7y ago
  > For how many users was this 100% of their business? … but they still chose to ignore all of the prominent warnings and architectural guidance, not to mention avoiding use of the services which have HA built-in. I mean, I'm sympathetic to anyone who had a bad day with a forced learning experience but it's not like this is some dark secret.
  - kerkeslager 7y ago
    > but they still chose to ignore all of the prominent warnings and architectural guidance So that's it, blame the user and caveat emptor? > not to mention avoiding use of the services which have HA built-in. Many (all?) of these services tightly couple you to Amazon, so avoiding them is a very reasonable decision.
    - acdha 7y ago
      > So that's it, blame the user and caveat emptor? If I sell you a loaf of bread and you complain that it's not a sandwich, is it anything else? > Many (all?) of these services tightly couple you to Amazon, so avoiding them is a very reasonable decision. That's just a cop-out: checking the “multi-AZ” box in RDS completely avoided this problem with zero lock-in. If you're deploying containers, you have multiple options which are portable and avoid this completely. If you're deploying EC2 instances, again you have options with very limited lock-in (e.g. auto-scaling with multiple AZs). More importantly, that's also a business decision: if you're that worried about lock-in you are accepting responsibility to operate the alternatives. For example, following industry-standard practice might suggest that you run everything in Kubernetes in multiple AZs, regions, or providers but it would never support running everything in a single AZ.

staticvar 7y ago

Do you have a source detailing this issue in us-east-1? I don't see a recent post-event summary from AWS mentioning us-east-1 https://aws.amazon.com/premiumsupport/technology/pes/ https://aws.amazon.com/premiumsupport/technology/pes/

j2bax 7y ago
Here is what they said in my support panel. This event made for a fairly tense Saturday morning for my development team and I. [01:30 PM PDT] At 4:33 AM PDT one of ten data centers in one of the six Availability Zones in the US-EAST-1 Region saw a failure of utility power. Our backup generators came online immediately but began failing at around 6:00 AM PDT. This impacted 7.5% of EC2 instances and EBS volumes in the Availability Zone. Power was fully restored to the impacted data center at 7:45 AM PDT. By 10:45 AM PDT, all but 1% of instances had been recovered, and by 12:30 PM PDT only 0.5% of instances remained impaired. Since the beginning of the impact, we have been working to recover the remaining instances and volumes. A small number of remaining instances and volumes are hosted on hardware which was adversely affected by the loss of power. We continue to work to recover all affected instances and volumes and will be communicating to the remaining impacted customers via the Personal Health Dashboard. For immediate recovery, we recommend replacing any remaining affected instances or volumes if possible.
- Terretta 7y ago
  Serious question, for market strategy understanding: HA would normally use two AZs, 21C architectures would use three AZs, and other patterns such as pilot light let you use additional AZs without a significant cost hit. Further, when you can make workload-handling instances so much smaller (even down into the T sizes, and with cattle patterns you can start to leverage spot pricing), each additional AZ you add to the 21C mix represents a smaller percentage of your capacity lost in an AZ outage. What factors have you choosing an automation powered CSP such as AWS but using only a single AZ out of a half dozen? If using a single AZ and tense during outages, why not use Hetzner, Softlayer (now IBM Cloud), etc.?
  - notyourday 7y ago
    If you are using anything other than VMs, the APIs are too different.
  - darkwater 7y ago
    Can you detail/link resources about "21C architectures" and "pilot light pattern"? Thanks!
    - jsjohnst 7y ago
      My guess on “21C architectures” is “21st century architectures”. I can’t seem to find anything on “pilot light pattern” in quick searches. Either way, GP is using obscure terminology at best.
      - lt 7y ago
        https://d1.awsstatic.com/whitepapers/aws-disaster-recovery.pdf https://d1.awsstatic.com/whitepapers/aws-disaster-recovery.p... > The term pilot light is often used to describe a DR scenario in which a minimal version of an environment is always running in the cloud. The idea of the pilot light is an analogy that comes from the gas heater. In a gas heater, a small flame that’s always on can quickly ignite the entire furnace to heat up a house

vectorEQ 7y ago

agree. like you imply, too many people rely heavily on their providers for business critical things like backups and redundancy. while generally big providers to a good job at this (and aws certainly does a good job at this), it does not mean there is a guarantee of any kind failures won't ever occur. thus the need to heavily invest in failure resistant technologies upon this borrowed infrastructure is arguably more important than picking 'the best' provider for the job. I would like to note that the response time suggested by the tweet is a bit bad, 4 days to realise something and send out response / alerts to customers is a bit slow even for amazon. But then again, everything goes slower for bigger things, and amazon is quite big i'd say. not sure what the SLA response time to such an incident is, so it might be within agreed times...

romaaeterna 7y ago
But the reason I pay AWS is so that I don't have to hire a team to take care of backups and redundancy on my side. If they can't be relied on, a lot of the justification for their cost markup goes out the window.
- cloakandswagger 7y ago
  It can be relied on, but it's up to you to configure it properly. AWS has no way of knowing how critical your application is and what level of redundancy it needs, and this has cost implications so they can't do it automatically.
  - romaaeterna 7y ago
    I'm pretty sure that RDS is EBS-backed.
    - acdha 7y ago
      … and RDS has a multi-AZ checkbox which does exactly what it claims. Anyone who used it did not have a problem with this outage.
- 7y ago

amykhar 7y ago

We were hit by it over the weekend, and because of our failsafes, our customers didn't notice a thing. Stuff happens. It's our job to prepare for these things as well.

blantonl 7y ago
Exactly. The benefits of the cloud allow you almost unlimited computing resources spread across enormous swaths of geography, power, and internet access. Therefore, if you were impacted, its your fault, not AWS. Sorry.
- nwsm 7y ago
  * If it's within your SLA
  - bdcravens 7y ago
    I don’t understand this statement. If your application is developed with fault tolerance, this is independent of any SLA.
victor106 7y ago
Curious to know what kind of failsafes you have in place so your customers were not impacted?
- smn1234 7y ago
  probably multi-az, multi-tier, stateless apps https://12factor.net/ https://12factor.net/ can help guide
- cloakandswagger 7y ago
  I'm not the original commenter but for an EC2 hosted application, any architecture that uses a VPC with proper cross-AZ subnetting and a load balancer would not have been impacted. EBS snapshots would minimize any data loss.

koolba 7y ago

EBS Snapshots don't just protect you in case of a failure to recover to something, they actually reduce the chance of failure[1]. Assuming that the S3 storage has no data loss[2], you lower the data loss possibility for your EBS volume to only those bits that have been changed since the last snapshot. [1]: https://stackoverflow.com/questions/13576363/does-taking-a-snapshot-of-an-ebs-volume-increase-reliability https://stackoverflow.com/questions/13576363/does-taking-a-s... [2]: Yes it's a big if but S3 for durability against data loss is like the US Treasury for risk free returns on T-Bills.

Narkov 7y ago
I actually posted that Stack Overflow question and I don't think any of the answers support my assumption and your statement that it reduces the chance of failure. The EBS volume will still fail but you can restore it from the higher reliability S3-backed EBS snapshot.

alienreborn 7y ago

Some AWS services can't span across multiple availability zones like EMR.

skywhopper 7y ago
Yes, and also for many managed services (like Kinesis or Lambda) you don't have any control over which AZs are in use, and so you have no choice but to wait for the AWS teams to address any problems caused on their own backends. Regardless, it's important to be aware of the risks of whatever tool you're using. It's unrealistic to expect any provider to be able to avoid failure entirely. You have to be aware of possible failure scenarios and have your own plan to address them.
eropple 7y ago
EMR seems kind of legacy from AWS's perspective these days. Probably very profitable, but they're seeming to try to push you towards things like Glue and Athena.
bshacklett 7y ago
Even in these cases, however, there are usually architectural methods for mitigating this kind of failure. In the case of EMR, using transient clusters with state stored in durable file stores such as EMRFS and vanilla S3 is one such option. It is inevitable that systems will fail. The best the industry can do is work to reduce the number of failures and understand failure modes well so that they can be planned for. AWS does a very good job of this in my experience. Regardless of whether applications are hosted in the cloud, on premises, co-located or in some hybrid configuration, it's important to design for that inevitable failure and keep business decision makers in the loop while doing so. Understanding requirements around RPO and RTO are extremely important in developing an architecture which meets the needs of the business, yet is still cost effective.

philwelch 7y ago

When people talk about data backups, they always say you should rehearse restoring your backups just so you're confident that the backups are complete. If you don't do that, you lose your data and your recovery fails because you never tested it. Likewise, I guess it's easy to speculate from the peanut gallery, but I wouldn't be surprised if the backup generators just hadn't been sufficiently tested and maintained because, well, they're backup generators.

skywhopper 7y ago

More info: Sounds like the backup generators operated fine, but only have about an hour of fuel in their individual storage tanks. There's an automated fuel delivery system to keep them topped up which is the piece that failed, meaning the individual generators didn't get their tanks refilled and eventually ran out of fuel. This was compounded by the operations staff being busy with trying to restore the main power feed and somehow (not explained) didn't get a timely notification that the fuel delivery system itself had failed. AWS says they plan to install a backup fuel delivery system at this particular datacenter (it wasn't detailed whether this SPOF is common at other datacenters or if this was an outlier) and that they have already updated their notifications to be more aggressive.

hamiltont 7y ago

Don't disagree with your point that this is overblown, but here's an important related point "Your nines are not my nines" - https://rachelbythebay.com/w/2019/07/15/giant/ https://rachelbythebay.com/w/2019/07/15/giant/

mk89 7y ago
If I recall correctly this is nicely covered in the book "Release it!" which BTW I recommend.
dymk 7y ago
But for a large number of 9s of people, AWS's 9s are their 9s.
joehandzik 7y ago
Reminds me of the hover text on this xkcd: https://xkcd.com/325/ https://xkcd.com/325/

kerkeslager 7y ago

> This seems to be getting slightly overblown in that thread. To be clear, this impacted one datacenter out of ten that make up one availability zone out of six in AWS’s us-east-1 region. So we are talking 2-3% at most of that region’s capacity was impacted. For how many users was this 100% of their business? Single-digit outage percentages for cloud services like AWS look like no big deal from the big perspective, which means that they often aren't a high priority. But when you're one of the customers in the 2-3%, it is a big deal, and you want it to be a high priority. Small hosting providers may have only 3 9s of availability, but when your site goes down their world stops until it's fixed. I've seen reports of 5 9s of availability from Amazon, but when your site goes down, their alerts are still all green and they'll call you back at their leisure.

gnur 7y ago
Honestly, if you are running on a single AZ in aws you are setting yourself up for failure..
acdha 7y ago
> For how many users was this 100% of their business? … but they still chose to ignore all of the prominent warnings and architectural guidance, not to mention avoiding use of the services which have HA built-in. I mean, I'm sympathetic to anyone who had a bad day with a forced learning experience but it's not like this is some dark secret.
- kerkeslager 7y ago
  > but they still chose to ignore all of the prominent warnings and architectural guidance So that's it, blame the user and caveat emptor? > not to mention avoiding use of the services which have HA built-in. Many (all?) of these services tightly couple you to Amazon, so avoiding them is a very reasonable decision.
  - acdha 7y ago
    > So that's it, blame the user and caveat emptor? If I sell you a loaf of bread and you complain that it's not a sandwich, is it anything else? > Many (all?) of these services tightly couple you to Amazon, so avoiding them is a very reasonable decision. That's just a cop-out: checking the “multi-AZ” box in RDS completely avoided this problem with zero lock-in. If you're deploying containers, you have multiple options which are portable and avoid this completely. If you're deploying EC2 instances, again you have options with very limited lock-in (e.g. auto-scaling with multiple AZs). More importantly, that's also a business decision: if you're that worried about lock-in you are accepting responsibility to operate the alternatives. For example, following industry-standard practice might suggest that you run everything in Kubernetes in multiple AZs, regions, or providers but it would never support running everything in a single AZ.