4 ms·

Do you have a source detailing this issue in us-east-1? I don't see a recent post-event summary from AWS mentioning us-east-1 https://aws.amazon.com/premiumsupp

Do you have a source detailing this issue in us-east-1? I don't see a recent post-event summary from AWS mentioning us-east-1 https://aws.amazon.com/premiumsupport/technology/pes/ https://aws.amazon.com/premiumsupport/technology/pes/

j2bax 7y ago
Here is what they said in my support panel. This event made for a fairly tense Saturday morning for my development team and I. [01:30 PM PDT] At 4:33 AM PDT one of ten data centers in one of the six Availability Zones in the US-EAST-1 Region saw a failure of utility power. Our backup generators came online immediately but began failing at around 6:00 AM PDT. This impacted 7.5% of EC2 instances and EBS volumes in the Availability Zone. Power was fully restored to the impacted data center at 7:45 AM PDT. By 10:45 AM PDT, all but 1% of instances had been recovered, and by 12:30 PM PDT only 0.5% of instances remained impaired. Since the beginning of the impact, we have been working to recover the remaining instances and volumes. A small number of remaining instances and volumes are hosted on hardware which was adversely affected by the loss of power. We continue to work to recover all affected instances and volumes and will be communicating to the remaining impacted customers via the Personal Health Dashboard. For immediate recovery, we recommend replacing any remaining affected instances or volumes if possible.
- Terretta 7y ago
  Serious question, for market strategy understanding: HA would normally use two AZs, 21C architectures would use three AZs, and other patterns such as pilot light let you use additional AZs without a significant cost hit. Further, when you can make workload-handling instances so much smaller (even down into the T sizes, and with cattle patterns you can start to leverage spot pricing), each additional AZ you add to the 21C mix represents a smaller percentage of your capacity lost in an AZ outage. What factors have you choosing an automation powered CSP such as AWS but using only a single AZ out of a half dozen? If using a single AZ and tense during outages, why not use Hetzner, Softlayer (now IBM Cloud), etc.?
  - notyourday 7y ago
    If you are using anything other than VMs, the APIs are too different.
  - darkwater 7y ago
    Can you detail/link resources about "21C architectures" and "pilot light pattern"? Thanks!
    - jsjohnst 7y ago
      My guess on “21C architectures” is “21st century architectures”. I can’t seem to find anything on “pilot light pattern” in quick searches. Either way, GP is using obscure terminology at best.
      - lt 7y ago
        https://d1.awsstatic.com/whitepapers/aws-disaster-recovery.pdf https://d1.awsstatic.com/whitepapers/aws-disaster-recovery.p... > The term pilot light is often used to describe a DR scenario in which a minimal version of an environment is always running in the cloud. The idea of the pilot light is an analogy that comes from the gas heater. In a gas heater, a small flame that’s always on can quickly ignite the entire furnace to heat up a house
        jsjohnst 7y ago
        > In a gas heater, a small flame that’s always on can quickly ignite the entire furnace to heat up a house Sure, I’m being nitpicky, but that’s a terrible explanation of the purpose of a pilot light in a legacy furnace.
    - deleted 7y ago
      [deleted]
  - fulafel 7y ago
    A lot of organisations are very very conservative and slow moving, and AWS seemed the most likely thing to still be around after/during the 15 year project to shift things out of private data centers.

j2bax 7y ago

Here is what they said in my support panel. This event made for a fairly tense Saturday morning for my development team and I. [01:30 PM PDT] At 4:33 AM PDT one of ten data centers in one of the six Availability Zones in the US-EAST-1 Region saw a failure of utility power. Our backup generators came online immediately but began failing at around 6:00 AM PDT. This impacted 7.5% of EC2 instances and EBS volumes in the Availability Zone. Power was fully restored to the impacted data center at 7:45 AM PDT. By 10:45 AM PDT, all but 1% of instances had been recovered, and by 12:30 PM PDT only 0.5% of instances remained impaired. Since the beginning of the impact, we have been working to recover the remaining instances and volumes. A small number of remaining instances and volumes are hosted on hardware which was adversely affected by the loss of power. We continue to work to recover all affected instances and volumes and will be communicating to the remaining impacted customers via the Personal Health Dashboard. For immediate recovery, we recommend replacing any remaining affected instances or volumes if possible.

Terretta 7y ago
Serious question, for market strategy understanding: HA would normally use two AZs, 21C architectures would use three AZs, and other patterns such as pilot light let you use additional AZs without a significant cost hit. Further, when you can make workload-handling instances so much smaller (even down into the T sizes, and with cattle patterns you can start to leverage spot pricing), each additional AZ you add to the 21C mix represents a smaller percentage of your capacity lost in an AZ outage. What factors have you choosing an automation powered CSP such as AWS but using only a single AZ out of a half dozen? If using a single AZ and tense during outages, why not use Hetzner, Softlayer (now IBM Cloud), etc.?
- notyourday 7y ago
  If you are using anything other than VMs, the APIs are too different.
- darkwater 7y ago
  Can you detail/link resources about "21C architectures" and "pilot light pattern"? Thanks!
  - jsjohnst 7y ago
    My guess on “21C architectures” is “21st century architectures”. I can’t seem to find anything on “pilot light pattern” in quick searches. Either way, GP is using obscure terminology at best.
    - lt 7y ago
      https://d1.awsstatic.com/whitepapers/aws-disaster-recovery.pdf https://d1.awsstatic.com/whitepapers/aws-disaster-recovery.p... > The term pilot light is often used to describe a DR scenario in which a minimal version of an environment is always running in the cloud. The idea of the pilot light is an analogy that comes from the gas heater. In a gas heater, a small flame that’s always on can quickly ignite the entire furnace to heat up a house
      - jsjohnst 7y ago
        > In a gas heater, a small flame that’s always on can quickly ignite the entire furnace to heat up a house Sure, I’m being nitpicky, but that’s a terrible explanation of the purpose of a pilot light in a legacy furnace.
  - deleted 7y ago
    [deleted]
- fulafel 7y ago
  A lot of organisations are very very conservative and slow moving, and AWS seemed the most likely thing to still be around after/during the 15 year project to shift things out of private data centers.