3 ms·

Something I'm not getting there. OK, one zone went down in AWS. But there supposed to be multiple ones? And if somebody knows how to use AWS so that one zone go

by smsm42 5y ago

Something I'm not getting there. OK, one zone went down in AWS. But there supposed to be multiple ones? And if somebody knows how to use AWS so that one zone going down wouldn't knock over the whole thing, it should be Amazon? I thought the whole point of this multi-zone cloud setup would be so that even if one zone goes down, it could survive and perform as usual - maybe a little slower for a bit while backups kick in and such, but after initial short period everything should be fine? Here we're talking about the whole system going down hard. Was the problem much wider than reported - i.e. affecting all (US) zones at least, or is it that Amazon doesn't know or doesn't care to use its own infrastructure to build a robust system?

DantesKite 5y ago
It’s beginning to sound like a cyberattack to me.
tekno45 5y ago
This zone hosts the infra to manage the rest of the zones. DNS management is also here and can easily cause a global failure.
matteotom 5y ago
This affected on region, us-east-1, which is the oldest and largest aws region, and also hosts many core aws services. Each region has several availability zones (AZs) that are basically whole datacenters. In theory AZs should be mostly isolated, but as we saw bugs happen. Reading between the lines of the status updates, it sounds like this either affected some core infra shared between AZs, or was from a change rolled out safely previously (maybe last night) and failed later (eg due to increased load in the morning). At the end of the day, building redundant systems is expensive (and may introduce whole new bugs), so they probably did the math and figured the risk of a whole region outage was less than the cost of building redundancy in some systems that are hard to make redundant.
- smsm42 5y ago
  The whole concept that there are "core AWS services" living in one single zone sounds like anti-thesis to everything AWS should be about. What's the use of having all this nice distributed setup - and paying for it! - if a single failure in a single zone takes everything down anyway? I mean sure, maybe they built it on a shoe-string budget years ago - but since then, they had years and billions, and still didn't bother to fix it?
tyingq 5y ago
There's a lot of pitfalls. You have to diligent about stuff like making sure you're using region-specific endpoints like sts.us-west-2.amazonaws.com instead of sts.amazonaws.com. And some services, like Route53 and Cloudfront have built-in dependencies on us-east-1. There's also internal dependencies that are sometimes not obvious until an outage occurs. Though, as you say, none of that's a great excuse for Amazon itself. I know Alexa devices, Ring Devices, Prime Video, imdb.com, etc, also all had issues in the first hours of this outage.

by smsm42 5y ago

DantesKite 5y ago

It’s beginning to sound like a cyberattack to me.

tekno45 5y ago

This zone hosts the infra to manage the rest of the zones. DNS management is also here and can easily cause a global failure.

matteotom 5y ago

This affected on region, us-east-1, which is the oldest and largest aws region, and also hosts many core aws services. Each region has several availability zones (AZs) that are basically whole datacenters. In theory AZs should be mostly isolated, but as we saw bugs happen. Reading between the lines of the status updates, it sounds like this either affected some core infra shared between AZs, or was from a change rolled out safely previously (maybe last night) and failed later (eg due to increased load in the morning). At the end of the day, building redundant systems is expensive (and may introduce whole new bugs), so they probably did the math and figured the risk of a whole region outage was less than the cost of building redundancy in some systems that are hard to make redundant.

smsm42 5y ago
The whole concept that there are "core AWS services" living in one single zone sounds like anti-thesis to everything AWS should be about. What's the use of having all this nice distributed setup - and paying for it! - if a single failure in a single zone takes everything down anyway? I mean sure, maybe they built it on a shoe-string budget years ago - but since then, they had years and billions, and still didn't bother to fix it?

tyingq 5y ago

There's a lot of pitfalls. You have to diligent about stuff like making sure you're using region-specific endpoints like sts.us-west-2.amazonaws.com instead of sts.amazonaws.com. And some services, like Route53 and Cloudfront have built-in dependencies on us-east-1. There's also internal dependencies that are sometimes not obvious until an outage occurs. Though, as you say, none of that's a great excuse for Amazon itself. I know Alexa devices, Ring Devices, Prime Video, imdb.com, etc, also all had issues in the first hours of this outage.