5 ms·

Amazon makes it pretty clear that Availability Zones within the same region can fail simultaneously. In fact, a Region being down is defined as multiple AZs wi

by akashs 15y ago

Amazon makes it pretty clear that Availability Zones within the same region can fail simultaneously. In fact, a Region being down is defined as multiple AZs within that zone being down according to the SLA. And since that 99.95% promise applies to Regions and not AZs, multiple AZs within the same region being down will be fairly common. Edit: One more point. In the SLA, you'll find the following: “Region Unavailable” and “Region Unavailability” means that more than one Availability Zone in which you are running an instance, within the same Region, is “Unavailable” to you. What it implies is that if you do not spread across multiple Availability Zones, you will then have less than 99.95% uptime. So spreading across AZs should still reduce your downtime, just not beyond that 99.95% http://aws.amazon.com/ec2-sla/ http://aws.amazon.com/ec2-sla/

jpdoctor 15y ago
The SLA uses great legal weasel words: "AWS will use commercially reasonable efforts to make Amazon EC2 available" So anything that is beyond commercially reasonable is outside the SLA. In truth, as with all businesses, the reputation for uptime weighs more heavily than the written contract. It will be interesting to see how the AWS people attempt to make amends.
- tomkarlo 15y ago
  "Commercially reasonable" is a standard legal term used to define efforts short of "best efforts". It allows for the party also look out for its own commercial interest in a way that's consistent with industry practice. So, for example, if Amazon had to choose between fulfilling the SLA and keeping it's own retail site up, it could be held liable under a "best efforts" standard but not under a "commercially reasonable" standard. It's kind of unfair to describe these as "weasel words" when it's unlikely that any decent lawyer would let them sign up to something that exposes them to more liability than this. Customers who are using any cloud service provider have to expect reasonable steps to maintain availability, not an absolute promise.
justinsb 15y ago
I have to disagree with you. The SLA is just a legal agreement that really serves to limit AWS's liability. Here's what the main EC2 page says: "Availability Zones are distinct locations that are engineered to be insulated from failures in other Availability Zones and provide inexpensive, low latency network connectivity to other Availability Zones in the same Region. By launching instances in separate Availability Zones, you can protect your applications from failure of a single location." http://aws.amazon.com/ec2/ http://aws.amazon.com/ec2/ That's the spec that everyone was building to, but that isn't what is happening. Of course you're right, multiple AZs can fail at the same time, but I read the above as saying that they should fail independently/coincidentally (until the entire Region fails).
- jeresig 15y ago
  "Of course you're right, multiple AZs can fail at the same time, but I read the above as saying that they should fail independently/coincidentally." As far as I know we've heard nothing to the contrary from Amazon - it's totally possible that multiple AZs happened to fail independently/coincidentally. Perhaps it was simultaneous equipment failure? Or maybe one AZ failed and a sufficient number of people attempted to "fail over" to another AZ causing a chain reaction of failure?
  - justinsb 15y ago
    It is possible. I think it's exceptionally unlikely. The one bit of information we have suggests that the root cause was a networking issue, which suggests SPOF.
    - leoc 15y ago
      If I had to, I'd guess that AWS's messaging/monitoring/control infrastructure is likely to be the SPOF, as in the 2008 outage: http://status.aws.amazon.com/s3-20080720.html http://status.aws.amazon.com/s3-20080720.html It's an obvious weak spot in the independence/isolation of AWS' nodes, and it would seem to be the one most likely to cause failure to reach across more than one AZ. (Apart from a stampede from one, affected AZ to others perhaps.)

by akashs 15y ago

jpdoctor 15y ago

The SLA uses great legal weasel words: "AWS will use commercially reasonable efforts to make Amazon EC2 available" So anything that is beyond commercially reasonable is outside the SLA. In truth, as with all businesses, the reputation for uptime weighs more heavily than the written contract. It will be interesting to see how the AWS people attempt to make amends.

tomkarlo 15y ago
"Commercially reasonable" is a standard legal term used to define efforts short of "best efforts". It allows for the party also look out for its own commercial interest in a way that's consistent with industry practice. So, for example, if Amazon had to choose between fulfilling the SLA and keeping it's own retail site up, it could be held liable under a "best efforts" standard but not under a "commercially reasonable" standard. It's kind of unfair to describe these as "weasel words" when it's unlikely that any decent lawyer would let them sign up to something that exposes them to more liability than this. Customers who are using any cloud service provider have to expect reasonable steps to maintain availability, not an absolute promise.

justinsb 15y ago

I have to disagree with you. The SLA is just a legal agreement that really serves to limit AWS's liability. Here's what the main EC2 page says: "Availability Zones are distinct locations that are engineered to be insulated from failures in other Availability Zones and provide inexpensive, low latency network connectivity to other Availability Zones in the same Region. By launching instances in separate Availability Zones, you can protect your applications from failure of a single location." http://aws.amazon.com/ec2/ http://aws.amazon.com/ec2/ That's the spec that everyone was building to, but that isn't what is happening. Of course you're right, multiple AZs can fail at the same time, but I read the above as saying that they should fail independently/coincidentally (until the entire Region fails).

jeresig 15y ago
"Of course you're right, multiple AZs can fail at the same time, but I read the above as saying that they should fail independently/coincidentally." As far as I know we've heard nothing to the contrary from Amazon - it's totally possible that multiple AZs happened to fail independently/coincidentally. Perhaps it was simultaneous equipment failure? Or maybe one AZ failed and a sufficient number of people attempted to "fail over" to another AZ causing a chain reaction of failure?
- justinsb 15y ago
  It is possible. I think it's exceptionally unlikely. The one bit of information we have suggests that the root cause was a networking issue, which suggests SPOF.
  - leoc 15y ago
    If I had to, I'd guess that AWS's messaging/monitoring/control infrastructure is likely to be the SPOF, as in the 2008 outage: http://status.aws.amazon.com/s3-20080720.html http://status.aws.amazon.com/s3-20080720.html It's an obvious weak spot in the independence/isolation of AWS' nodes, and it would seem to be the one most likely to cause failure to reach across more than one AZ. (Apart from a stampede from one, affected AZ to others perhaps.)