4 ms·

I have to disagree with you. The SLA is just a legal agreement that really serves to limit AWS's liability. Here's what the main EC2 page says: "Availability

I have to disagree with you. The SLA is just a legal agreement that really serves to limit AWS's liability. Here's what the main EC2 page says: "Availability Zones are distinct locations that are engineered to be insulated from failures in other Availability Zones and provide inexpensive, low latency network connectivity to other Availability Zones in the same Region. By launching instances in separate Availability Zones, you can protect your applications from failure of a single location." http://aws.amazon.com/ec2/ http://aws.amazon.com/ec2/ That's the spec that everyone was building to, but that isn't what is happening. Of course you're right, multiple AZs can fail at the same time, but I read the above as saying that they should fail independently/coincidentally (until the entire Region fails).

jeresig 16y ago
"Of course you're right, multiple AZs can fail at the same time, but I read the above as saying that they should fail independently/coincidentally." As far as I know we've heard nothing to the contrary from Amazon - it's totally possible that multiple AZs happened to fail independently/coincidentally. Perhaps it was simultaneous equipment failure? Or maybe one AZ failed and a sufficient number of people attempted to "fail over" to another AZ causing a chain reaction of failure?
- justinsb 16y ago
  It is possible. I think it's exceptionally unlikely. The one bit of information we have suggests that the root cause was a networking issue, which suggests SPOF.
  - leoc 16y ago
    If I had to, I'd guess that AWS's messaging/monitoring/control infrastructure is likely to be the SPOF, as in the 2008 outage: http://status.aws.amazon.com/s3-20080720.html http://status.aws.amazon.com/s3-20080720.html It's an obvious weak spot in the independence/isolation of AWS' nodes, and it would seem to be the one most likely to cause failure to reach across more than one AZ. (Apart from a stampede from one, affected AZ to others perhaps.)
bphogan 16y ago
We always, always use the SLA offered by a vendor as the basis for our information. We trust it more than any marketing page, sales pitch, tech support FAQ, or anything else. That's what they'll hide behind, so that's what I'll have in mind when I design my setup.
- justinsb 16y ago
  I think it's great to check the SLA. However, there's enough wiggle room in the AWS SLA that I think this outage could continue for the rest of the month, and Amazon would still not owe a penny. I don't even know that the SLA covers this outage, because network connectivity isn't affected. Even if Amazon breach their SLA, I think they only have to refund 10% of one month's bill per year - i.e. a 1% discount. I suspect they'd make a good profit even if they paid out a full 10% refund every month. Unless an SLA is accelerated - i.e. >100% refund - I don't think it's worth taking particularly seriously. Of course if an SLA only guarantees 95% uptime, that's probably a big hint to design for failure!
  - bphogan 16y ago
    Yeah but I don't care about getting my money back as much as I care about how much they claim to be down. It's like the hard disk maker that gives you a 1 year warranty vs a 5 year warranty... which one believes in their product more? :)
    - justinsb 16y ago
      It's a good analogy and I certainly accept your point. It could just be a marketing thing though: Suppose it's the same hard disk with a black sticker instead of a blue sticker. Drive with 1 yr warranty @ $100, 5 yr warranty @ $150, 20% additional failure rate over the extra 4 years, 50% redemption rate on failed drives. Cost per replaced drives = 20% * 50% * ($100 + $30 processing costs) = $13 = $37 profit. Totally fictitious numbers to try to prove my point, of course :-) But as the SLA becomes increasingly low in value, the signalling value decreases in my book. (Edit - fixed my math!)
    - 16y ago
pessimist 16y ago
I'm sorry, designing your service without taking in to account the SLA is just stupid. See how Netflix survived the failure for example. Now if you understand the SLA and still choose not to do cross-region deployments, then you've taken a cost/complexity vs uptime trade-off, which may well be right for you. quora.com probably is ok - who cares if its down for a day?

jeresig 16y ago

"Of course you're right, multiple AZs can fail at the same time, but I read the above as saying that they should fail independently/coincidentally." As far as I know we've heard nothing to the contrary from Amazon - it's totally possible that multiple AZs happened to fail independently/coincidentally. Perhaps it was simultaneous equipment failure? Or maybe one AZ failed and a sufficient number of people attempted to "fail over" to another AZ causing a chain reaction of failure?

justinsb 16y ago
It is possible. I think it's exceptionally unlikely. The one bit of information we have suggests that the root cause was a networking issue, which suggests SPOF.
- leoc 16y ago
  If I had to, I'd guess that AWS's messaging/monitoring/control infrastructure is likely to be the SPOF, as in the 2008 outage: http://status.aws.amazon.com/s3-20080720.html http://status.aws.amazon.com/s3-20080720.html It's an obvious weak spot in the independence/isolation of AWS' nodes, and it would seem to be the one most likely to cause failure to reach across more than one AZ. (Apart from a stampede from one, affected AZ to others perhaps.)

bphogan 16y ago

We always, always use the SLA offered by a vendor as the basis for our information. We trust it more than any marketing page, sales pitch, tech support FAQ, or anything else. That's what they'll hide behind, so that's what I'll have in mind when I design my setup.

justinsb 16y ago
I think it's great to check the SLA. However, there's enough wiggle room in the AWS SLA that I think this outage could continue for the rest of the month, and Amazon would still not owe a penny. I don't even know that the SLA covers this outage, because network connectivity isn't affected. Even if Amazon breach their SLA, I think they only have to refund 10% of one month's bill per year - i.e. a 1% discount. I suspect they'd make a good profit even if they paid out a full 10% refund every month. Unless an SLA is accelerated - i.e. >100% refund - I don't think it's worth taking particularly seriously. Of course if an SLA only guarantees 95% uptime, that's probably a big hint to design for failure!
- bphogan 16y ago
  Yeah but I don't care about getting my money back as much as I care about how much they claim to be down. It's like the hard disk maker that gives you a 1 year warranty vs a 5 year warranty... which one believes in their product more? :)
  - justinsb 16y ago
    It's a good analogy and I certainly accept your point. It could just be a marketing thing though: Suppose it's the same hard disk with a black sticker instead of a blue sticker. Drive with 1 yr warranty @ $100, 5 yr warranty @ $150, 20% additional failure rate over the extra 4 years, 50% redemption rate on failed drives. Cost per replaced drives = 20% * 50% * ($100 + $30 processing costs) = $13 = $37 profit. Totally fictitious numbers to try to prove my point, of course :-) But as the SLA becomes increasingly low in value, the signalling value decreases in my book. (Edit - fixed my math!)
  - 16y ago

pessimist 16y ago

I'm sorry, designing your service without taking in to account the SLA is just stupid. See how Netflix survived the failure for example. Now if you understand the SLA and still choose not to do cross-region deployments, then you've taken a cost/complexity vs uptime trade-off, which may well be right for you. quora.com probably is ok - who cares if its down for a day?