5 ms·

So far the comments have focused on the technical aspects outlined as points of failure in Amazon's summary: grid failure, diesel generator failure, and the com

So far the comments have focused on the technical aspects outlined as points of failure in Amazon's summary: grid failure, diesel generator failure, and the complexities of the amazon stack. What are your thoughts on Amazon's professionalism in their response and action plan going forward? If you're an AWS customer does this style of response keep you on board?

rdl 14y ago
The level of clarity in post incident reporting by Amazon is excellent. During-incident, sub-par. Amazon seems to try to minimize any "more than just a single AZ is affected" in their realtime reporting during outages. There's also a disconnect between the graphics and the text. What I don't like is that they make repeated promises about AZes and features which are repeatedly shown to be untrue. They also have never disclosed their testing methodology, which leads me to assume there isn't much of one. That makes me unlikely to rely on any AWS services I can't myself fully exercise, or which haven't shown themselves to be robust. S3, sure. EC2, sure (except don't depend on modifying instances on any given day during an outage). EBS, oh hell no. ELB, probably not my first choice, and certainly not my sole ft/ha layer. Route 53, which I haven't messed with, actually does seem really good, but since it's complex, I'm scared given the track record of the other complex components.
- bifrost 14y ago
  "What I don't like is that they make repeated promises about AZes and features which are repeatedly shown to be untrue." Thats pretty much SOP in the hosting business, nobody really knows whats going to happen because nobody really knows how to test. Most developers can/do not understand what ops failures are like, and therefore most testing is only superficial.
  - rdl 14y ago
    Right, but I think we've established Amazon is actually a software company which happens to run a retail business and contract for some datacenters. Same way Google is either a supercomputer company or an advertising company which uses search to get users cheaply.
giulianob 14y ago
Well if you look at their status page ( http://status.aws.amazon.com/ http://status.aws.amazon.com/ ) none of the statuses shows red. Yellow clearly says its for performance issues and red is for service disruptions. If this isn't service disruption then I don't know what is.
- donavanm 14y ago
  Check your SLA definitions. http://www.virtualizationpractice.com/technically-the-amazon-cloud-didn%E2%80%99t-actually-fail-10451/ http://www.virtualizationpractice.com/technically-the-amazon... Edit: Not that I agree or disagree with that SLA. Just noting it is documented somewhere besides a glyph in a legend.
Cloven 14y ago
Reading between the lines from both the posted note and their persistent failure to provide correct statuses: the ops guys over there are in full CYA desperation mode somewhere around 100% of the time, and a culture of 'it wasn't me', fuelled either by job fear, or promotion fear, or fear of being noticed by bezos, is in full bloom.

rdl 14y ago

The level of clarity in post incident reporting by Amazon is excellent. During-incident, sub-par. Amazon seems to try to minimize any "more than just a single AZ is affected" in their realtime reporting during outages. There's also a disconnect between the graphics and the text. What I don't like is that they make repeated promises about AZes and features which are repeatedly shown to be untrue. They also have never disclosed their testing methodology, which leads me to assume there isn't much of one. That makes me unlikely to rely on any AWS services I can't myself fully exercise, or which haven't shown themselves to be robust. S3, sure. EC2, sure (except don't depend on modifying instances on any given day during an outage). EBS, oh hell no. ELB, probably not my first choice, and certainly not my sole ft/ha layer. Route 53, which I haven't messed with, actually does seem really good, but since it's complex, I'm scared given the track record of the other complex components.

bifrost 14y ago
"What I don't like is that they make repeated promises about AZes and features which are repeatedly shown to be untrue." Thats pretty much SOP in the hosting business, nobody really knows whats going to happen because nobody really knows how to test. Most developers can/do not understand what ops failures are like, and therefore most testing is only superficial.
- rdl 14y ago
  Right, but I think we've established Amazon is actually a software company which happens to run a retail business and contract for some datacenters. Same way Google is either a supercomputer company or an advertising company which uses search to get users cheaply.

giulianob 14y ago

Well if you look at their status page ( http://status.aws.amazon.com/ http://status.aws.amazon.com/ ) none of the statuses shows red. Yellow clearly says its for performance issues and red is for service disruptions. If this isn't service disruption then I don't know what is.

donavanm 14y ago
Check your SLA definitions. http://www.virtualizationpractice.com/technically-the-amazon-cloud-didn%E2%80%99t-actually-fail-10451/ http://www.virtualizationpractice.com/technically-the-amazon... Edit: Not that I agree or disagree with that SLA. Just noting it is documented somewhere besides a glyph in a legend.

Cloven 14y ago

Reading between the lines from both the posted note and their persistent failure to provide correct statuses: the ops guys over there are in full CYA desperation mode somewhere around 100% of the time, and a culture of 'it wasn't me', fuelled either by job fear, or promotion fear, or fear of being noticed by bezos, is in full bloom.