9 ms·

AWS Service Disruption Post Mortem

• gojomo 15y ago
  I doubt this is the last time we'll hear of a "re-mirroring storm" in an oversaturated cloud.
  • sliverstorm 15y ago
    Oversaturated? How do you figure? 13% were unable to re-mirror, which means 87% were able to. In short, nearly 40% of the 'cloud' was free space.
    • gojomo 15y ago
      The 're-mirroring storm' occurred when all free space was exhausted. At that point, the EBS storage resources were oversaturated. Then the 'EBS control plane' started to fail because 'slow API calls began to back up and resulted in thread starvation'. At that point, the EBS processing resources were oversaturated. Then other nearby systems got wet.
• thehodge 15y ago
  An automatic 100% credit for 10 days usage, thats pretty good IMO
  • tomjen3 15y ago
    Well yes, except that that is usually peanuts compared to the lost income from your service being down. Really the only purpose of a SLA penalty is to incentivize the provider to keep the network reliable.
    • RyanKearney 15y ago
      I still don't know why people keep throwing this out there. Yes, if you were effected and it caused you to lose a lot of income because your business could not operate when EC2 was on the fritz then it's your fault, not Amazons. You can't keep blaming Amazon because you didn't build a fault-tolerant application. It's like blaming your electric company because your home is lit by one super huge flood light which burnt out and as a result you couldn't see or get any work done because you kept no backup bulbs in the house.
      • wanderr 15y ago
        I think it's more like getting upset with your cable provider when their service goes down leaving you stranded with no internet. Arguably, it's your fault that you don't have any internet, you could have had redundant internet access between cable and DSL and you have no one else to blame for you decision to pay for only one.
    • com 15y ago
      I totally agree with your comment about the SLA penalty as an incentive to the provider to take reasonable measures to ensure service. But that's just in general. When negotiating bespoke SLA penalty clauses, it can be very illuminating for both sides to discuss lost profit + lost confidence + additional costs to the customer and suggest that these be factored in to the penalty clause. My experience: both the customer and supplier tend to take a deep breath to evaluate whether this deal is a good one for either of them and begin to reassess their level of risk. In a off-the-shelf service like Amazon, you as a customer are welcome to suggest a change of penalty to your Amazon account manager, and unless you're something like the US government, you will probably be directed to other cloud providers or your own internal IT organisation!
      • cosmicray 15y ago
        > In a off-the-shelf service like Amazon, you as a customer are welcome to suggest a change of penalty to your Amazon account manager, and unless you're something like the US government, you will probably be directed to other cloud providers or your own internal IT organisation! What that suggests to me, is that the time has arrived for an external organization, one that sells loss-of-business protection against such failures, needs to become involved. Such an organization, should enough cloud customers subscribe to it, would become an influence upon services like AWS. I'm not sure I 'like' this idea, but the premise that a customer is using the cloud service at the whim of whatever the provider decides is best practice needs to be revisited.
  • dholowiski 15y ago
    I agree, the credit is nice, although for me it amounts to about $6. The lost opportunity for me is huge... I have a QR code app, and missed the chance to have my QR code in a major print magazine ad. That's missed traffic, and missed exposure to the company who was going to use it - that was worth way more than $6 to me. Unfortunately as a hobby site, it's tough to justify multiple servers for redundancy
    • dedward 15y ago
      You seem to be saying it was worth a lot to you and not worth much to you at the same time - that makes no sense.
• thebootstrapper 15y ago
  Reminds me again, Distributed System are hard and the first fallacies "The network is reliable"
• charper 15y ago
  Seems there is always this issue. System fails. Systems try to repair themselves. Systems saturate something which stops them from repairing. Systems all loop aggressively bringing it all down.
• hobbes 15y ago
  >...one of the standard steps is to shift traffic off of one of the redundant routers in the primary EBS network to allow the upgrade to happen. The traffic shift was executed incorrectly... This supports the theory that between 50%-80% of outages are caused by human error, regardless of the resilience of the underlying infrastructure.
  • tomjen3 15y ago
    Which leaves a question: why not engineer around humans, such that they are never needed in the day-to-day running of the systems?
    • henrikschroder 15y ago
      "The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future."
      • tomjen3 15y ago
        Sure now, but I wanted to know why they didn't do that from the beginning.
        • michael_dorfman 15y ago
          Why they didn't do what? "Increase the automation"? I suspect that they have been doing that from the beginning. It's an ongoing process. I hate to quote Rumsfeld, but there are known unknowns, and unknown unknowns. Of course you want to eliminate the latter-- but there's (necessarily) no way you can ever know that you've done so.
    • neworbit 15y ago
      Any high availability system inherently aims for that, but "never" is a strong word. Sooner or later, you need to upgrade or replace physical hardware.
    • 15y ago
• nodata 15y ago
  tl;dr version?
  • deleted 15y ago
    [deleted]
  • chrisboesing 15y ago
    An EBS node in a EBS cluster is connected to two networks. One is used for the traffic to and from the EBS volumes(Primary network), the other is used to replicate the EBS volume on a EBS node to a different EBS node(Secondary network). Amazon wanted to upgrade the capacity of the primary network. Their standard step doing this is to shift the traffic to a redundant router. This step was executed incorrectly. This resulted in the traffic not being routed to the primary network but instead to the secondary network which has less capacity. All this traffic satured the secondary network and resulted in the EBS volumes becoming "stuck". When the traffic got routed the right way all the EBS volumes were trying to remirror. Part of the remirroring process is that the EBS volumes search the cluster for free space to remirror to. The EBS cluster couldn't handle this load and new capacity was needed for the EBS cluster. Amazon offers a 10 day credit equal to 100% of their usage of EBS Volumes, EC2 Instances and RDS database instances. This credit will be automatically applied to the next bill.
  • neuroelectronic 15y ago
    The muppets in TechOps borked a switch switch.
  • jerhewet 15y ago
    > tl;dr version? Only a fool would try to run their business out of "the cloud".
• senthilnayagam 15y ago
  AWS was numero uno in terms of customer visibility and the image of a pathbreaking cloud service, before the incident. Lack in transparency in reaching out to customers is the biggest mistake what AWS did. They would learn from their mistakes, their servers and networks would be more reliable than ever. This incident has given a reason for people to look at multi-cloud operation capability, for disaster recovery and backup reasons. AWS monopoly would be gone, there would be many new standards which would be proposed to bring in interoperability and for migrations between clouds.
• nicpottier 15y ago
  tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber of people there at that point was insane. By far the smartest bunch I've ever worked with, and a place where I learned habits that serve me well to this day. I know the guys that started the AWS group and they were the best of that already insanely selective group. It is easy to be an arm chair coach and scream that the network changes should have been automated in the first place, or that they should have predicted this storm, but that ignores just how fantastically hard what they are doing is and how fantastically well it works 99(how many 9's now?)% of the time. In short, take my word for it, the people working on this are smarter than you and me, by an order of magnitude. There is no way you could do better, and it is unlikely that if you are building anything that needs more than a handful of servers you could build anything more reliable.
  • g123g 15y ago
    Agreed. It has actually given a very nice excuse to developers like me. Whenever your manager comes up to find why something that you are working on is not up, you can simply reply with something like this - If AWS with some of the smartest engineers can be down for that long do you think that our crappy service will be up 100% of time?
    • deleted 15y ago
      [deleted]
  • deleted 15y ago
    [deleted]
  • fauigerzigerk 15y ago
    If it's too hard for the best then the concept is dead. But I don't believe it. They made some avoidable mistakes. Just look at the pattern emerging from these kinds of incidents. There's an automatic cluster recovery mechanism that works for individual node failures but makes matters worse once a larger number of nodes fail. I wonder whether they did extensive testing or simulation of that scenario. The initial root cause is probably unpredictable because there may be many, but what follows is not unpredictable. I'm not ready to concede that because they are such an insanely smart elite group of people we just have to live with week long outages.
• Smerity 15y ago
  > The nodes in an EBS cluster are connected to each other via two networks. The primary network is a high bandwidth network... The secondary network, the replication network, is a lower capacity network used as a back-up network... This network is not designed to handle all traffic from the primary network but rather provide highly-reliable connectivity between EBS nodes inside of an EBS cluster. During maintenance instead of shifting traffic off of one of the redundant routers the traffic was routed onto the lower capacity network. There was human error involved but the network issue only provoked latent bugs in the system that should have been picked out during disaster recovery testing. Automatic recovery that isn't properly tested is a dangerous beast; it can cause problems faster and broader than any team of humans are capable of handling.
• thebootstrapper 15y ago
  One of the main cause for "re-mirroring storm," is node not backing off from finding a replica. Here's Twitter Back off decider implementation (Java) https://github.com/twitter/commons/blob/master/src/java/com/twitter/common/util/BackoffDecider.java https://github.com/twitter/commons/blob/master/src/java/com/... When last time i looked i was little clueless on this. Now I find its usage.
  • biot 15y ago
    Actual URL with Libya dependency removed: https://github.com/twitter/commons/blob/master/src/java/com/twitter/common/util/BackoffDecider.java https://github.com/twitter/commons/blob/master/src/java/com/... HN doesn't have a 140 character limit, so there's no need to post an obfuscated shortened link.
    • thebootstrapper 15y ago
      My fault. Edited the link. Thanks.
• assiotis 15y ago
  I find it surprising that they did not and do not plan to employ any sort of interlocks/padded walls. What I mean is, if the system is exhibiting some very abnormal state (e.g #remirror_event above a fixed threshold or more than x standard deviations above average) then automated repair actions should probably stop and the issue should be escalated to a human.
  • neuroelectronic 15y ago
    They will probably do that now. They will probably also make sure they have a powerful SOP for network upgrades as well.
• mml 15y ago
  Did I read this correctly in paragraph 2: " For two periods during the first day of the issue, the degraded EBS cluster affected the EBS APIs and caused high error rates and latencies for EBS calls to these APIs across the entire US East Region." Their "control plane" network for the EBS clusters span availability zones in a region? If so, this would be the fatal flaw.
  • ra 15y ago
    And a breach of the "share nothing" tenet, which is quite important here.
    • extension 15y ago
      I don't see how transparently replicated storage could be implemented without sharing something.
      • ra 15y ago
        In which case they shouldn't replicate across availability zones. 'shared nothing' is the only way to islandize failures.
        • space-monkey 15y ago
          And that's why they have multiple fully-isolated regions. Availability zones are a purposeful tradeoff that provides easier to use service with higher inter-zone communication performance and lower cost.
          • ra 15y ago
            No, that's not what Amazon says. The following is from the AWS web site [1]: > Availability Zones are distinct locations that are engineered to be insulated from failures in other Availability Zones and provide inexpensive, low latency network connectivity to other Availability Zones in the same Region. By launching instances in separate Availability Zones, you can protect your applications from failure of a single location. No mention of tradeoffs. [1] http://aws.amazon.com/ec2/ http://aws.amazon.com/ec2/
• epi0Bauqu 15y ago
  They should also allow one-time moves of reserved instances between availability zones.
  • amock 15y ago
    What would the purpose of that be?
    • epi0Bauqu 15y ago
      They want to help people make better use of multiple availability zones. People may have reserved a bunch of instances, but would be better off distributing those more effectively across zones.
      • amock 15y ago
        That makes sense. I wonder if something like the free realm transfers in World of Warcraft would make sense. Maybe the AZ mapping randomization keeps things balanced but encouraging multi-az deployments seems like a good idea.
• mikiem 15y ago
  The whole thing is just too complicated to be highly-available. There will be more problems, but I wish them luck.
  • ra 15y ago
    That's a bit defeatist. In my first year of uni, one of our lecturers drummed something into us: Q. How do you eat an elephant? A. One bite at a time
• deleted 15y ago
  [deleted]
• bigiain 15y ago
  "The traffic shift was executed incorrectly and rather than routing the traffic to the other router on the primary network, the traffic was routed onto the lower capacity redundant EBS network." I'm _so_ glad I wasn't "that guy"! (I have been "that guy" before, but never with quite such a serious fuckup multiplier in place...)
• wanderr 15y ago
  I highly recommend that anyone who was surprised by this outage, or the description of the chain reaction of failures that lead to it, read Systemantics. It is a dry but amusing exploration of the seemingly universal fact that every complex system is always operating in a state of failure, but the complexity, failovers and multiple layers can hide this, until the last link in the chain finally breaks, usually with catastrophic results.
  • gruseom 15y ago
    read Systemantics Oh yes. It's a classic that deserves to be much better known. Anybody engaged with complex systems - such as software or software projects - will find all kinds of suggestive things in there. As for "dry"... come now, it's hilarious and has cartoons. Basically, just get it. Here, I'll help: http://www.amazon.com/Systems-Bible-Beginners-Guide-Large/dp/0961825170 http://www.amazon.com/Systems-Bible-Beginners-Guide-Large/dp... (They ruined the title but it's the same book.)
• gord 15y ago
  This article reads like nonsense - but this is not a criticism of AWS. The real problem is there is no good mathematical model of distributed behaviour, from which statistical guarantees can be made. I think we're at the limit of what the smartest people can achieve with hand crafted code. Most likely new math will give rise to new tools and languages, in which the next generation of reliable distributed systems will be written. Without this advance we will have storage networks that aren't reliable, an internet that can be taken down by one organization, botnets that are unkillable and patchy network security.
  • brown9-2 15y ago
    I'm curious - what about this post-mortem "reads like nonsense"?
    • gord 15y ago
      All of it, the general approach is wrong - so the nonsense is the part where you believe your current set of abstractions about distributed networks are adequate. Also the part where you reapply those same abstractions to fix the hole, without realizing that the problem is you simply don't yet have tools that are capable of writing a robust system - despite the evidence to the contrary. If a day long outage of this scale is not enough to make us rethink distributed systems, what is?
• rdl 15y ago
  I still don't see a good justification for keeping the ebs control plane exposed to failure across multiple availability zones in a region. Until that is fixed, I would not depend on AZs for real fault tolerance.
• leoc 15y ago
  Compare to the 2008 post-mortem: http://status.aws.amazon.com/s3-20080720.html http://status.aws.amazon.com/s3-20080720.html Messaging infrastructure as single point of failure? Check. http://news.ycombinator.com/item?id=2472227 http://news.ycombinator.com/item?id=2472227
• bretthopper 15y ago
  I've been noticing a trend recently when reading about large scale failures of any system: it's never just one thing. AWS EBS outage, Fukushima, Chernobyl, even the great Chicago Fire (forgive me for comparing AWS to those events). Sure there's always a "root" cause, but more importantly, it's the related events that keep adding up to make the failure even worse. I can only imagine how many minor failures happen world wide on a daily basis where there's only a root cause and no further chain of events. Once a system is sufficiently complex, I'm not sure it's possible to make it completely fault-tolerant. I'm starting to believe that there's always some chain of events which would lead to a massive failure. And the more complex a system is, the more "chains of failure" exist. It would also become increasingly difficult to plan around failures. edit: The Logic of Failure is recommended to anyone wanted to know more about this subject: http://www.amazon.com/Logic-Failure-Recognizing-Avoiding-Situations/dp/0201479486 http://www.amazon.com/Logic-Failure-Recognizing-Avoiding-Sit...
  • akuchling 15y ago
    A similar point is made in Gene Weingarten's "Fatal Distraction" (http://www.pulitzer.org/works/2010-Feature-Writing http://www.pulitzer.org/works/2010-Feature-Writing), which was about parents who forget a child in the car. Excerpt: "[British psychologist James Reason] likens the layers to slices of Swiss cheese, piled upon each other, five or six deep. The holes represent small, potentially insignificant weaknesses. Things will totally collapse only rarely, he says, but when they do, it is by coincidence -- when all the holes happen to align so that there is a breach through the entire system."
  • siculars 15y ago
    This is an interesting point you hit on and something that is stressed in scuba diving. Basically whenever you go out on a dive you only want to change one thing at a time. Only one thing can be "new" or "untested" or "new to you". Otherwise you run the risk of being task overloaded which leads to cascading failure - potentially catastrophic and/or nonrecoverable.
  • rapind 15y ago
    I've been an AWS (S3, EC2, SQS) user for over 3 years now and this article detailing their systems at a mid-level is kind of scaring me off of their platform. It just sounds so complicated and I'm not sure I want to rely on it for anything critical until I can really understand it myself. Also, a couple other complex systems for your trend are; financial markets and commercial jets.
• johndbritton 15y ago
  "We will look to provide customers with better tools to create multi-AZ applications that can support the loss of an entire Availability Zone without impacting application availability. We know we need to help customers design their application logic using common design patterns. In this event, some customers were seriously impacted, and yet others had resources that were impacted but saw nearly no impact on their applications."
• mcpherrinm 15y ago
  There's a quote I found interesting that hasn't been noted here yet: "This required the time-consuming process of physically relocating excess server capacity from across the US East Region and installing that capacity into the degraded EBS cluster." And if I read this description of the re-mirror storm correctly, I think that implies Amazon had to increase the size of it's EBS cluster in the affected zone by 13%, which considering the timeline seems fairly impressive.
• pwzeus 15y ago
  I for once just want to say that claps to them for figuring this out , nailing it down in fixing it in just few days. After reading this if feels like issue at such massive level can take large amount of time to fix.
• moe 15y ago
  Now that's what I call a post mortem. Kudos to the author.
• AdamGibbins 15y ago
  I found this rather entertaining: http://intraspirit.net/images/aws-explained.png http://intraspirit.net/images/aws-explained.png
  • fawxtin 15y ago
    Im getting a 404, is this the same? (Amazon service disruption "explained" by an employee) http://lgv.s3.amazonaws.com/AmazonFail_explainedByEmployee.jpg http://lgv.s3.amazonaws.com/AmazonFail_explainedByEmployee.j...
• VladRussian 15y ago
  interesting, several weeks ago someone (reddit?) has already hit the problems with EBS availability. Did Amazon paid attention and analyzed the problem back then? Or let it just pass?
• JasonMoyMN 15y ago
  What few people seem to acknowledge is that the post-mortem doesn't matter. It may or may not help Amazon, because the next outage (which will occur, as it does in any data center) will likely have completely different causes than this one. The post-mortem does not help the consumer in the least. Whether the cause of the outage was the long series of complicated steps explained in the post-mortem, or whether it was due to Engineer X accidentally tripping over a cable, the effect is the same. The publication of the post-mortem is simply meaningless. It would make much more sense for them to publish details of their entire architecture and infrastructure.
• mauricio 15y ago
  It's strange we haven't heard more from users of the 0.07% of EBS volumes that were corrupted and unrecoverable during the outage. I just assumed there was no data loss as a result of the outage.