7 ms·

tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from

tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber of people there at that point was insane. By far the smartest bunch I've ever worked with, and a place where I learned habits that serve me well to this day. I know the guys that started the AWS group and they were the best of that already insanely selective group. It is easy to be an arm chair coach and scream that the network changes should have been automated in the first place, or that they should have predicted this storm, but that ignores just how fantastically hard what they are doing is and how fantastically well it works 99(how many 9's now?)% of the time. In short, take my word for it, the people working on this are smarter than you and me, by an order of magnitude. There is no way you could do better, and it is unlikely that if you are building anything that needs more than a handful of servers you could build anything more reliable.

g123g 16y ago
Agreed. It has actually given a very nice excuse to developers like me. Whenever your manager comes up to find why something that you are working on is not up, you can simply reply with something like this - If AWS with some of the smartest engineers can be down for that long do you think that our crappy service will be up 100% of time?
- deleted 16y ago
  [deleted]
deleted 16y ago
[deleted]
fauigerzigerk 16y ago
If it's too hard for the best then the concept is dead. But I don't believe it. They made some avoidable mistakes. Just look at the pattern emerging from these kinds of incidents. There's an automatic cluster recovery mechanism that works for individual node failures but makes matters worse once a larger number of nodes fail. I wonder whether they did extensive testing or simulation of that scenario. The initial root cause is probably unpredictable because there may be many, but what follows is not unpredictable. I'm not ready to concede that because they are such an insanely smart elite group of people we just have to live with week long outages.
- ra 16y ago
  I think it's more of a learning experience. In the grand scheme of things, it's still day one for computing services like AWS.
- nicpottier 16y ago
  The problem with all this is that you never hear about the many many failures that occur (daily, hourly at the scale they operate?) that don't cause an outage.
- scott_s 16y ago
  I think you're drawing the wrong conclusions from the grandparent post. It's not These guys are the best, so we will just have to live with these failures. It's These guys are smart people sailing in uncharted waters, so they're going to get a little lost now and then. The second conclusion implies that these waters will eventually become charted, so everyone will be able to avoid these problems in the future.
  - fauigerzigerk 16y ago
    To me, the parent as well as your comment sounds a little bit too apologetic. The waters are not totally uncharted and the outage was disproportionate. AWS isn't a research project, it's a commercial offering, so they have to take some blame. That re-mirroring storm was not completely unforeseeable. It's exactly the thing you have to consider when designing these kinds of systems.
jwe 16y ago
It's not so much about thinking that somebody else could do better. By doing what they do they create the expectation that they _are_ doing better than everybody else.
- lemming 16y ago
  And let's face it, despite bugs and human error that should be avoidable, they are.
  - ssmoot 16y ago
    At that scale, MAYBE. In the last two years my workplace has gotten pretty good at handling SAN outages (due to terrible Oracle equipment). Put simply, this set of scenarios can't happen at my work-place. We don't have that level of automation, there's only a pair of SAN systems in the mirrors, there's no "hunting for capacity". I'd suggest most businesses are closer to this than AWS. There's a lot of sugar coating going around saying "You couldn't build a better space-shuttle", and that's probably true. But if I only need an extremely reliable bicycle that's a false argument. The Simplest Thing That Could Possibly Work doesn't apply just to programming.
kamaal 16y ago
The best are human beings too! I mean even the best work in two ways. Build on best principles already known, and make your own best principles. The first depends on how much you know, how much time you have taken to learn, implement, practice and perfect. The second depends on how far you can see. In either case you can always overlook or fail to predict the even the easily foreseeable future. And that happens due to many reasons plain human error, or even over confidence which is some times the case with the best. The best way out of this problem is what Jeff Atwood had blogged some time days back. Is to keep failing, and keep failing in different ways. And each failure needs to be translated into to lessons of some sort and then the solution of it a best practice. Even the Netflix model of failing purposefully will do. There is no way the best can be flawless. Nothing is flawless, so as long it is done by a human.
e98cuenc 16y ago
I've been working at Google, and had a similar experience. But I do think in many cases you can actually do better. It's not that you're smarter than Amazon/Google/... engineers, it's that you're solving a problem that it's orders of magnitude easier. Your start-up is probably not dealing with setting up & maintaining 200K servers as Amazon, so up to a limit you can actually be better going on your own.
- nicpottier 16y ago
  Yep, that was my point with the 'handful of servers' line. But even that is pretty darn hard. You still have to deal with split brain situations, do all the right things in those cases etc.. If you are a tiny startup that isn't going to hire a dedicated sysadmin (very few of which can build that type of thing without buying some expensive hardware) then EC2 is probably a better choice. Even just the hardware costs make it pretty braindead. We just shut down the dedicated three system setup I built because it just didn't make sense financially, and that despite me spending a month learning the intricacies of heartbeat it still had odd failure scenarios that we don't experience with EC2. Again, I'm just not that smart. :)
ekidd 16y ago
Given a choice between hosting servers on AWS, and trying to build my own reliable infrastructure with a single sysadmin, I'll take AWS in a heartbeat. But I do want to quibble with one of your points: It is easy to be an arm chair coach and scream that... they should have predicted this storm I'm not as smart as the AWS developers, and I have a lot less experience with large-scale distributed systems. But thanks to my own cluelessness, I've blown up smaller distributed systems, and I've learned one important lesson: Almost nobody is smart enough to understand automatic error-recovery code. Features like automated volume remirroring or multi-AZ failover increase the load on an already stressed system, and they often cause this kind of "storm." So I've learned to distrust intelligence in these matters. If you want to understand how your system reacts when things start going wrong, you have to find a way to simulate (or cause) large-scale failures: This is something that Google does really really well by the way, I've watched them turn of 25 core routers simultaneously carrying hundreds of gigabits worth of data, just to verify that what they think will happen, does happen. http://news.ycombinator.com/item?id=2475112 http://news.ycombinator.com/item?id=2475112 You also need to pay particular attention to components with substantial, ongoing problems, and make sure you don't let known issues linger: I work at Amazon EC2 and I can tell you what's going on (thanks to this handy throwaway account). What's happening is the EBS team gets inundated with support tickets due to their half-assed product. Here's the hilarious part: whenever we've asked them why they don't fix the main issue, they keep telling us that they're too busy with tickets. What they don't seem to realize is that if they fixed the core issue the tickets would go away. http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_down_for_6_of_the_last_24_hours/c1l7vy1 http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d... Now, I'm not saying I could have done any better than Amazon (evidence suggests otherwise). But I do know that I'm not smart enough to understand these systems without testing them to destruction, and aggressively fixing the root causes of known problems.
- SoftwareMaven 16y ago
  I would be surprised if Amazon didn't do testing similar to the Google quote above. AWS is insanely complicated, and the problem with an insanely complicated system is there are infinite failure modes (or such a big number it is close enough to infinite for human purposes). As such, it is impossible to test all of them.
ianso 16y ago
Agreed with the expertise thing, but to add to your summary of the post-mortem, it looks like human error compounded by: 1) An architecture bug (the EBS "control plane" cuts across Availability Zones and EBS clusters, leading to a single point of failure: this is what broke the "service contract"), 2) a spec/programming bug: No aggressive back-off on retry attempts of EBS ops, and 3) two separate logical bugs: the race condition in the EBS nodes & problems with MySQL replication. I think that's everything. It just goes to show, most disasters in very well-engineered systems are generally the result of a series of things all going wrong at once, not individual failures...
- OllieJones 16y ago
  Agree completely. We are still in the early days of AWS-like technology. Providers and users both will experience some serious issues in the years to come. Heck, users of municipal water systems still experience outages and that technology is arguably mature. The really encouraging thing is that the Amazon post-mortem writeup indicates they're taking it very seriously.
rlpb 16y ago
> There is no way you could do better, and it is unlikely that if you are building anything that needs more than a handful of servers you could build anything more reliable. I can't disagree, but there is one key benefit to not using the cloud for some services. When your company is working on an important deadline, your sysadmins could choose not to implement that pending network configuration change during that crucial period. You can control your own at-risk times, which you can't generally do with IaaS. As with everything, it's a trade off.
leoc 16y ago
See, this makes me uneasy. The best and brightest work very hard to avoid memory errors in their C codebases then, when errors occur anyway, console themselves that no-one could have done any better. The less brilliant look into more automatic forms of memory management. AT&T hired the best and brightest to make heroic efforts so that the old circuit-switched Bell system could achieve mediocre reliability. I certainly hope that the best and brightest at AWS aren't intending to tackle their enormous control-backplane SPOF by assiduously patching every bug that turns up in its behaviour.
icedpulleys 16y ago
My tl;dr reads differently: "we lacked appropriate congestion control in our EBS recovery algorithms." The trigger was a bad and unexpected network configuration change, but the error was that the attempted recovery by the stuck volumes was that it was uncontrolled. I don't think that anyone is knocking the intelligence of the AWS engineers, or saying that anyone else could do it better. Just like NASA engineers and scientists are incredibly intelligent and good at what they do, systems can become complicated enough that unexpected errors creep into the system, and not any particular component.
JohnsonB 16y ago
A single manual configuration mistake should never be able cause this type of complete system failure. For AWS to be engineered this way, the engineers you met must not have been as smart as they seemed. EBS may be extremely sophisticated in other respects, but if this type of thing was even a possibility, the AWS team are very fallible, and probably far from the best in their field. >In short, take my word for it, the people working on this are smarter than you and me, by an order of magnitude. There is no way you could do better, and it is unlikely that if you are building anything that needs more than a handful of servers you could build anything more reliable. Ever since the AWS outage, I've seen a number of these "the AWS guys are so smart, I've met them." type comments. And then paraphrased: "There sort-of can't be that much to blame on them because of how smart they are, and they are so smart anyway, who could do better?" That's not a valid argument, not everyone is equally impressed by an individuals intelligence, perhaps your assessment is wrong. And even if someone is insanely smart, they still can commit practical errors which indicates they are smart, but still flawed in their understanding of engineering in significant ways. Perhaps AWS simply does need a higher caliber of engineer that wouldn't miss out on these dead-simple safeguards that would have prevented this outage.

g123g 16y ago

Agreed. It has actually given a very nice excuse to developers like me. Whenever your manager comes up to find why something that you are working on is not up, you can simply reply with something like this - If AWS with some of the smartest engineers can be down for that long do you think that our crappy service will be up 100% of time?

deleted 16y ago
[deleted]

deleted 16y ago

[deleted]

fauigerzigerk 16y ago

If it's too hard for the best then the concept is dead. But I don't believe it. They made some avoidable mistakes. Just look at the pattern emerging from these kinds of incidents. There's an automatic cluster recovery mechanism that works for individual node failures but makes matters worse once a larger number of nodes fail. I wonder whether they did extensive testing or simulation of that scenario. The initial root cause is probably unpredictable because there may be many, but what follows is not unpredictable. I'm not ready to concede that because they are such an insanely smart elite group of people we just have to live with week long outages.

ra 16y ago
I think it's more of a learning experience. In the grand scheme of things, it's still day one for computing services like AWS.
nicpottier 16y ago
The problem with all this is that you never hear about the many many failures that occur (daily, hourly at the scale they operate?) that don't cause an outage.
scott_s 16y ago
I think you're drawing the wrong conclusions from the grandparent post. It's not These guys are the best, so we will just have to live with these failures. It's These guys are smart people sailing in uncharted waters, so they're going to get a little lost now and then. The second conclusion implies that these waters will eventually become charted, so everyone will be able to avoid these problems in the future.
- fauigerzigerk 16y ago
  To me, the parent as well as your comment sounds a little bit too apologetic. The waters are not totally uncharted and the outage was disproportionate. AWS isn't a research project, it's a commercial offering, so they have to take some blame. That re-mirroring storm was not completely unforeseeable. It's exactly the thing you have to consider when designing these kinds of systems.

jwe 16y ago

It's not so much about thinking that somebody else could do better. By doing what they do they create the expectation that they _are_ doing better than everybody else.

lemming 16y ago
And let's face it, despite bugs and human error that should be avoidable, they are.
- ssmoot 16y ago
  At that scale, MAYBE. In the last two years my workplace has gotten pretty good at handling SAN outages (due to terrible Oracle equipment). Put simply, this set of scenarios can't happen at my work-place. We don't have that level of automation, there's only a pair of SAN systems in the mirrors, there's no "hunting for capacity". I'd suggest most businesses are closer to this than AWS. There's a lot of sugar coating going around saying "You couldn't build a better space-shuttle", and that's probably true. But if I only need an extremely reliable bicycle that's a false argument. The Simplest Thing That Could Possibly Work doesn't apply just to programming.

kamaal 16y ago

The best are human beings too! I mean even the best work in two ways. Build on best principles already known, and make your own best principles. The first depends on how much you know, how much time you have taken to learn, implement, practice and perfect. The second depends on how far you can see. In either case you can always overlook or fail to predict the even the easily foreseeable future. And that happens due to many reasons plain human error, or even over confidence which is some times the case with the best. The best way out of this problem is what Jeff Atwood had blogged some time days back. Is to keep failing, and keep failing in different ways. And each failure needs to be translated into to lessons of some sort and then the solution of it a best practice. Even the Netflix model of failing purposefully will do. There is no way the best can be flawless. Nothing is flawless, so as long it is done by a human.

e98cuenc 16y ago

I've been working at Google, and had a similar experience. But I do think in many cases you can actually do better. It's not that you're smarter than Amazon/Google/... engineers, it's that you're solving a problem that it's orders of magnitude easier. Your start-up is probably not dealing with setting up & maintaining 200K servers as Amazon, so up to a limit you can actually be better going on your own.

nicpottier 16y ago
Yep, that was my point with the 'handful of servers' line. But even that is pretty darn hard. You still have to deal with split brain situations, do all the right things in those cases etc.. If you are a tiny startup that isn't going to hire a dedicated sysadmin (very few of which can build that type of thing without buying some expensive hardware) then EC2 is probably a better choice. Even just the hardware costs make it pretty braindead. We just shut down the dedicated three system setup I built because it just didn't make sense financially, and that despite me spending a month learning the intricacies of heartbeat it still had odd failure scenarios that we don't experience with EC2. Again, I'm just not that smart. :)

ekidd 16y ago

Given a choice between hosting servers on AWS, and trying to build my own reliable infrastructure with a single sysadmin, I'll take AWS in a heartbeat. But I do want to quibble with one of your points: It is easy to be an arm chair coach and scream that... they should have predicted this storm I'm not as smart as the AWS developers, and I have a lot less experience with large-scale distributed systems. But thanks to my own cluelessness, I've blown up smaller distributed systems, and I've learned one important lesson: Almost nobody is smart enough to understand automatic error-recovery code. Features like automated volume remirroring or multi-AZ failover increase the load on an already stressed system, and they often cause this kind of "storm." So I've learned to distrust intelligence in these matters. If you want to understand how your system reacts when things start going wrong, you have to find a way to simulate (or cause) large-scale failures: This is something that Google does really really well by the way, I've watched them turn of 25 core routers simultaneously carrying hundreds of gigabits worth of data, just to verify that what they think will happen, does happen. http://news.ycombinator.com/item?id=2475112 http://news.ycombinator.com/item?id=2475112 You also need to pay particular attention to components with substantial, ongoing problems, and make sure you don't let known issues linger: I work at Amazon EC2 and I can tell you what's going on (thanks to this handy throwaway account). What's happening is the EBS team gets inundated with support tickets due to their half-assed product. Here's the hilarious part: whenever we've asked them why they don't fix the main issue, they keep telling us that they're too busy with tickets. What they don't seem to realize is that if they fixed the core issue the tickets would go away. http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_down_for_6_of_the_last_24_hours/c1l7vy1 http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d... Now, I'm not saying I could have done any better than Amazon (evidence suggests otherwise). But I do know that I'm not smart enough to understand these systems without testing them to destruction, and aggressively fixing the root causes of known problems.

SoftwareMaven 16y ago
I would be surprised if Amazon didn't do testing similar to the Google quote above. AWS is insanely complicated, and the problem with an insanely complicated system is there are infinite failure modes (or such a big number it is close enough to infinite for human purposes). As such, it is impossible to test all of them.

ianso 16y ago

Agreed with the expertise thing, but to add to your summary of the post-mortem, it looks like human error compounded by: 1) An architecture bug (the EBS "control plane" cuts across Availability Zones and EBS clusters, leading to a single point of failure: this is what broke the "service contract"), 2) a spec/programming bug: No aggressive back-off on retry attempts of EBS ops, and 3) two separate logical bugs: the race condition in the EBS nodes & problems with MySQL replication. I think that's everything. It just goes to show, most disasters in very well-engineered systems are generally the result of a series of things all going wrong at once, not individual failures...

OllieJones 16y ago
Agree completely. We are still in the early days of AWS-like technology. Providers and users both will experience some serious issues in the years to come. Heck, users of municipal water systems still experience outages and that technology is arguably mature. The really encouraging thing is that the Amazon post-mortem writeup indicates they're taking it very seriously.

rlpb 16y ago

> There is no way you could do better, and it is unlikely that if you are building anything that needs more than a handful of servers you could build anything more reliable. I can't disagree, but there is one key benefit to not using the cloud for some services. When your company is working on an important deadline, your sysadmins could choose not to implement that pending network configuration change during that crucial period. You can control your own at-risk times, which you can't generally do with IaaS. As with everything, it's a trade off.

leoc 16y ago

See, this makes me uneasy. The best and brightest work very hard to avoid memory errors in their C codebases then, when errors occur anyway, console themselves that no-one could have done any better. The less brilliant look into more automatic forms of memory management. AT&T hired the best and brightest to make heroic efforts so that the old circuit-switched Bell system could achieve mediocre reliability. I certainly hope that the best and brightest at AWS aren't intending to tackle their enormous control-backplane SPOF by assiduously patching every bug that turns up in its behaviour.

icedpulleys 16y ago

My tl;dr reads differently: "we lacked appropriate congestion control in our EBS recovery algorithms." The trigger was a bad and unexpected network configuration change, but the error was that the attempted recovery by the stuck volumes was that it was uncontrolled. I don't think that anyone is knocking the intelligence of the AWS engineers, or saying that anyone else could do it better. Just like NASA engineers and scientists are incredibly intelligent and good at what they do, systems can become complicated enough that unexpected errors creep into the system, and not any particular component.

JohnsonB 16y ago

A single manual configuration mistake should never be able cause this type of complete system failure. For AWS to be engineered this way, the engineers you met must not have been as smart as they seemed. EBS may be extremely sophisticated in other respects, but if this type of thing was even a possibility, the AWS team are very fallible, and probably far from the best in their field. >In short, take my word for it, the people working on this are smarter than you and me, by an order of magnitude. There is no way you could do better, and it is unlikely that if you are building anything that needs more than a handful of servers you could build anything more reliable. Ever since the AWS outage, I've seen a number of these "the AWS guys are so smart, I've met them." type comments. And then paraphrased: "There sort-of can't be that much to blame on them because of how smart they are, and they are so smart anyway, who could do better?" That's not a valid argument, not everyone is equally impressed by an individuals intelligence, perhaps your assessment is wrong. And even if someone is insanely smart, they still can commit practical errors which indicates they are smart, but still flawed in their understanding of engineering in significant ways. Perhaps AWS simply does need a higher caliber of engineer that wouldn't miss out on these dead-simple safeguards that would have prevented this outage.