6 ms·

AWS Post-Mortem

145 points by zeit_geist 15y ago

larrycatinspace 15y ago
I'm thinking AWS needs to implement the Availability Zones: AZ-ChaosMonkey and AZ-ChaosApe. Having a dedicated playground for breaking things where they can start to observe how this complex system reacts to simple failures and gaps in assumptions.
- harshaw 15y ago
  Sure. Presumably Amazon has a test lab that replicates multiple zones :) Perhaps your point is that Amazon should make this test lab public so people can contribute to the QA effort? IIRC many of these datacenter failures start with a utility company power outage followed by a failure of the secondary power systems (I'm thinking of some past failures at softlayer and other providers). I wonder if it is prohibitively expensive to do a real life system test on a big data center (or prohibitively expensive once the data center is on line). For example, how often do they turn off one of the mains (unexpectedly) to see what happens with the backup system?
  - gwern 15y ago
    It's hard to believe that they wouldn't test the mains by doing just that. I visited the NY LaGuardia TRACON recently - built perhaps 40 or 50 years ago? - and saw the generator & battery room, where they turn off the utility power every few months just to see whether things are working. So it's not exactly a new idea or an idea that other life-&-death-mission-critical operations don't dare use.
  - byoung2 15y ago
    IIRC many of these datacenter failures start with a utility company power outage followed by a failure of the secondary power systems That happened at Rackspace a few years back: http://techcrunch.com/2009/06/30/what-went-down-at-rackspace-yesterday-a-power-outage-and-some-backup-failures/ http://techcrunch.com/2009/06/30/what-went-down-at-rackspace... I have an account with GoGrid, and they do a regular testing of their backup generators. I'm not sure if they throw the switch on the mains, though.
    - saetaes 15y ago
      Monthly generator testing is, and should be, standard for any data center. Same with the UPSes - monthly testing to make sure they can handle the load long enough for the generators to kick in. Throwing the switch on the mains is probably not happening anywhere on a regular basis, though. There may be "routine" events (some sort of electrical infrastructure upgrade) that causes the data center to be put onto generator power, but throwing the mains just to test is a very risky endeavor, and one that a data center provider who has very high power availability guarantees with expensive penalties, is not like likely to undertake.
      - jwatte 15y ago
        Why would throwing the switch be risky? It's supposed to be HA. If it doesn't work, that's a bug, and you fix it! Just like backups are not backups until they have been restored (we verify this by making our data warehouse depend on the backup) and hot standbys aren't standbys until switched in (we do this to databases regularly.) Netflix apparently has a chaos generator that randomly kills machines as a standard process. If you're supposed to deal with failure, make sure you're dealing with failure regularly!
        Kadin 15y ago
        > Netflix apparently has a chaos generator that randomly kills machines as a standard process. This sounds pretty neat, but a quick Google didn't turn up any information about it besides this post. Do you know of anywhere to get more information on what they're doing? It sounds like a sensible idea, although I can only imagine trying to implement it would be ... challenging, for most companies/organizations.
        dreww 15y ago
        check out item no 3. on this list, which is AWS lessons-learned: http://techblog.netflix.com/2010/12/5-lessons-weve-learned-using-aws.html http://techblog.netflix.com/2010/12/5-lessons-weve-learned-u... see also: http://techblog.netflix.com/2011/04/lessons-netflix-learned-from-aws-outage.html http://techblog.netflix.com/2011/04/lessons-netflix-learned-... and: http://techblog.netflix.com/2011/07/netflix-simian-army.html http://techblog.netflix.com/2011/07/netflix-simian-army.html for the other simian themed services they've developed for care and feeding of their AWS stuff.
  - cperciva 15y ago
    I wonder if it is prohibitively expensive to do a real life system test on a big data center It's probably prohibitively dangerous. Backup power systems don't have many-nines of reliability; generators which are reliable enough for the once-a-decade event when a car crash knocks out your utility power aren't anywhere near the reliability needed to run your datacentre for an hour every month as a test.
    - ams6110 15y ago
      On the other hand, just as with database backups, making them is only half of the story. You have to test restores/recovery. Does your plan actually work? What have you overlooked? What edge cases do you need to accommodate? Many data centers will test backup power generation regularly just for this reason. It's not unheard of at all and the risk of a problem at a planned time is worth the confidence in knowing that the system is more likely to work when needed at an unexpected time.
    - emaste 15y ago
      Actually, if you don't run test your generator regularly it's very unlikely to work when you do need it. Here's a doc from cummins, a generator mfgr: http://www.cumminspower.com/www/literature/technicalpapers/PT-7004-Maintenance-en.pdf http://www.cumminspower.com/www/literature/technicalpapers/P... It claims that the generator should be run for 30 minutes every month, loaded to at least one third of the rated capacity. So testing every month is exactly what you want to do.
      - rdl 15y ago
        Right, but the thing you don't test is the transfer switch/sync gear. Powering up the generator and dumping the output as heat weekly is pretty standard practice.
        ams6110 15y ago
        Also don't forget to check the fuel tanks. With the rise in fuel prices the past couple of years, theft of diesel from backup generators has become more common.
- spartango 15y ago
  This is in the pipe.
mtkd 15y ago
It's a good communication from Amazon - maybe a little too long - could use a summary block at top. The compensation looks generous too.
- grourk 15y ago
  There's a summary block at the bottom -- but it's not a summary.
saturn 15y ago
As someone who has put a considerable amount of resources moving things into cloud computing - I wanted to believe. But I have changed my mind. Cloud computing scales the efficiencies, yes. It also scales the problems. And because of this, AWS is by several orders of magnitude the worst of my current hosts. I have dedicated servers. No downtime in past year. I have a couple of cloud servers with rackspace. No downtime (although i don't recommend them). I have some VPSes with local providers. No downtime. AWS? More than 24hrs downtime in the last year. Seriously, for someone trying to run web sites reliably - screw that. I'm not using AWS any more. And don't even get me started on the apologists. "EBS slow as treacle? Well you should have been running a multi zone raid-20 redundant array! Duh!". "EC2 instances dying at random? Well you should architect and implement a multi-master failover intelligent grid!" I used to be under some kind of crazy delusional spell that the above was correct and it was somehow my fault that I wasn't correctly adapting to AWS's numerous failings. Well, no more. Now I realise that I should just stick with the super reliable service I know and love from traditional operators. You need to programmatically grow and shrink your app server flock? Great, use AWS. For the other 99.999% of us - stick with what you were using before.
- quanticle 15y ago
  Is there any reason in particular why you wouldn't recommend Rackspace?
  - saturn 15y ago
    Hm. Well, I don't like them. It's subjective, you might disagree. But off the top of my head: 1. Contracts. They want 1 years minimum contracts for any dedicated servers. For truly gargantuan orders I could understand this but for one puny server? Never. 2. Their definition of "cloud" is different from mine. To use their "cloud" services your servers need to be public facing, ie on public IPs. Want them on your own VPN? You can stil get the cloud prices but not the API. you create and cancel servers via tickets. This is different from VPS how? 3. Sloooow provisioning - even if you are able to use their "public cloud" API to provision a server - prepare to wait hours for it to be done, leading me to suspect it does nothing more than email a tech to provision a VPS and hook it up somehow. Oh, you can't pause them to save money either, again making me think these "cloud" servers are nothing more than slicehosts with an extra layer of abstraction Is that enough? I could go on.
rdl 15y ago
The thing I wonder about is wtf they didn't manually switch to generator when their automatic controls failed. They had presumably ~5 minutes of UPS; it took them 40 minutes to do this. This probably isn't directly Amazon's fault, but whatever contract datacenter they are using in Europe (probably a PTT, or possibly an international carrier; really curious what facility) I'm wary of using >1 generators to back up loads, thus requiring sync on generators for backup anyway -- much more comfortable with splitting the load up by room and having one generator per, with some kind of switch to allow for pulling generators out for maintenance. This pretty much limits you to 2-3MW per room (the largest economical diesel gensets), but that's not horrible. Really high reliability sites actually run onsite generation as PRIMARY (since it's less reliable to start), and then utility as backup. With the right onsite generation equipment, it can be cheaper/more efficient than the grid, too (by using combined cycle; use heat output to run cooling directly). Still, the 365 Main power outages take the cake; they used rotational UPSes (generators with huge flywheels) which had software bugs such that if input power got turned off and on several times (a common utility failure mode), the unit shut itself off entirely. Doh.
- jwatte 15y ago
  They explained that a ground fault prevented generators from delivering power. Manual start doesn't help in that case.
  - rdl 15y ago
    From what I read, they said ground fault confused their PLCs (synchro gear for paralleling multiple generators). This shouldn't affect the generator (engine, generator) outputting power. Electronics are much more sensitive to ground faults, etc. than mechanical and electrical devices. A big manual transfer switch (as backup), which is presumably what they ended up using, is fairly bulletproof.
jwatte 15y ago
I thought best practice backup power was to use large flywheels for re-generation, and spin up diesel engines to power the wheel in the event of a loss. That way, there is no phase synchronization issue, just a mechanical clutch. Seems like this outage could have been prevented with better gear?
o1iver 15y ago
There seems to be a pretty simple solution to these problems: diversification. Like most things in life, putting all your eggs in one basket it not the right choice. The people who use only AWS or only RackSpace or only 1&1 are equally wrong. What you have to do it diversify. Run a production site ghost on some other platform (software/hardware bugs, ...), run by some other provider (bankruptcy, theft, ...) in another country (power cuts, earthquake, ...). As soon as the primary goes down you switch on the secondary. The probability of a total blackout is then squared: 10^-3*10^-3=10^-6. The great thing with these "cloud" platforms is that your secondary system can even "go to sleep" saving you money and then spin up instances as soon as the primary goes down. This is by the way how banks, airport-systems and probably the NSA do it!
marcamillion 15y ago
It seems to me that Amazon Web Services will never truly be VERY stable. Not because I am being cynical, but just based on the nature of what they are doing. They are the biggest provider of large scale cloud-based computing services. They are pushing the boundaries. They are bound to always come upon problems that no one has ever seen before (including themselves) just based on the very nature of their business. So if you are looking for 'rock-solid reliability', maybe it is better to wait for another big company (Google, Apple, etc.) to come behind and fix all the mistakes that Amazon made the first time. That being said, I use AWS and I love it. Granted, I don't use EBS (not directly, via Heroku) and yes I have encountered downtime recently, it's not that big of a deal. I know they aren't messing around, and they are in uncharted territory. I can't reasonably expect them to have the best uptime for a platform that no one has ever built or done before, on the first time around the block. That's very unreasonable. That being said, I will continue using them from now until I outgrow them or the economics becomes painful, because the value I get with paying for what I use far outweighs 24 - 48 hours of downtime per year.
gfodor 15y ago
For all those complaining about AWS I think it's important to not fall into the trap of throwing all of Amazon's services into the same bucket. EBS (and hence, RDS) have shown time and time again to be the most complex offerings and more prone to failure. Generally speaking, at least for now, the parts of your system built on top of EBS should be carefully architected to survive in the face of erratic EBS latency, data corruption, or even downtime. (All of which are part of the standard AWS contract, but happen much more often in practice than if you are used to the mean failure time of a hard disk sitting in a cage.) This pattern leads me to believe that services such as VoltDB that do not directly rely upon attached storage will prove to be the paradigm necessary to get reliable cloud computing, at least in the AWS ecosystem. On-demand provisioning of disk is an extraordinary hard problem, and a world where local ephemeral storage provides durability through redundancy across nodes and AZ's is probably where we are headed.
robryan 15y ago
This seems to share a lot of parallels with the last big outage in terms of the API request overload and the EBS replication. Seems like the system need to be able to tell a bit better between a node going down and require a remirror and most of an availability zone going down.
Kartick 15y ago
Isn't the real problem attempting to build an abstraction that lets programmers assume the datacenter won't go down? Aren't you better off building a system like Google's Megastore/BigTable that assumes datacenters go down, and handling it in software? The obvious downside is, of course, that you'll have to rewrite all your software, but doesn't that give higher reliability at the end of the day? Disclosure: I work for Google, but on stuff totally disconnected with any of this.

145 points by zeit_geist 15y ago

I'm thinking AWS needs to implement the Availability Zones: AZ-ChaosMonkey and AZ-ChaosApe. Having a dedicated playground for breaking things where they can start to observe how this complex system reacts to simple failures and gaps in assumptions.

harshaw 15y ago
Sure. Presumably Amazon has a test lab that replicates multiple zones :) Perhaps your point is that Amazon should make this test lab public so people can contribute to the QA effort? IIRC many of these datacenter failures start with a utility company power outage followed by a failure of the secondary power systems (I'm thinking of some past failures at softlayer and other providers). I wonder if it is prohibitively expensive to do a real life system test on a big data center (or prohibitively expensive once the data center is on line). For example, how often do they turn off one of the mains (unexpectedly) to see what happens with the backup system?
- gwern 15y ago
  It's hard to believe that they wouldn't test the mains by doing just that. I visited the NY LaGuardia TRACON recently - built perhaps 40 or 50 years ago? - and saw the generator & battery room, where they turn off the utility power every few months just to see whether things are working. So it's not exactly a new idea or an idea that other life-&-death-mission-critical operations don't dare use.
- byoung2 15y ago
  IIRC many of these datacenter failures start with a utility company power outage followed by a failure of the secondary power systems That happened at Rackspace a few years back: http://techcrunch.com/2009/06/30/what-went-down-at-rackspace-yesterday-a-power-outage-and-some-backup-failures/ http://techcrunch.com/2009/06/30/what-went-down-at-rackspace... I have an account with GoGrid, and they do a regular testing of their backup generators. I'm not sure if they throw the switch on the mains, though.
  - saetaes 15y ago
    Monthly generator testing is, and should be, standard for any data center. Same with the UPSes - monthly testing to make sure they can handle the load long enough for the generators to kick in. Throwing the switch on the mains is probably not happening anywhere on a regular basis, though. There may be "routine" events (some sort of electrical infrastructure upgrade) that causes the data center to be put onto generator power, but throwing the mains just to test is a very risky endeavor, and one that a data center provider who has very high power availability guarantees with expensive penalties, is not like likely to undertake.
    - jwatte 15y ago
      Why would throwing the switch be risky? It's supposed to be HA. If it doesn't work, that's a bug, and you fix it! Just like backups are not backups until they have been restored (we verify this by making our data warehouse depend on the backup) and hot standbys aren't standbys until switched in (we do this to databases regularly.) Netflix apparently has a chaos generator that randomly kills machines as a standard process. If you're supposed to deal with failure, make sure you're dealing with failure regularly!
      - Kadin 15y ago
        > Netflix apparently has a chaos generator that randomly kills machines as a standard process. This sounds pretty neat, but a quick Google didn't turn up any information about it besides this post. Do you know of anywhere to get more information on what they're doing? It sounds like a sensible idea, although I can only imagine trying to implement it would be ... challenging, for most companies/organizations.
        dreww 15y ago
        check out item no 3. on this list, which is AWS lessons-learned: http://techblog.netflix.com/2010/12/5-lessons-weve-learned-using-aws.html http://techblog.netflix.com/2010/12/5-lessons-weve-learned-u... see also: http://techblog.netflix.com/2011/04/lessons-netflix-learned-from-aws-outage.html http://techblog.netflix.com/2011/04/lessons-netflix-learned-... and: http://techblog.netflix.com/2011/07/netflix-simian-army.html http://techblog.netflix.com/2011/07/netflix-simian-army.html for the other simian themed services they've developed for care and feeding of their AWS stuff.
- cperciva 15y ago
  I wonder if it is prohibitively expensive to do a real life system test on a big data center It's probably prohibitively dangerous. Backup power systems don't have many-nines of reliability; generators which are reliable enough for the once-a-decade event when a car crash knocks out your utility power aren't anywhere near the reliability needed to run your datacentre for an hour every month as a test.
  - ams6110 15y ago
    On the other hand, just as with database backups, making them is only half of the story. You have to test restores/recovery. Does your plan actually work? What have you overlooked? What edge cases do you need to accommodate? Many data centers will test backup power generation regularly just for this reason. It's not unheard of at all and the risk of a problem at a planned time is worth the confidence in knowing that the system is more likely to work when needed at an unexpected time.
  - emaste 15y ago
    Actually, if you don't run test your generator regularly it's very unlikely to work when you do need it. Here's a doc from cummins, a generator mfgr: http://www.cumminspower.com/www/literature/technicalpapers/PT-7004-Maintenance-en.pdf http://www.cumminspower.com/www/literature/technicalpapers/P... It claims that the generator should be run for 30 minutes every month, loaded to at least one third of the rated capacity. So testing every month is exactly what you want to do.
    - rdl 15y ago
      Right, but the thing you don't test is the transfer switch/sync gear. Powering up the generator and dumping the output as heat weekly is pretty standard practice.
      - ams6110 15y ago
        Also don't forget to check the fuel tanks. With the rise in fuel prices the past couple of years, theft of diesel from backup generators has become more common.
spartango 15y ago
This is in the pipe.

mtkd 15y ago

It's a good communication from Amazon - maybe a little too long - could use a summary block at top. The compensation looks generous too.

grourk 15y ago
There's a summary block at the bottom -- but it's not a summary.

saturn 15y ago

As someone who has put a considerable amount of resources moving things into cloud computing - I wanted to believe. But I have changed my mind. Cloud computing scales the efficiencies, yes. It also scales the problems. And because of this, AWS is by several orders of magnitude the worst of my current hosts. I have dedicated servers. No downtime in past year. I have a couple of cloud servers with rackspace. No downtime (although i don't recommend them). I have some VPSes with local providers. No downtime. AWS? More than 24hrs downtime in the last year. Seriously, for someone trying to run web sites reliably - screw that. I'm not using AWS any more. And don't even get me started on the apologists. "EBS slow as treacle? Well you should have been running a multi zone raid-20 redundant array! Duh!". "EC2 instances dying at random? Well you should architect and implement a multi-master failover intelligent grid!" I used to be under some kind of crazy delusional spell that the above was correct and it was somehow my fault that I wasn't correctly adapting to AWS's numerous failings. Well, no more. Now I realise that I should just stick with the super reliable service I know and love from traditional operators. You need to programmatically grow and shrink your app server flock? Great, use AWS. For the other 99.999% of us - stick with what you were using before.

quanticle 15y ago
Is there any reason in particular why you wouldn't recommend Rackspace?
- saturn 15y ago
  Hm. Well, I don't like them. It's subjective, you might disagree. But off the top of my head: 1. Contracts. They want 1 years minimum contracts for any dedicated servers. For truly gargantuan orders I could understand this but for one puny server? Never. 2. Their definition of "cloud" is different from mine. To use their "cloud" services your servers need to be public facing, ie on public IPs. Want them on your own VPN? You can stil get the cloud prices but not the API. you create and cancel servers via tickets. This is different from VPS how? 3. Sloooow provisioning - even if you are able to use their "public cloud" API to provision a server - prepare to wait hours for it to be done, leading me to suspect it does nothing more than email a tech to provision a VPS and hook it up somehow. Oh, you can't pause them to save money either, again making me think these "cloud" servers are nothing more than slicehosts with an extra layer of abstraction Is that enough? I could go on.

rdl 15y ago

The thing I wonder about is wtf they didn't manually switch to generator when their automatic controls failed. They had presumably ~5 minutes of UPS; it took them 40 minutes to do this. This probably isn't directly Amazon's fault, but whatever contract datacenter they are using in Europe (probably a PTT, or possibly an international carrier; really curious what facility) I'm wary of using >1 generators to back up loads, thus requiring sync on generators for backup anyway -- much more comfortable with splitting the load up by room and having one generator per, with some kind of switch to allow for pulling generators out for maintenance. This pretty much limits you to 2-3MW per room (the largest economical diesel gensets), but that's not horrible. Really high reliability sites actually run onsite generation as PRIMARY (since it's less reliable to start), and then utility as backup. With the right onsite generation equipment, it can be cheaper/more efficient than the grid, too (by using combined cycle; use heat output to run cooling directly). Still, the 365 Main power outages take the cake; they used rotational UPSes (generators with huge flywheels) which had software bugs such that if input power got turned off and on several times (a common utility failure mode), the unit shut itself off entirely. Doh.

jwatte 15y ago
They explained that a ground fault prevented generators from delivering power. Manual start doesn't help in that case.
- rdl 15y ago
  From what I read, they said ground fault confused their PLCs (synchro gear for paralleling multiple generators). This shouldn't affect the generator (engine, generator) outputting power. Electronics are much more sensitive to ground faults, etc. than mechanical and electrical devices. A big manual transfer switch (as backup), which is presumably what they ended up using, is fairly bulletproof.

jwatte 15y ago

I thought best practice backup power was to use large flywheels for re-generation, and spin up diesel engines to power the wheel in the event of a loss. That way, there is no phase synchronization issue, just a mechanical clutch. Seems like this outage could have been prevented with better gear?

o1iver 15y ago

There seems to be a pretty simple solution to these problems: diversification. Like most things in life, putting all your eggs in one basket it not the right choice. The people who use only AWS or only RackSpace or only 1&1 are equally wrong. What you have to do it diversify. Run a production site ghost on some other platform (software/hardware bugs, ...), run by some other provider (bankruptcy, theft, ...) in another country (power cuts, earthquake, ...). As soon as the primary goes down you switch on the secondary. The probability of a total blackout is then squared: 10^-3*10^-3=10^-6. The great thing with these "cloud" platforms is that your secondary system can even "go to sleep" saving you money and then spin up instances as soon as the primary goes down. This is by the way how banks, airport-systems and probably the NSA do it!

marcamillion 15y ago

It seems to me that Amazon Web Services will never truly be VERY stable. Not because I am being cynical, but just based on the nature of what they are doing. They are the biggest provider of large scale cloud-based computing services. They are pushing the boundaries. They are bound to always come upon problems that no one has ever seen before (including themselves) just based on the very nature of their business. So if you are looking for 'rock-solid reliability', maybe it is better to wait for another big company (Google, Apple, etc.) to come behind and fix all the mistakes that Amazon made the first time. That being said, I use AWS and I love it. Granted, I don't use EBS (not directly, via Heroku) and yes I have encountered downtime recently, it's not that big of a deal. I know they aren't messing around, and they are in uncharted territory. I can't reasonably expect them to have the best uptime for a platform that no one has ever built or done before, on the first time around the block. That's very unreasonable. That being said, I will continue using them from now until I outgrow them or the economics becomes painful, because the value I get with paying for what I use far outweighs 24 - 48 hours of downtime per year.

gfodor 15y ago

For all those complaining about AWS I think it's important to not fall into the trap of throwing all of Amazon's services into the same bucket. EBS (and hence, RDS) have shown time and time again to be the most complex offerings and more prone to failure. Generally speaking, at least for now, the parts of your system built on top of EBS should be carefully architected to survive in the face of erratic EBS latency, data corruption, or even downtime. (All of which are part of the standard AWS contract, but happen much more often in practice than if you are used to the mean failure time of a hard disk sitting in a cage.) This pattern leads me to believe that services such as VoltDB that do not directly rely upon attached storage will prove to be the paradigm necessary to get reliable cloud computing, at least in the AWS ecosystem. On-demand provisioning of disk is an extraordinary hard problem, and a world where local ephemeral storage provides durability through redundancy across nodes and AZ's is probably where we are headed.

robryan 15y ago

This seems to share a lot of parallels with the last big outage in terms of the API request overload and the EBS replication. Seems like the system need to be able to tell a bit better between a node going down and require a remirror and most of an availability zone going down.

Kartick 15y ago

Isn't the real problem attempting to build an abstraction that lets programmers assume the datacenter won't go down? Aren't you better off building a system like Google's Megastore/BigTable that assumes datacenters go down, and handling it in software? The obvious downside is, of course, that you'll have to rewrite all your software, but doesn't that give higher reliability at the end of the day? Disclosure: I work for Google, but on stuff totally disconnected with any of this.