3 ms·
We (Netflix) have done a bunch of presentations on it which are on our slideshare page and across the internet. After this issue is over I can give a longer an
by jedberg 14y ago
We (Netflix) have done a bunch of presentations on it which are on our slideshare page and across the internet.
After this issue is over I can give a longer answer. In short, we've just evacuated the affected zone and are mostly recovered.
- klapinat0r 14y agoNetflix recently launched here, and I've been unaffected so far. And +1 for the slideshare page. Their techblog is also worth following: http://techblog.netflix.com/ http://techblog.netflix.com/
- typicalrunt 14y agoI'm guessing your Chaos Gorilla helped to harden your architecture against this threat. Since you've mostly recovered, how did your system do? Are there side-cases that Chaos Gorilla didn't touch?
- ohashi 14y agoEDIT: I WAS WRONG. Chaos Monkey and Chaos Gorilla both exist and simulate different forms of chaos.
- philwelch 14y agoMonkey takes down single hosts; Gorilla simulates the loss of an entire AZ.
- onethumb 14y agoChaos Monkey takes down instances and such. Chaos Gorilla takes down entire AZs. :)
- zackzackzack 14y ago"Create More Failures Currently, Netflix uses a service called "Chaos Monkey" to simulate service failure. Basically, Chaos Monkey is a service that kills other services. We run this service because we want engineering teams to be used to a constant level of failure in the cloud. Services should automatically recover without any manual intervention. We don't however, simulate what happens when an entire AZ goes down and therefore we haven't engineered our systems to automatically deal with those sorts of failures. Internally we are having discussions about doing that and people are already starting to call this service "Chaos Gorilla"." http://techblog.netflix.com/2011_04_01_archive.html http://techblog.netflix.com/2011_04_01_archive.html
- ohashi 14y agoMy apologies! I was wrong.
- benwerd 14y agoI'd love to hear that longer answer, fwiw!