6 ms·

This would be a great time to post a guide to architecting systems for failover using AWS. Anyone got a great guide?

by benwerd 14y ago

petejansson 14y ago
The reference architectures and white papers here are helpful: http://aws.amazon.com/architecture/ http://aws.amazon.com/architecture/
caseysoftware 14y ago
We (Twilio) have released a number of articles & presentations in this area: http://www.slideshare.net/twilio/highavailability-infrastructure-in-the-cloud-evan-cooke-web-20-expo-nyc-2011 http://www.slideshare.net/twilio/highavailability-infrastruc... http://www.twilio.com/engineering/2011/04/22/why-twilio-wasnt-affected-by-todays-aws-issues/ http://www.twilio.com/engineering/2011/04/22/why-twilio-wasn... It's strategy as opposed to how-to but the principles apply.
- caseysoftware 14y ago
  And Netflix's Simian Army is awesome: http://techblog.netflix.com/2011/07/netflix-simian-army.html http://techblog.netflix.com/2011/07/netflix-simian-army.html They've even released the "Chaos Monkey" open source: http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html http://techblog.netflix.com/2012/07/chaos-monkey-released-in...
  - wyck 14y ago
    Yet netflix is currently down, so maybe there is a problem with the chimp army?
    - bermanoid 14y ago
      The problem is likely the same as usual: if the damn control plane is down, it doesn't matter how robust your failover architecture is, because your requests to bring up new machines go unanswered. There's pretty much no way to architect around that one as an AWS user (apart from going fully multi-cloud, but "nobody" actually does that, at least at scale), and I'm kind of shocked that those bits of AWS are still not robust against "single AZ outages", given that they're involved in pretty much every one of these incidents and make them affect people on the entire cloud...
      - koide 14y ago
        > apart from going fully multi-cloud, but "nobody" actually does that, at least at scale Pirate Bay might disagree with that sentence: http://torrentfreak.com/pirate-bay-moves-to-the-cloud-becomes-raid-proof-121017/ http://torrentfreak.com/pirate-bay-moves-to-the-cloud-become...
      - taligent 14y ago
        Apparently iCloud is multi cloud (AWS and Azure). But regardless it's not like all of EC2 went down just one or two AZs. So why couldn't traffic be migrated transprently to other AZs/regions ?
    - nathannecro 14y ago
      Netflix not down for me here (Eastern US).
  - caseysoftware 14y ago
    I'm watching Netflix from Austin, so it's not entirely down in the US.
- j45 14y ago
  Sweet, thanks for sharing!
deleted 14y ago
[deleted]
deleted 14y ago
[deleted]
hkarthik 14y ago
I wish there was. Architecting a system for failover requires a mindset change, a culture change within your org, and the right technology that lets you build for it while not slowing yourself down too much. The last point being the hardest. I would argue that none of the common full stack frameworks that startups use are fault tolerant enough for AWS. Most of them have multiple failure points that can quickly bring down entire apps.
jedberg 14y ago
We (Netflix) have done a bunch of presentations on it which are on our slideshare page and across the internet. After this issue is over I can give a longer answer. In short, we've just evacuated the affected zone and are mostly recovered.
- klapinat0r 14y ago
  Netflix recently launched here, and I've been unaffected so far. And +1 for the slideshare page. Their techblog is also worth following: http://techblog.netflix.com/ http://techblog.netflix.com/
- typicalrunt 14y ago
  I'm guessing your Chaos Gorilla helped to harden your architecture against this threat. Since you've mostly recovered, how did your system do? Are there side-cases that Chaos Gorilla didn't touch?
  - ohashi 14y ago
    EDIT: I WAS WRONG. Chaos Monkey and Chaos Gorilla both exist and simulate different forms of chaos.
    - philwelch 14y ago
      Monkey takes down single hosts; Gorilla simulates the loss of an entire AZ.
    - onethumb 14y ago
      Chaos Monkey takes down instances and such. Chaos Gorilla takes down entire AZs. :)
    - zackzackzack 14y ago
      "Create More Failures Currently, Netflix uses a service called "Chaos Monkey" to simulate service failure. Basically, Chaos Monkey is a service that kills other services. We run this service because we want engineering teams to be used to a constant level of failure in the cloud. Services should automatically recover without any manual intervention. We don't however, simulate what happens when an entire AZ goes down and therefore we haven't engineered our systems to automatically deal with those sorts of failures. Internally we are having discussions about doing that and people are already starting to call this service "Chaos Gorilla"." http://techblog.netflix.com/2011_04_01_archive.html http://techblog.netflix.com/2011_04_01_archive.html
ultrasaurus 14y ago
PagerDuty's CTO gave a talk that mentions it: http://blog.pagerduty.com/2012/10/ensuring-the-call-goes-out-every-time/ http://blog.pagerduty.com/2012/10/ensuring-the-call-goes-out...
- pla3rhat3r 14y ago
  +1

This would be a great time to post a guide to architecting systems for failover using AWS. Anyone got a great guide?

by benwerd 14y ago

This would be a great time to post a guide to architecting systems for failover using AWS. Anyone got a great guide?

The reference architectures and white papers here are helpful: http://aws.amazon.com/architecture/ http://aws.amazon.com/architecture/

caseysoftware 14y ago

We (Twilio) have released a number of articles & presentations in this area: http://www.slideshare.net/twilio/highavailability-infrastructure-in-the-cloud-evan-cooke-web-20-expo-nyc-2011 http://www.slideshare.net/twilio/highavailability-infrastruc... http://www.twilio.com/engineering/2011/04/22/why-twilio-wasnt-affected-by-todays-aws-issues/ http://www.twilio.com/engineering/2011/04/22/why-twilio-wasn... It's strategy as opposed to how-to but the principles apply.

caseysoftware 14y ago
And Netflix's Simian Army is awesome: http://techblog.netflix.com/2011/07/netflix-simian-army.html http://techblog.netflix.com/2011/07/netflix-simian-army.html They've even released the "Chaos Monkey" open source: http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html http://techblog.netflix.com/2012/07/chaos-monkey-released-in...
- wyck 14y ago
  Yet netflix is currently down, so maybe there is a problem with the chimp army?
  - bermanoid 14y ago
    The problem is likely the same as usual: if the damn control plane is down, it doesn't matter how robust your failover architecture is, because your requests to bring up new machines go unanswered. There's pretty much no way to architect around that one as an AWS user (apart from going fully multi-cloud, but "nobody" actually does that, at least at scale), and I'm kind of shocked that those bits of AWS are still not robust against "single AZ outages", given that they're involved in pretty much every one of these incidents and make them affect people on the entire cloud...
    - koide 14y ago
      > apart from going fully multi-cloud, but "nobody" actually does that, at least at scale Pirate Bay might disagree with that sentence: http://torrentfreak.com/pirate-bay-moves-to-the-cloud-becomes-raid-proof-121017/ http://torrentfreak.com/pirate-bay-moves-to-the-cloud-become...
    - taligent 14y ago
      Apparently iCloud is multi cloud (AWS and Azure). But regardless it's not like all of EC2 went down just one or two AZs. So why couldn't traffic be migrated transprently to other AZs/regions ?
  - nathannecro 14y ago
    Netflix not down for me here (Eastern US).
- caseysoftware 14y ago
  I'm watching Netflix from Austin, so it's not entirely down in the US.
j45 14y ago
Sweet, thanks for sharing!

deleted 14y ago

[deleted]

deleted 14y ago

[deleted]

hkarthik 14y ago

I wish there was. Architecting a system for failover requires a mindset change, a culture change within your org, and the right technology that lets you build for it while not slowing yourself down too much. The last point being the hardest. I would argue that none of the common full stack frameworks that startups use are fault tolerant enough for AWS. Most of them have multiple failure points that can quickly bring down entire apps.

jedberg 14y ago

We (Netflix) have done a bunch of presentations on it which are on our slideshare page and across the internet. After this issue is over I can give a longer answer. In short, we've just evacuated the affected zone and are mostly recovered.

klapinat0r 14y ago
Netflix recently launched here, and I've been unaffected so far. And +1 for the slideshare page. Their techblog is also worth following: http://techblog.netflix.com/ http://techblog.netflix.com/
typicalrunt 14y ago
I'm guessing your Chaos Gorilla helped to harden your architecture against this threat. Since you've mostly recovered, how did your system do? Are there side-cases that Chaos Gorilla didn't touch?
- ohashi 14y ago
  EDIT: I WAS WRONG. Chaos Monkey and Chaos Gorilla both exist and simulate different forms of chaos.
  - philwelch 14y ago
    Monkey takes down single hosts; Gorilla simulates the loss of an entire AZ.
  - onethumb 14y ago
    Chaos Monkey takes down instances and such. Chaos Gorilla takes down entire AZs. :)
  - zackzackzack 14y ago
    "Create More Failures Currently, Netflix uses a service called "Chaos Monkey" to simulate service failure. Basically, Chaos Monkey is a service that kills other services. We run this service because we want engineering teams to be used to a constant level of failure in the cloud. Services should automatically recover without any manual intervention. We don't however, simulate what happens when an entire AZ goes down and therefore we haven't engineered our systems to automatically deal with those sorts of failures. Internally we are having discussions about doing that and people are already starting to call this service "Chaos Gorilla"." http://techblog.netflix.com/2011_04_01_archive.html http://techblog.netflix.com/2011_04_01_archive.html

ultrasaurus 14y ago

PagerDuty's CTO gave a talk that mentions it: http://blog.pagerduty.com/2012/10/ensuring-the-call-goes-out-every-time/ http://blog.pagerduty.com/2012/10/ensuring-the-call-goes-out...

pla3rhat3r 14y ago
+1