4 ms·
> The advantage of having three regions is that any one region can fail, and the other regions will step in handle all the members in the failed region. At a p
by curun1r 9y ago
> The advantage of having three regions is that any one region can fail, and the other regions will step in handle all the members in the failed region.
At a previous employer, we met with AWS engineers who told us that this sort of thinking wasn't correct. Availability zones are for failure tolerance. Regions are conceptually different and for things like moving compute closer to clients, mitigating political instability or ensuring that data is held in a certain jurisdiction. To back up that claim, they told us that amazon.com ran out of a single region, us-west-2.
As an aside, another interesting point from that meeting was that autoscaling wasn't a good strategy for scaling up in the event of the loss of an availability zone. The reason why is that in those scenarios, a lot of other customers will be trying to fail over as well and autoscaling can get very slow. Instead of autoscaling, they recommended that we run 150% of our per-AZ capacity in 3 AZs so that if an AZ went offline, the remaining two would already have the necessary capacity. We still used autoscaling, but it was for responding to spikes and troughs in our own load, not failure scenarios.
I'm curious whether these two observations are correct since it seems like a lot of people view regions and autoscaling through a failure-recovery lens.
- aaronblohowiak 9y agoSometimes there have been region-wide issues. Running cold has its own set of problems. We do not rely exclusively on auto scaling when we do failover. Disclaimer: while I work at Netflix, these views are not necessarily those of my employer
- mimerme 9y agoI think this is another case of "your not really supposed to...but it works ¯\_(ツ)_/¯" For your first point, I think it depends on your usage scenario. With a big company that requires good performance to satisfy their users, like Netflix, having multiple regions is a given. But what the article probably missed is that they also most likely use AZs within the same region to mitigate failures, but since they have multiple regions at hand why not use them to mitigate failures as well? And their monthly tests are a simulation of a worst case scenario if an entire region goes down. As for autoscaling I also think that people think of it as a magic solution. More users? Autoscale. Zone down? Autoscale. But the problem arises when they think that it's instant because it isn't. Things have to happen behind the scenes, but for many people, they only care about what happens on stage, so for them configuring autoscale is just better because it's easier than load balancing on 3 different AZs.
- HighPlainsDrftr 9y agoJust my $0.02. The Netflix client/SDK was configured to contact 3 different regions. I'm guessing somehow, on the backend, it was replicating the state of your request to the other three regions. If one regions were to fail, the client would query the other two regions and regain its state. Having 3 regions available - the other two should be able to pick up the load of losing one without hitting capacity.
- jonny_eh 9y agoIIRC Netflix went down in 2012 when all of us-east went offline. I bet they now have multi-region followers to mitigate this scenario.