4 ms·
I may have read it incorrectly myself, but I interpreted this as meaning the control plane was balanced across availability zones in order to provide durability
by jsdalton 15y ago
I may have read it incorrectly myself, but I interpreted this as meaning the control plane was balanced across availability zones in order to provide durability in the face of a failure of one of the zones. In other words, Amazon is ensured their control plane is operational at all times.
The API failures were ultimately tied to the network problems that occurred, not to a failure of the control plane.
EDIT: I should finish reading before I reply. :) It would appear that the network issue in the one availability zone was so severe that the control plane ran out of threads to service API requests to any of the availability zones.
So while it's true the underlying problem was a network issue, the fact that the the control plane is spread across availability zones was responsible for part of the outage that occurred across the whole region.
My totally unqualified assessment of this aspect of the outage is that, while it might make sense to have a control plane spread across availability zones, they presumably need to have isolated control planes for each zone, instead of a shared plane as they seemingly have now.
- evangineer 15y agoThey seem to have settled on a halfway house, pushing more of the control plane functionality down into the EBS clusters and making the remaining shared control plane more robust to the sort of failures that arose this time.