4 ms·
This does seem to indicate a notable lack of isolation for the blast radius between DO datacenters. Would be interesting to see the post mortem.
by alexeldeib 7y ago
This does seem to indicate a notable lack of isolation for the blast radius between DO datacenters. Would be interesting to see the post mortem.
- protomyth 7y agoI get the feeling that whoever writes the post-mortem is going to have a bit of pressure to assure folks that there is isolation going forward.
- swsieber 7y agoWhoever broke it is going to feel significant pressure to actually isolate things too.
- klodolph 7y agoThat would be a bad sign that there’s something wrong with the culture. I would hope for a postmortem that identified flaws that genuinely needed to be fixed.
- viraptor 7y agoThose are not mutually exclusive and actually a good idea. You want to fix this specific issue, but also ensure that whatever process took down one DC doesn't affect other DCs. That's scaling and redundancy 101 - not sure why it would be something wrong.
- klodolph 7y ago> Those are not mutually exclusive and actually a good idea. The goals “assuring folks that there is isolation” and “identifying flaws that need to be fixed” are somewhat contrary to each other. The post-mortem should identify flaws in systems, processes, and thinking. It should not try to assure people that there is isolation when there is evidence to the contrary. > You want to fix this specific issue, but also ensure that whatever process took down one DC doesn't affect other DCs. This was a multi-regional failure. So, this specific issue is also an isolation problem, among other things. You will want to ensure that this problem doesn’t happen again but you shouldn’t assure that it won’t.
- protomyth 7y agoI would think having all zones go down is a flaw that genuinely needed to be fixed.
- klodolph 7y agoThat’s not the flaw, that’s the outcome. The purpose of a post-mortem is to identify the flaws that caused that outcome, and ways to fix those flaws.
- notyourday 7y agoPropagating mistakes across all the things is devops
- jbarham 7y agoDevOps Borat: "To make error is human. To propagate error to all server in automatic way is #devops." https://twitter.com/DEVOPS_BORAT/status/41587168870797312 https://twitter.com/DEVOPS_BORAT/status/41587168870797312
- nodesocket 7y agoGoogle Cloud recently had a global outdate. DevOps tools that interact with all resources across data centers are primarily the culprit.
- sterlind 7y agoGoogle's RCA is here, I believe: https://status.cloud.google.com/incident/cloud-networking/19009 https://status.cloud.google.com/incident/cloud-networking/19... Reading between the lines, it looks like their maintenance system needed to take down several Borg clusters within a single AZ, and their BGP route reflectors all ran from the same set of logical clusters. They'd tried to set up geo-redundancy by having different BGP speakers across different AZs, but they were all parented by the same set of logical clusters, and the maintenance engine descheduled all of them together. Then the network ran okay ("designed to fail static for short periods of time") until the routes expired, after which routes got withdrawn and traffic blackholed. They realized the issue within an hour.. unfortunately, since they took down multiple replicas of their network control plane, they lost Paxos primary and had to rebuild configuration. (Disclaimer: I work in Azure, I just find it fascinating to look at Google's RCAs because failure provides an insight into their architecture and risk engineering.)
- temikus 7y agoOr just bad rollout procedures.
- alexeldeib 7y agopotato, potato ;)