10 ms·
Feb 27 2023 GCP Outage Incident Report
- numbsafari 4y agoMaybe the folks as Fly shouldn't feel so alone.
- fulafel 4y agoI wonder how much people in the know really believe in singling out a single root cause to these HA system failures.
- nailer 4y agoIt’s really odd to see comments like this faded out from downvotes. Anyone from the devops or SRE or distributed systems would ask the same. For example why are there no processes to check for snapshot integrity or if they are, why were they not used?
- pgwhalen 4y agoIt’s downvoted because it’s a straw man, the linked article isn’t suggesting there was a single root cause.
- verdverm 4y agoDid you read it? They pointed out several issues that caused this and several mitigations to prevent it from happening again. I have yet to see anyone else with public RCAs as good as Google Cloud
- fulafel 4y agoYes - under the root cause heading isn't the following raised as "the" root cause? > During a routine update to the critical elements snapshot data, an incomplete snapshot was inadvertently shared which removed several sites from the topology map.
- verdverm 4y agoYes, that is one sentence in the full analysis which describes the core of what happened. There are other sentences which describe several contributing factors.
- sillybov3456 4y agoRight, the thing about root causes is that you can always keep digging. For instance why was an incomplete snapshot shared? And then the why for that why, and on and on until you reach the singularity at the beginning of the universe, which can logically be the only real root cause of anything. Root cause just means whatever is enough to make your boss satisfied.
- notimetorelax 4y agoLook at the ‘Remediation and Prevention’ section for the fixes. Root cause of an incident is always one, but means to prevent it are multiple.
- fulafel 4y agoRight, but is it the right mental model to demand a root cause to be delivered as the outcome to the investigation, when there are lots of things going wrong and process & architecture problems that make the involved failures fatal.
- notimetorelax 4y agoThis follows Google’s published post mortem template: https://sre.google/sre-book/example-postmortem https://sre.google/sre-book/example-postmortem What you’re saying can go into the lessons learned section.
- fulafel 4y agoThat template is a bit better wrt the cognitive problem in question since the heading is "root causes" (plural). So it doesnt' conflict epistemic idea that failure modes are caused by interacting events and processes. Graph vs tree etc.
- toast0 4y agoWell, I don’t think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. Done.
- FigmentEngine 4y agotldr; no system failure is human error. if a human can cause this, then your system lacks adequate controls and mechanism. the root cause is the lack of controls, not the human error
- toast0 4y agoBuilding a system without sufficient controls is a classic (human) error in system design.
- fulafel 4y agoI chuckled in agreement but then I started thinking if it could be actually a better term. Human error analysis might be a little blamey.. but availability design bug or something might be good.
- ccooffee 4y ago> During a routine update to the critical elements snapshot data, an incomplete snapshot was inadvertently shared which removed several sites from the topology map. I wish this went into more detail about how an incomplete snapshot was created and how the incomplete snapshot was valid-enough to sort-of work. I'm supposing that whatever interchange format was in use does not have any "END" delimiters (e.g. closing quotes/braces), nor any checksumming to ensure the entire message was delivered. I'm mildly surprised that there wasn't a failsafe to prevent automatically replacing a currently-in-use snapshot with one that lacks many of the services. (Adding a "type 'yes, I mean this'" user interaction widget is my preferred approach to avoid this class of problem in admin interfaces.)
- bragr 4y ago>I'm supposing that whatever interchange format was in use does not have any "END" delimiters (e.g. closing quotes/braces), nor any checksumming to ensure the entire message was delivered. Those only ensure you get the whole message, not that the message makes sense.
- ccooffee 4y agoQuite true. I was assuming that the incomplete snapshot was a transmission error or storage error. It's quite possible that the bug was an error of omission inside the data itself (e.g. someone accidentally removed an important key-value mapping from the data generator).
- jgrahamc 4y agoOne of the first things I did at Cloudflare was change the format of a file that contained vital information about the mapping between IPs and zones (no longer used these days) so that it had something like: START <count> <sha1> . . . END because it was just slurped up line by line assuming EOF was good.
- sitkack 4y agoEvery mission critical file eventually gets version numbers and checksums.
- we_never_see_it 4y ago[flagged]
- nijave 4y agoCan a book store?
- pb7 4y agoAn ad company with some of the most reliable infrastructure on the planet since its inception? Yeah, I think it can.
- SteveNuts 4y agoThey already were an infrastructure company, they just realized they could sell it to outsiders (as Amazon had already very much proved).
- fdgsdfogijq 4y agoI always think about how impossible it will be for GCP to compete with AWS. The work culture at AWS has been brutal for a decade. High standards for work, and insane amounts of oncall/ops reduction. A burn and churn machine that has created AWS. Google is a laid back company with great technology, but not the culture to grind out every detail of getting cloud to really work. Microsoft is another story altogether, as they already have a ton of corporate relationships to bring clients.
- baggy_trough 4y agoWell personally, I find GCP highly reliable and easier to understand than AWS.
- abofh 4y agoHopefully you can fund their operations for the coming quarter then, linked in suggests even GCP SRE's are on the chopping block.
- zamnos 4y agoYeah. No one ever went wrong with hosting in US-east-1. Oh wait.
- ec109685 4y agoAWS is architected in a way that makes global failures harder, e.g. VPC’s only span a region.
- deleted 4y ago[deleted]
- _aaed 4y agoOxford comma as well, how controversial can it get
- londons_explore 4y agoAt googles scale and reliability target, I would hope they have multiple independent worldwide networks. Each network would have its own config plane and data plane. Changes would only be made to one at once. Perhaps even different teams managing them so that one rogue or hacked employee can't take down the whole lot. Then, if someone screws up and pushes a totally nuts config, it will only impact one network. User traffic would flow just fine over the other networks. Obviously there would need to be thought on which network data will be routed over, failover logic between one and another, etc. And that failover logic would all be site specific, and rolled out site by site, so there is again so single point of global failure.
- sillybov3456 4y agoThat isn't really how production networks work in my uneducated opinion. If they are connected to the production network then they are the production network, and the level of isolation required to make that not the case would be so extreme as to make things potentially more unreliable. Others can correct me if I'm wrong about this. All I know is that the production network where I work is not air gapped in the way that would be required to truthfully consider testing networks a non production environment, so non prod changes typically wind up in front of the change review board anyway. Ask your own sites network engineers and see if they have similar constraints because I would be interested to hear more perspectives on that. One other thing I will say is that the abstractions of "config plane" and "data plane" and "control plane" don't really exist on real physical systems. That is mostly an abstraction created for applications people, those systems are not going to be totally blocked from interacting with eachother, they kind of have to. So if any of your "planes" are shared with production it is a production environment.
- rkeene2 4y agoThat would mean that all networks which peer with the Internet would necessarily be considered Production. This isn't that reasonable outside certain niches (i.e., national government networks). Instead, what's commonly done is to provide a Controlled Interface (to borrow a term from those national government networks) that gates which things are at which level of trust. This is where security boundaries are enforced -- and if they are sound security boundaries things on either side can't reasonably damage the other side.
- dboreham 4y ago"Automated clever thing wasn't as clever as it needed to be"
- 7a1c9427 4y ago> Google's automation systems mitigated this failure by pushing a complete topology snapshot during the next programming cycle. The proper sites were restored and the network converged by 05:05 US/Pacific. I think this is the most understated part of the whole report. The bad thing happened due to "automated clever thing" and then the system "automagically" mitigated it in ~7 minutes. Likely before a human had even figured out what had gone wrong.
- Jensson 4y agoHow would you otherwise do it? Anything that automatically pushes updates should monitor for rapid increase in errors afterwards and roll back if so. You should do at least that if you are working on a critical system.
- metadat 4y agoSure, in an ideal world this is how nearly everything would work. Getting a complex system to a level of maturity where this is feasible to do at scale in real life and actually work well is a respectable and non-trivial achievement. I don't know if Amazon or Azure are able to confidently and effectively put in such automatic remediation measures globally. My sense is there are humans involved to triage and fix unusual types of outages at every other cloud provider, including the other bigs. Leaving a comment on a message board saying how things ought to work is one thing (there's nothing wrong with your comment, I like it!); I only want to highlight, bold, and underscore how successfully achieving this level of automatic remediation atop a large and dynamic system is uncommon and noteworthy.
- jiggawatts 4y agoBased on the public RCAs of outages that occurred at Azure, it seems that their recovery processes are largely manual, or manually triggered.
- bakul 4y agoTo update a critical file atomically we used to first create an updated copy on the same filesystem and then rename it. Is this not possible on GCP?
- kristjansson 4y agoScaling anything with filesystem-like semantics to GCPs size is … pretty hard.
- bakul 4y agoMay be not with filesystem semantics but I would think providing atomic updates would be a very useful thing.
- ehPReth 4y agoI wish literally everywhere had (mandated?) detailed public RFOs like this. My residential ISP down for 20 minutes? Tell me more about the cable headend or your bad firmware push, please!
- LostLetterbox2 4y agoAnyone want to share what a programming cycle is? (The complete snapshot was restored with the next programming cycle)
- jvolkman 4y ago"Programming" here seems to refer to configuring routing/topology stuff on the network. So the "programming cycle" is the loop of computing and deploying that configuration.
- namok 4y ago[dead]
- dominic6676 4y ago[dead]