3 ms·

I wonder how much people in the know really believe in singling out a single root cause to these HA system failures.

nailer 4y ago
It’s really odd to see comments like this faded out from downvotes. Anyone from the devops or SRE or distributed systems would ask the same. For example why are there no processes to check for snapshot integrity or if they are, why were they not used?
- pgwhalen 4y ago
  It’s downvoted because it’s a straw man, the linked article isn’t suggesting there was a single root cause.
verdverm 4y ago
Did you read it? They pointed out several issues that caused this and several mitigations to prevent it from happening again. I have yet to see anyone else with public RCAs as good as Google Cloud
- fulafel 4y ago
  Yes - under the root cause heading isn't the following raised as "the" root cause? > During a routine update to the critical elements snapshot data, an incomplete snapshot was inadvertently shared which removed several sites from the topology map.
  - verdverm 4y ago
    Yes, that is one sentence in the full analysis which describes the core of what happened. There are other sentences which describe several contributing factors.
  - sillybov3456 4y ago
    Right, the thing about root causes is that you can always keep digging. For instance why was an incomplete snapshot shared? And then the why for that why, and on and on until you reach the singularity at the beginning of the universe, which can logically be the only real root cause of anything. Root cause just means whatever is enough to make your boss satisfied.
notimetorelax 4y ago
Look at the ‘Remediation and Prevention’ section for the fixes. Root cause of an incident is always one, but means to prevent it are multiple.
- fulafel 4y ago
  Right, but is it the right mental model to demand a root cause to be delivered as the outcome to the investigation, when there are lots of things going wrong and process & architecture problems that make the involved failures fatal.
  - notimetorelax 4y ago
    This follows Google’s published post mortem template: https://sre.google/sre-book/example-postmortem https://sre.google/sre-book/example-postmortem What you’re saying can go into the lessons learned section.
    - fulafel 4y ago
      That template is a bit better wrt the cognitive problem in question since the heading is "root causes" (plural). So it doesnt' conflict epistemic idea that failure modes are caused by interacting events and processes. Graph vs tree etc.
toast0 4y ago
Well, I don’t think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. Done.
- FigmentEngine 4y ago
  tldr; no system failure is human error. if a human can cause this, then your system lacks adequate controls and mechanism. the root cause is the lack of controls, not the human error
  - toast0 4y ago
    Building a system without sufficient controls is a classic (human) error in system design.
- fulafel 4y ago
  I chuckled in agreement but then I started thinking if it could be actually a better term. Human error analysis might be a little blamey.. but availability design bug or something might be good.

It’s really odd to see comments like this faded out from downvotes. Anyone from the devops or SRE or distributed systems would ask the same. For example why are there no processes to check for snapshot integrity or if they are, why were they not used?

pgwhalen 4y ago
It’s downvoted because it’s a straw man, the linked article isn’t suggesting there was a single root cause.

verdverm 4y ago

Did you read it? They pointed out several issues that caused this and several mitigations to prevent it from happening again. I have yet to see anyone else with public RCAs as good as Google Cloud

fulafel 4y ago
Yes - under the root cause heading isn't the following raised as "the" root cause? > During a routine update to the critical elements snapshot data, an incomplete snapshot was inadvertently shared which removed several sites from the topology map.
- verdverm 4y ago
  Yes, that is one sentence in the full analysis which describes the core of what happened. There are other sentences which describe several contributing factors.
- sillybov3456 4y ago
  Right, the thing about root causes is that you can always keep digging. For instance why was an incomplete snapshot shared? And then the why for that why, and on and on until you reach the singularity at the beginning of the universe, which can logically be the only real root cause of anything. Root cause just means whatever is enough to make your boss satisfied.

notimetorelax 4y ago

Look at the ‘Remediation and Prevention’ section for the fixes. Root cause of an incident is always one, but means to prevent it are multiple.

fulafel 4y ago
Right, but is it the right mental model to demand a root cause to be delivered as the outcome to the investigation, when there are lots of things going wrong and process & architecture problems that make the involved failures fatal.
- notimetorelax 4y ago
  This follows Google’s published post mortem template: https://sre.google/sre-book/example-postmortem https://sre.google/sre-book/example-postmortem What you’re saying can go into the lessons learned section.
  - fulafel 4y ago
    That template is a bit better wrt the cognitive problem in question since the heading is "root causes" (plural). So it doesnt' conflict epistemic idea that failure modes are caused by interacting events and processes. Graph vs tree etc.

toast0 4y ago

Well, I don’t think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. Done.

FigmentEngine 4y ago
tldr; no system failure is human error. if a human can cause this, then your system lacks adequate controls and mechanism. the root cause is the lack of controls, not the human error
- toast0 4y ago
  Building a system without sufficient controls is a classic (human) error in system design.
fulafel 4y ago
I chuckled in agreement but then I started thinking if it could be actually a better term. Human error analysis might be a little blamey.. but availability design bug or something might be good.