6 ms·

Root cause of failure, root cause of success

76 points by henrik_w 5y ago

mappu 5y ago
Happy families are all alike; every unhappy family is unhappy in its own way. C functions have one return code for success, and a thousand for the different ways something can go wrong. You can very clearly root cause a failure. In the reverse case you can identify aspects of the success whose absence would have caused a greater chance of failure, but there are still many more ways for a project to go wrong than there are for it to go right.
- di4na 5y ago
  [Citation needed] that the way it can go wrong are more and that they are different. The research community on complex system safety disagree with you here.
xyzelement 5y ago
> Or maybe several people are sluggish today because they’re sick. ... When the boulder falls, it means that the collection of processes weren’t able to compensate for the disturbance. But there’s no single problem, no root cause, that you can point to. Perhaps I agree with the idea in an academic way, but in practice seeking the deepest root cause possible is valuable. OK - the boulder fell because enough people were sick that day. How did we end up with a staffing policy that allowed this? OH - because the person in charge of this is a junior manager How did a junior manager get to be in charge? OH - because our top leadership didn't realize how important this boulder is. Oh - so the real thing we need to do is educate top leadership on how important this mission is. Then they will find a good leader for it. He will then design a robust staffing policy. The point here is that this fixed other problems along the way - getting a better leader in place solves more than just this one instance of a problem. I get it, it's all recursive systems of balancing the boulder all the way down. But in practice you can make things exponentially better by digging for the deepest root and fixing there.
- di4na 5y ago
  [Citation needed] cause the academic research on complex system safety disagree with the fact that doing what you said make the system better
- kqr 5y ago
  But there is not "a deepest" root cause. It's a complex network of causality. By limiting yourself to one path of the network, and by restricting yourself to blaming the deepest politically acceptable thing to blame (this is what happens in practise) you're not fully exploring the network and you are bound to end up in a situation at least as expensive as a more thorough process that casts a wider net, e.g. a system-based causal analysis based on constraints and dynamic control structures. ---- If I had time, I'd type out the steps more explicitly, but important questions that an analysis process must raise: - Why is this rock here to begin with? Did it need to be before? Does it need to be now? - How far can we move it? - What do we do to salvage the situation when it tumbles down? - Why do we need people to hold it up? What other solutions were explored? Why were they rejected? Has the context changed? - Why isn't there a system that monitors how well the rock is held up? Why do we not have systems that warn about faltering assumptions, like cliff slippiness? - Can we make it cheaper to roll the rock back up? Etc. These questions come up when you dig wider, not deeper. Any process that doesn't produce (and ideally answer) these questions is suboptimal.
  - xyzelement 5y ago
    I agree with the questions you're asking, in reality I'd ask those too but wanted to constrain my post to make a point. That point is, again, the attempt to get to root cause (whether you go deep or wide) is valuable. The idea that you won't get the ONE TRUE ANSWER shouldn't stop that effort. An organization that landed at either your or my answer is better off than one that landed at neither.
    - kqr 5y ago
      With that clarification, I think we agree on content, but disagree on terminology. "Root cause" tends, in my experience, to make people think of a singular causal path (sometimes even called a chain) which just... stops, at some convenient place. And this is "the root cause". I strongly oppose that picture of failure.
jes 5y ago
I always stumble a little over the concept of a root cause. It feels subjective to me. What makes a cause a root cause?
- teddyh 5y ago
  Technically, there is no actual root cause, as you suspect. But what we are really looking for is something actionable – some “root” cause which we can affect in order to avoid getting the same problem in the future.
  - rramadass 5y ago
    >we are really looking for is something actionable Great viewpoint!
  - di4na 5y ago
    *in order to feel like we did something so we do not have to keep thinking about "will it happen again" every night when going to sleep. Remark that this does not mean we made it happen less. Just that we soothed the part of our brain that was scared.
- nenadg 5y ago
  Root cause doesn't have to be a single instance of recognizable failure. It can we wrapped variable X, that contains whatever non-deterministic event, process or physical instance. That's the best one can do in certain circumstances.
- inglor_cz 5y ago
  Some phrases fall apart when you analyze them too much. What is the root cause of, say, scurvy? Some millions of years ago, our monkey ancestors lost the ability to manufacture vitamin C in their own bodies. Since then, anyone who stops eating enough vitamin C faces scurvy. But this is not actionable, as another commenter says. We cannot yet fix our genomes to bring internal production of vitamin C back online. So we say that root cause of scurvy is lack of vitamin C in that person's diet, which can be fixed very easily now. If/when technologies like CRISPR mature enough that playing around with genes and their expression becomes a routine thing, we might change our genomes so that vitamin C gets produced internally again, thus shifting the perceived root cause.
gweinberg 5y ago
Really bad reasoning. A chain fails with one weak link. A chain doesn't succeed because of one strong link, they all have to be strong. There's nothing remotely like symmetry.
- rramadass 5y ago
  >A chain fails with one weak link. A chain doesn't succeed because of one strong link, they all have to be strong. There's nothing remotely like symmetry You absolutely nailed it! This is why when people ascribe Success to any one factor, i simply roll my eyes. Personal Characteristics/Motivation, Timing, Circumstances, Random Chance etc. all have a role. The relative proportion of their weights will vary for any given event but all are necessary. As a learning and adaptive organism we do have some control over some of the factors i.e. ability to change their relative weights but everyone of them still count and cannot be neglected. On the other hand, Failure is the general status-quo because it is so easy to be unaware/miss/neglect any single factor which might be essential (i.e. has a higher weightage) to achieve the goal. Looked at from the above pov, it becomes clear that Failure is unavoidable and not something to be afraid of but is an essential learning method, thus; trial and error methodology. The importance of personal initiative and self-effort also now becomes evident; it is the only way we have to control and tame the various factors to the extent possible in our favour.
- jimmySixDOF 5y ago
  But likewise, not all failures are necessarily equivalent either and a form of success is how you structure the links to allow for that.
- ummonk 5y ago
  There are still multiple causes of failure there though. Why was the chain manufactured with a weak link? Why wasn't the weak link discovered during periodic inspection / testing? Why wasn't there a second chain to add redundancy and back up the first?
  - rramadass 5y ago
    You have missed the point. It is not that there cannot be multiple points of failure; it is that it takes just one point of failure to bring the entire system down. As long as there are dependant and moving parts failure will be inherent in the system; it cannot be "inspected/tested" away. Adding redundancy does not remove failure it merely transforms/postpones it. Failure is the "natural" state; we intuitively understand this through the concept of entropy always increasing (or more colloquially; "everything is going to shit") unless we do something about it i.e. expend energy and do work.
deleted 5y ago
[deleted]
leroman 5y ago
After 20 years in the industry, it seems like success is silent, nobody really appreciates something that works well, by its nature it seems to make the “success” look easy. Root cause of success is not failing for all the possible failures.. :shrug:
- dgb23 5y ago
  This is why it seems so hard to sell higher quality software in some cases. The party paying might not appreciate the invisible robustness, might underestimate the cost of failure or overestimate the ability of an operator. Maybe there is a way to make those things more visible and clear?
  - leroman 5y ago
    To an extent it's your fault for taking the position and accepting the status-quo, my advise is to get hired where the manager has the same "standards" that you aspire to, if you assume you will be able to "raise the bar", you are not going to have a good time there. It's either already in the DNA of the company, or you will probably have to force it into your standards..
    - Radim 5y ago
      dgb23 makes an astute observation that the market (demand side) seems to undervalue SW robustness => writing robust SW has poor ROI => not a viable strategy, product will be outcompeted => company goes bust => DON'T DO IT. That's a gross simplification of course. But even simplified thought experiments can be useful. I'm puzzled by your reply though, about "having a good time" and "get a manager with high standards". How is that relevant to dbg23's point?
      - leroman 5y ago
        My observation is made from experiences, you are suggesting that if something is good for the market then that's where the market will go.. Unfortunately my experience is that decisions are made by individuals who have their own ORG politics to play out.. they don't really care about this "market".. > I'm puzzled by your reply though, about "having a good time" and "talking to a manager". How is that relevant to dbg23's point? I believe dbg23's point is coming from frustration of what needs to happen to get to this "success" as a software engineer ("to create higher quality software"), if one side believes it's good enough to have "meh quality software - but good sales team", I was sharing my thought on how to deal with this frustration..
venkat223 5y ago
The root cause may be many small nudges and later turns to the last observable event.The direction the system turns can be manipulated contingently to suit the capabilities of the causation Finally ego and ignorance and lack of long-term perspectives remain as causes.
127 5y ago
Find a product market fit. Figure out how to escape the competition enough time to make a profit. That's about it.
bgribble 5y ago
I read the article as a critique of “root cause of failure”, making the point that failures are always failures of systems, not individual elements. I agree with this basic premise… if a single fat-fingered SQL command can bring down a service, for a random example, it’s not the operator that is really at fault but the system that has the operator typing SQL directly at the database to do maintenance. However, unless you identify the specific causal pathway that led to the failure you can’t improve the systems! Root cause attribution strategies like “5 whys” do a pretty good job of teasing out the layers of a system failure to identify both the specific causal pathway and the systematic failures, if you keep it going long enough.
itsmejeff 5y ago
I really like thought exercises like this, but find it troublesome when authors (like this one) do not offer alternatives that address the weakness of the common approach, they just just point the weaknesses out and metaphorically “shrug”. Left with this, people will still revert to the status quo because they’ve been offered no true alternative that provides more value.

76 points by henrik_w 5y ago

Happy families are all alike; every unhappy family is unhappy in its own way. C functions have one return code for success, and a thousand for the different ways something can go wrong. You can very clearly root cause a failure. In the reverse case you can identify aspects of the success whose absence would have caused a greater chance of failure, but there are still many more ways for a project to go wrong than there are for it to go right.

di4na 5y ago
[Citation needed] that the way it can go wrong are more and that they are different. The research community on complex system safety disagree with you here.

xyzelement 5y ago

> Or maybe several people are sluggish today because they’re sick. ... When the boulder falls, it means that the collection of processes weren’t able to compensate for the disturbance. But there’s no single problem, no root cause, that you can point to. Perhaps I agree with the idea in an academic way, but in practice seeking the deepest root cause possible is valuable. OK - the boulder fell because enough people were sick that day. How did we end up with a staffing policy that allowed this? OH - because the person in charge of this is a junior manager How did a junior manager get to be in charge? OH - because our top leadership didn't realize how important this boulder is. Oh - so the real thing we need to do is educate top leadership on how important this mission is. Then they will find a good leader for it. He will then design a robust staffing policy. The point here is that this fixed other problems along the way - getting a better leader in place solves more than just this one instance of a problem. I get it, it's all recursive systems of balancing the boulder all the way down. But in practice you can make things exponentially better by digging for the deepest root and fixing there.

di4na 5y ago
[Citation needed] cause the academic research on complex system safety disagree with the fact that doing what you said make the system better
kqr 5y ago
But there is not "a deepest" root cause. It's a complex network of causality. By limiting yourself to one path of the network, and by restricting yourself to blaming the deepest politically acceptable thing to blame (this is what happens in practise) you're not fully exploring the network and you are bound to end up in a situation at least as expensive as a more thorough process that casts a wider net, e.g. a system-based causal analysis based on constraints and dynamic control structures. ---- If I had time, I'd type out the steps more explicitly, but important questions that an analysis process must raise: - Why is this rock here to begin with? Did it need to be before? Does it need to be now? - How far can we move it? - What do we do to salvage the situation when it tumbles down? - Why do we need people to hold it up? What other solutions were explored? Why were they rejected? Has the context changed? - Why isn't there a system that monitors how well the rock is held up? Why do we not have systems that warn about faltering assumptions, like cliff slippiness? - Can we make it cheaper to roll the rock back up? Etc. These questions come up when you dig wider, not deeper. Any process that doesn't produce (and ideally answer) these questions is suboptimal.
- xyzelement 5y ago
  I agree with the questions you're asking, in reality I'd ask those too but wanted to constrain my post to make a point. That point is, again, the attempt to get to root cause (whether you go deep or wide) is valuable. The idea that you won't get the ONE TRUE ANSWER shouldn't stop that effort. An organization that landed at either your or my answer is better off than one that landed at neither.
  - kqr 5y ago
    With that clarification, I think we agree on content, but disagree on terminology. "Root cause" tends, in my experience, to make people think of a singular causal path (sometimes even called a chain) which just... stops, at some convenient place. And this is "the root cause". I strongly oppose that picture of failure.

jes 5y ago

I always stumble a little over the concept of a root cause. It feels subjective to me. What makes a cause a root cause?

teddyh 5y ago
Technically, there is no actual root cause, as you suspect. But what we are really looking for is something actionable – some “root” cause which we can affect in order to avoid getting the same problem in the future.
- rramadass 5y ago
  >we are really looking for is something actionable Great viewpoint!
- di4na 5y ago
  *in order to feel like we did something so we do not have to keep thinking about "will it happen again" every night when going to sleep. Remark that this does not mean we made it happen less. Just that we soothed the part of our brain that was scared.
nenadg 5y ago
Root cause doesn't have to be a single instance of recognizable failure. It can we wrapped variable X, that contains whatever non-deterministic event, process or physical instance. That's the best one can do in certain circumstances.
inglor_cz 5y ago
Some phrases fall apart when you analyze them too much. What is the root cause of, say, scurvy? Some millions of years ago, our monkey ancestors lost the ability to manufacture vitamin C in their own bodies. Since then, anyone who stops eating enough vitamin C faces scurvy. But this is not actionable, as another commenter says. We cannot yet fix our genomes to bring internal production of vitamin C back online. So we say that root cause of scurvy is lack of vitamin C in that person's diet, which can be fixed very easily now. If/when technologies like CRISPR mature enough that playing around with genes and their expression becomes a routine thing, we might change our genomes so that vitamin C gets produced internally again, thus shifting the perceived root cause.

gweinberg 5y ago

Really bad reasoning. A chain fails with one weak link. A chain doesn't succeed because of one strong link, they all have to be strong. There's nothing remotely like symmetry.

rramadass 5y ago
>A chain fails with one weak link. A chain doesn't succeed because of one strong link, they all have to be strong. There's nothing remotely like symmetry You absolutely nailed it! This is why when people ascribe Success to any one factor, i simply roll my eyes. Personal Characteristics/Motivation, Timing, Circumstances, Random Chance etc. all have a role. The relative proportion of their weights will vary for any given event but all are necessary. As a learning and adaptive organism we do have some control over some of the factors i.e. ability to change their relative weights but everyone of them still count and cannot be neglected. On the other hand, Failure is the general status-quo because it is so easy to be unaware/miss/neglect any single factor which might be essential (i.e. has a higher weightage) to achieve the goal. Looked at from the above pov, it becomes clear that Failure is unavoidable and not something to be afraid of but is an essential learning method, thus; trial and error methodology. The importance of personal initiative and self-effort also now becomes evident; it is the only way we have to control and tame the various factors to the extent possible in our favour.
jimmySixDOF 5y ago
But likewise, not all failures are necessarily equivalent either and a form of success is how you structure the links to allow for that.
ummonk 5y ago
There are still multiple causes of failure there though. Why was the chain manufactured with a weak link? Why wasn't the weak link discovered during periodic inspection / testing? Why wasn't there a second chain to add redundancy and back up the first?
- rramadass 5y ago
  You have missed the point. It is not that there cannot be multiple points of failure; it is that it takes just one point of failure to bring the entire system down. As long as there are dependant and moving parts failure will be inherent in the system; it cannot be "inspected/tested" away. Adding redundancy does not remove failure it merely transforms/postpones it. Failure is the "natural" state; we intuitively understand this through the concept of entropy always increasing (or more colloquially; "everything is going to shit") unless we do something about it i.e. expend energy and do work.

deleted 5y ago

[deleted]

leroman 5y ago

After 20 years in the industry, it seems like success is silent, nobody really appreciates something that works well, by its nature it seems to make the “success” look easy. Root cause of success is not failing for all the possible failures.. :shrug:

dgb23 5y ago
This is why it seems so hard to sell higher quality software in some cases. The party paying might not appreciate the invisible robustness, might underestimate the cost of failure or overestimate the ability of an operator. Maybe there is a way to make those things more visible and clear?
- leroman 5y ago
  To an extent it's your fault for taking the position and accepting the status-quo, my advise is to get hired where the manager has the same "standards" that you aspire to, if you assume you will be able to "raise the bar", you are not going to have a good time there. It's either already in the DNA of the company, or you will probably have to force it into your standards..
  - Radim 5y ago
    dgb23 makes an astute observation that the market (demand side) seems to undervalue SW robustness => writing robust SW has poor ROI => not a viable strategy, product will be outcompeted => company goes bust => DON'T DO IT. That's a gross simplification of course. But even simplified thought experiments can be useful. I'm puzzled by your reply though, about "having a good time" and "get a manager with high standards". How is that relevant to dbg23's point?
    - leroman 5y ago
      My observation is made from experiences, you are suggesting that if something is good for the market then that's where the market will go.. Unfortunately my experience is that decisions are made by individuals who have their own ORG politics to play out.. they don't really care about this "market".. > I'm puzzled by your reply though, about "having a good time" and "talking to a manager". How is that relevant to dbg23's point? I believe dbg23's point is coming from frustration of what needs to happen to get to this "success" as a software engineer ("to create higher quality software"), if one side believes it's good enough to have "meh quality software - but good sales team", I was sharing my thought on how to deal with this frustration..

venkat223 5y ago

The root cause may be many small nudges and later turns to the last observable event.The direction the system turns can be manipulated contingently to suit the capabilities of the causation Finally ego and ignorance and lack of long-term perspectives remain as causes.

127 5y ago

Find a product market fit. Figure out how to escape the competition enough time to make a profit. That's about it.

bgribble 5y ago

I read the article as a critique of “root cause of failure”, making the point that failures are always failures of systems, not individual elements. I agree with this basic premise… if a single fat-fingered SQL command can bring down a service, for a random example, it’s not the operator that is really at fault but the system that has the operator typing SQL directly at the database to do maintenance. However, unless you identify the specific causal pathway that led to the failure you can’t improve the systems! Root cause attribution strategies like “5 whys” do a pretty good job of teasing out the layers of a system failure to identify both the specific causal pathway and the systematic failures, if you keep it going long enough.

itsmejeff 5y ago

I really like thought exercises like this, but find it troublesome when authors (like this one) do not offer alternatives that address the weakness of the common approach, they just just point the weaknesses out and metaphorically “shrug”. Left with this, people will still revert to the status quo because they’ve been offered no true alternative that provides more value.