4 ms·

Many years ago, I worked at Amazon, and it was at the time quite fond of the "five whys" approach to root cause analysis: say what happened, ask why that happen

Many years ago, I worked at Amazon, and it was at the time quite fond of the "five whys" approach to root cause analysis: say what happened, ask why that happened, ask why that in turn happened, and keep going until you get to some very fundamental problem. I was asked to write up such a document for an incident where our team had written a new feature which, upon launch, did absolutely nothing. Our team had accidentally mistyped a flag name on the last day before we handed it to a test team, the test team examined the (nonfunctional) tool for a few weeks and blessed it, and then upon turning it on, it failed to do anything. My five whys document was most about "what part of our process led to a multiweek test effort that would greenlight a tool that does nothing that it is required to do." I recall my manager handing the doc back to me and saying that I needed to completely redo it because it was unacceptable for us to blame another team for our team's bug, which is how I learned that you can make a five why process blame any team you find convenient by choosing the question. I quit not too long after that.

sanman8119 1y ago
A very relatable experience, lot of pressure to stop the Whys at the dev team and not question larger leadership or organizational moves
hshdhdhehd 1y ago
Interesting one. My first thought is why is rolling out a new system to prod that is not used yet an incident? I dont think "being in prod" is sufficient. There should be tiers of service and a brand new service should not be on a tier where it having teething issues is an incident. > what part of our process led to a multiweek test effort that would greenlight a tool that does nothing that it is required to do would be interested to see the doc, but imagine you'd branch off the causes, one branch of the tree is: UAT didnt pick up the bug. why didn't UAT pick up the bug? .... (you'd need that teams help). I think that team would have something that is a contributing cause. You shouldn't rely on UAT to pick up a bug in a released product. However just because it is not a root cause doesnt mean it shouldn't be addressed. Today's contributing cause can be tomorrow's root cause! So yeah yiu dont blame another team but you also dont shield another team from one of their systems needing attention! The wording matters alot though. The way you worded the question seems a little loaded. But you may be paraphrasing? 5 whys are usually more like "Why did they papaya team not detect the bug before deployment?" Whereas > what part of our process led to a multiweek test effort that would greenlight a tool that does nothing that it is required to do Is more emotive. Sounds like a cross examiners question which isn't the vibe you'd want to go for. 5 whys should be 5 drys. Nothing spicy!
- NikolaNovak 1y ago
  That's how we do it - there are "branches" to most of our RCAs, and in fact, we have separate sections for root cause analysis (things which directly or indirectly contribute to incident, which are a branched / fractal 5 whys) and lessons learned (things which did not necessarily contribute to incident but which upon reflection we can do better - frequently incident management or communication or reporting or escalation etc). It took a while for all the teams to embrace the rca process without fear and finger pointing, but now that it's trusted and accepted, problem management stream / rca process probably the healthiest / best viewed of our streams and processes :-)
- CobrastanJorji 1y ago
  It was an incident because it was important to leadership. It was a marketing targeting feature that was advertised to the local executive with some excitement by the management, so they were excited to share the results of it, and when there weren't results on the anticipated launch date, they wanted answers, which meant the manager treated it as an incident.
  - hshdhdhehd 1y ago
    Oh geez! That is very bad (almost pointy haired) use of an incident process.
BeetleB 1y ago
My litmus test for these types of processes: If root causes like "Inflexible with timelines", or "Incentives are misaligned (e.g. prioritizing career over quality)" are not permitted, the whole process is a waste of time. Edit: You can see others commenting on precisely this. Examples: https://news.ycombinator.com/item?id=45573027 https://news.ycombinator.com/item?id=45573027 https://news.ycombinator.com/item?id=45573101 https://news.ycombinator.com/item?id=45573101 https://news.ycombinator.com/item?id=45572561 https://news.ycombinator.com/item?id=45572561 https://news.ycombinator.com/item?id=45572561 https://news.ycombinator.com/item?id=45572561
- ithkuil 1y ago
  Engineering is the art of making do with what you've got and sometimes you have to treat such unreasonable i positions just like any other constraint
- milesvp 1y ago
  Having done quite a bit of politicing at a centuries old, med sized company, I can tell you that what management wants you is the assurance that this particular problem won’t happen again. Ideally there will be an actionable outcome, so someone can check that off a todo list at a later meeting Though what I’ve found is that if you have enough clout you can add an addendum to the root cause analysis, and you can start getting into things like misaligned incentives. But always keep in mind, at best you can only point out this will mean this class of problem will keep happening. If you do this, know that there be dragons. You have to be very careful here, because for any sufficiently large company, misaligned incentives are largely defined by the org chart and it’s boudaries. You will be adding fuel to politics that is likely above your pay grade, and the fallout can be career changing. I was lucky to have a neutral reputation, as someine who cared more about the product than personal gain. So I got a lot of leeway when I said tonedeaf things. Even still I ended up in crosshairs once or twice in the 10 years I was at the company for having opinions about systemic problems.
  - BeetleB 1y ago
    > Having done quite a bit of politicing at a centuries old, med sized company, I can tell you that what management wants you is the assurance that this particular problem won’t happen again. I'm not disagreeing. I'm saying they should phrase it this way (and some do), instead of masking it with an insincere request for root causing. > Ideally there will be an actionable outcome, so someone can check that off a todo list at a later meeting Occasionally this is the right thing to do. And often this results in a very long checklist that slows the whole development down because they don't want to do a cost-benefit analysis of whether not having an occasional escape is worth the decrease in productivity. And this is because the incentives for the manager is such that the occasional escape is not OK. In reality, though, he will insist on an ever growing checklist without a compromise in velocity. And that's a great recipe for more escapes. That's the problem with root cause analyses. Sometimes the occasional escape is totally OK if you actually analyze the costs. But they want us to "analyze" while turning a blind eye to certain things. I've worked at places that understood this and didn't have this attitude. And places that didn't. Don't work for the latter. BTW, I should add that I'm not trying to give a cynical take. When I first learned the five whys, I applied it to my own problems (work and otherwise). And I found it to be wholly unsatisfying. For one thing, there usually isn't a root cause. There are multiple causes at play, and you need a branching algorithm to explore the space. More importantly, 5 (or 3) is an arbitrary number. If you keep at it, you'll almost always end up with "human nature" or "capitalism". Deciding when to stop the search is relatively arbitrary, and most people will pick a convenient endpoint. Much simpler is: 1. What can we do to prevent this from happening again? 2. Should we solve this problem? Expanding on the latter, I once worked at a place where my manager and his manager were militant about not polluting the codebase to solve problems caused by other tools. We'd sternly tell the customers that they need to go to the problematic tool's owner and fix them, and were ready to have that battle at senior levels. This was in a factory environment, so there were real $$ associated with bugs. And our team had a reputation for quality, and this was one of the reasons we had that reputation. All too often people use software as a workaround, and over the years there accumulate too many workarounds. We were, in a sense, a weapon upper management wielded to ensure other tools maintained their quality.
grogers 1y ago
Usually another team's failure is covered by their own independent report. That simplifies creating the report since you don't need to collaborate closely, but also prevents shifting the blame on to anyone else (because really, both teams had failures they should have caught independently). E.g. as the last why: Why did the testing team not catch that the feature was not functional? This is covered by LINK
- CobrastanJorji 1y ago
  If a root cause analysis is not cross team, how deep can the analysis possibly be? "Whoops, that question leads to this other process that our team doesn't directly control, guess we stop thinking about that!"
  - lijok 1y ago
    Pretty deep. It forces you to account for failures in other domains
  - taeric 1y ago
    If your root cause is cross team, then you wind up having to make some implicit assumptions on what the other team could have done. Is akin to ending with "because the gods got angry." Not really actionable. This is a classic "limit the scope of the feature." You want the document to be written and constrained to someone that is in a position to impact everything they talk about. If you think there was something more holistic, push for that, as well. Note you can discuss what other teams are doing. But do that in a way that is strictly factual. And then ask why that led your team to the failure that your team owns.
    - ghurtado 1y ago
      > on what the other team could have done If you're wondering what anyone "could have done", you've already missed the point of the article completely.
      - 1y ago
tayo42 1y ago
5 why's can be very political. You can make it take whatever direction you want to tell what ever story you want. I don't get why it's cargo culted the way it is
- stonemetal12 1y ago
  No, people can be very political. It doesn't matter what the process is. Hell, people even legislated the value of PI that one time.
  - numpad0 1y ago
    While that might be true, the five whys is notorious for slipping into a destructive "you/I suck and firing you/I solves the problem for good and I believe it makes everyone absolutely happy" style of false conclusions. Reportedly Toyota has organizational mitigations for that problem or reportedly the working culture there isn't so great after all. The bottom line is, it's a double edged sword to say the very least.
nobrains 1y ago
they way i handle this with my teams: any bugs caught by the QA team go against the developers. any bugs caught after QA green lights the go live go against the QA team. (Of course, discounting any bugs that are deemed acceptable for go live by the PM).
vivalahn 1y ago
The next org you went to, did they also use the Five Whys or did they get by with Four True Colors instead?
mrexroad 1y ago
I love 5+ why's. I find it to be a fantastic tool in many situations. Unfortunately, when leadership does not reward a culture of learning, Five Why's can become conflated with root cause analysis and just become a directed inquiry for reaching a politically expedient cause. The bigger the fuck up, the more it needs an impartial NTSB-like focus on learning and sharing to avoid them in the future. Fwiw, if I were your manager performing a root cause analysis, I'd mostly expect my team to be identifying contributing factors within their domain, and then we'd collate and analyze the factors with other respective teams to drill down to the root causes. I'd also have kicked back a doc that was mostly about blaming the other team.
- crazygringo 1y ago
  > I'd also have kicked back a doc that was mostly about blaming the other team. Agreed. If the test team messed up, then you need to answer the "why" your team didn't verify that the testing had actually been done. (And also why the team hadn't verified that the tool they'd sent to testing was even minimally functional to begin with.) Five whys are necessarily scoped to a set of people responsible. For things that happen outside that scope, the whys become about selection and verification.
  - dwattttt 1y ago
    Quis turmas probationum examinat? Validating that the build you produced works at all should be done by you, but there's also a whole team whose job it was to validate it; would you advocate for another team to test the testing teams tests? And more to the point, how do you write a 5 why's that explains how you'd typo'd a flag to turn a feature on, and another team validated that the feature worked?
    - crazygringo 1y ago
      Just saying, once you find out the testing team is unreliable, you make sure there's a form of evidence it actually got tested, that someone on your team reviews before signing off. It doesn't take a whole team. There are lots of mechanisms to produce that evidence. This is just how it works. If two checks aren't sufficient, it becomes three. Or four. Until problems stop making it through.
hinkley 1y ago
General trick in any project management is try to arrange for the work to be done by the group that has the most influence over it. Just as you should never take responsibility over something you are given no power over, you should move responsibility to where the power is (and if they won't take responsibility, you start carving out the edges of their power and hand it over to adjacent groups who will). I learned pretty early how to hack the 5 Why's in order to make sure something actionable but neither trivial nor overwhelming gets chosen as the response. And I often do it early enough in the meeting that I'm difficult to catch doing it. If I don't get invited I will sometimes crash the party, especially if the last analysis resulted only in performative work and no real progress. You get one, maybe two, and then I'm up in your business because mistakes don't mean you're stupid, but learning nothing from them does.
r_thambapillai 1y ago
while its clear that your testing team should have their own 5 Ys as well, I think its reasonable for the manager of your team asking you the question: how do we prevent this in the future? The unfortunate reality of large companies is that sometimes the quality and behaviour of other teams is (to some extent/on some time horizon) out of your control, and so the question for any given team lead is often "what can I do differently". It does seem that in this case there probably was some mechanism that your team could have had its own internal testing process that needn't replicate the full responsibilities of the testing team but could at least have caught this issue.
cyberax 1y ago
> very fundamental problem. ...5 billion years ago, the Earth coalesced from the dust cloud around the Sun...
philbo 1y ago
It sounds like there’s another failure here, which you could have documented. If the test team didn’t understand what they were meant to test, that’s a failure of communication. Simply saying “they were wrong” is not sufficient exploration of the failure so, if that’s the point your manager was making, I agree with them. Blaming a third party for misunderstanding is less useful than seeking to improve the clarity of your own communication.
conover 1y ago
In my experience, you can weaponize processes like the Five Whys or the Amazon Leadership Principles. I don’t think that means they don’t have any value. That being said, in this case, I agree with your manager. Both the QA team and your team had fundamental problems. Your team (I assume) verified the functionality which included X set of changes, and then your team made Y more changes (the flag) which were not verified. Ideally, once the functionality is verified, no more changes would be permitted in order to prevent this type of problem. The fundamental problems on the QA team were…extensive.

sanman8119 1y ago

A very relatable experience, lot of pressure to stop the Whys at the dev team and not question larger leadership or organizational moves

hshdhdhehd 1y ago

Interesting one. My first thought is why is rolling out a new system to prod that is not used yet an incident? I dont think "being in prod" is sufficient. There should be tiers of service and a brand new service should not be on a tier where it having teething issues is an incident. > what part of our process led to a multiweek test effort that would greenlight a tool that does nothing that it is required to do would be interested to see the doc, but imagine you'd branch off the causes, one branch of the tree is: UAT didnt pick up the bug. why didn't UAT pick up the bug? .... (you'd need that teams help). I think that team would have something that is a contributing cause. You shouldn't rely on UAT to pick up a bug in a released product. However just because it is not a root cause doesnt mean it shouldn't be addressed. Today's contributing cause can be tomorrow's root cause! So yeah yiu dont blame another team but you also dont shield another team from one of their systems needing attention! The wording matters alot though. The way you worded the question seems a little loaded. But you may be paraphrasing? 5 whys are usually more like "Why did they papaya team not detect the bug before deployment?" Whereas > what part of our process led to a multiweek test effort that would greenlight a tool that does nothing that it is required to do Is more emotive. Sounds like a cross examiners question which isn't the vibe you'd want to go for. 5 whys should be 5 drys. Nothing spicy!

NikolaNovak 1y ago
That's how we do it - there are "branches" to most of our RCAs, and in fact, we have separate sections for root cause analysis (things which directly or indirectly contribute to incident, which are a branched / fractal 5 whys) and lessons learned (things which did not necessarily contribute to incident but which upon reflection we can do better - frequently incident management or communication or reporting or escalation etc). It took a while for all the teams to embrace the rca process without fear and finger pointing, but now that it's trusted and accepted, problem management stream / rca process probably the healthiest / best viewed of our streams and processes :-)
CobrastanJorji 1y ago
It was an incident because it was important to leadership. It was a marketing targeting feature that was advertised to the local executive with some excitement by the management, so they were excited to share the results of it, and when there weren't results on the anticipated launch date, they wanted answers, which meant the manager treated it as an incident.
- hshdhdhehd 1y ago
  Oh geez! That is very bad (almost pointy haired) use of an incident process.

BeetleB 1y ago

My litmus test for these types of processes: If root causes like "Inflexible with timelines", or "Incentives are misaligned (e.g. prioritizing career over quality)" are not permitted, the whole process is a waste of time. Edit: You can see others commenting on precisely this. Examples: https://news.ycombinator.com/item?id=45573027 https://news.ycombinator.com/item?id=45573027 https://news.ycombinator.com/item?id=45573101 https://news.ycombinator.com/item?id=45573101 https://news.ycombinator.com/item?id=45572561 https://news.ycombinator.com/item?id=45572561 https://news.ycombinator.com/item?id=45572561 https://news.ycombinator.com/item?id=45572561

ithkuil 1y ago
Engineering is the art of making do with what you've got and sometimes you have to treat such unreasonable i positions just like any other constraint
milesvp 1y ago
Having done quite a bit of politicing at a centuries old, med sized company, I can tell you that what management wants you is the assurance that this particular problem won’t happen again. Ideally there will be an actionable outcome, so someone can check that off a todo list at a later meeting Though what I’ve found is that if you have enough clout you can add an addendum to the root cause analysis, and you can start getting into things like misaligned incentives. But always keep in mind, at best you can only point out this will mean this class of problem will keep happening. If you do this, know that there be dragons. You have to be very careful here, because for any sufficiently large company, misaligned incentives are largely defined by the org chart and it’s boudaries. You will be adding fuel to politics that is likely above your pay grade, and the fallout can be career changing. I was lucky to have a neutral reputation, as someine who cared more about the product than personal gain. So I got a lot of leeway when I said tonedeaf things. Even still I ended up in crosshairs once or twice in the 10 years I was at the company for having opinions about systemic problems.
- BeetleB 1y ago
  > Having done quite a bit of politicing at a centuries old, med sized company, I can tell you that what management wants you is the assurance that this particular problem won’t happen again. I'm not disagreeing. I'm saying they should phrase it this way (and some do), instead of masking it with an insincere request for root causing. > Ideally there will be an actionable outcome, so someone can check that off a todo list at a later meeting Occasionally this is the right thing to do. And often this results in a very long checklist that slows the whole development down because they don't want to do a cost-benefit analysis of whether not having an occasional escape is worth the decrease in productivity. And this is because the incentives for the manager is such that the occasional escape is not OK. In reality, though, he will insist on an ever growing checklist without a compromise in velocity. And that's a great recipe for more escapes. That's the problem with root cause analyses. Sometimes the occasional escape is totally OK if you actually analyze the costs. But they want us to "analyze" while turning a blind eye to certain things. I've worked at places that understood this and didn't have this attitude. And places that didn't. Don't work for the latter. BTW, I should add that I'm not trying to give a cynical take. When I first learned the five whys, I applied it to my own problems (work and otherwise). And I found it to be wholly unsatisfying. For one thing, there usually isn't a root cause. There are multiple causes at play, and you need a branching algorithm to explore the space. More importantly, 5 (or 3) is an arbitrary number. If you keep at it, you'll almost always end up with "human nature" or "capitalism". Deciding when to stop the search is relatively arbitrary, and most people will pick a convenient endpoint. Much simpler is: 1. What can we do to prevent this from happening again? 2. Should we solve this problem? Expanding on the latter, I once worked at a place where my manager and his manager were militant about not polluting the codebase to solve problems caused by other tools. We'd sternly tell the customers that they need to go to the problematic tool's owner and fix them, and were ready to have that battle at senior levels. This was in a factory environment, so there were real $$ associated with bugs. And our team had a reputation for quality, and this was one of the reasons we had that reputation. All too often people use software as a workaround, and over the years there accumulate too many workarounds. We were, in a sense, a weapon upper management wielded to ensure other tools maintained their quality.

grogers 1y ago

Usually another team's failure is covered by their own independent report. That simplifies creating the report since you don't need to collaborate closely, but also prevents shifting the blame on to anyone else (because really, both teams had failures they should have caught independently). E.g. as the last why: Why did the testing team not catch that the feature was not functional? This is covered by LINK

CobrastanJorji 1y ago
If a root cause analysis is not cross team, how deep can the analysis possibly be? "Whoops, that question leads to this other process that our team doesn't directly control, guess we stop thinking about that!"
- lijok 1y ago
  Pretty deep. It forces you to account for failures in other domains
- taeric 1y ago
  If your root cause is cross team, then you wind up having to make some implicit assumptions on what the other team could have done. Is akin to ending with "because the gods got angry." Not really actionable. This is a classic "limit the scope of the feature." You want the document to be written and constrained to someone that is in a position to impact everything they talk about. If you think there was something more holistic, push for that, as well. Note you can discuss what other teams are doing. But do that in a way that is strictly factual. And then ask why that led your team to the failure that your team owns.
  - ghurtado 1y ago
    > on what the other team could have done If you're wondering what anyone "could have done", you've already missed the point of the article completely.
    - 1y ago

tayo42 1y ago

5 why's can be very political. You can make it take whatever direction you want to tell what ever story you want. I don't get why it's cargo culted the way it is

stonemetal12 1y ago
No, people can be very political. It doesn't matter what the process is. Hell, people even legislated the value of PI that one time.
- numpad0 1y ago
  While that might be true, the five whys is notorious for slipping into a destructive "you/I suck and firing you/I solves the problem for good and I believe it makes everyone absolutely happy" style of false conclusions. Reportedly Toyota has organizational mitigations for that problem or reportedly the working culture there isn't so great after all. The bottom line is, it's a double edged sword to say the very least.

nobrains 1y ago

they way i handle this with my teams: any bugs caught by the QA team go against the developers. any bugs caught after QA green lights the go live go against the QA team. (Of course, discounting any bugs that are deemed acceptable for go live by the PM).

vivalahn 1y ago

The next org you went to, did they also use the Five Whys or did they get by with Four True Colors instead?

mrexroad 1y ago

I love 5+ why's. I find it to be a fantastic tool in many situations. Unfortunately, when leadership does not reward a culture of learning, Five Why's can become conflated with root cause analysis and just become a directed inquiry for reaching a politically expedient cause. The bigger the fuck up, the more it needs an impartial NTSB-like focus on learning and sharing to avoid them in the future. Fwiw, if I were your manager performing a root cause analysis, I'd mostly expect my team to be identifying contributing factors within their domain, and then we'd collate and analyze the factors with other respective teams to drill down to the root causes. I'd also have kicked back a doc that was mostly about blaming the other team.

crazygringo 1y ago
> I'd also have kicked back a doc that was mostly about blaming the other team. Agreed. If the test team messed up, then you need to answer the "why" your team didn't verify that the testing had actually been done. (And also why the team hadn't verified that the tool they'd sent to testing was even minimally functional to begin with.) Five whys are necessarily scoped to a set of people responsible. For things that happen outside that scope, the whys become about selection and verification.
- dwattttt 1y ago
  Quis turmas probationum examinat? Validating that the build you produced works at all should be done by you, but there's also a whole team whose job it was to validate it; would you advocate for another team to test the testing teams tests? And more to the point, how do you write a 5 why's that explains how you'd typo'd a flag to turn a feature on, and another team validated that the feature worked?
  - crazygringo 1y ago
    Just saying, once you find out the testing team is unreliable, you make sure there's a form of evidence it actually got tested, that someone on your team reviews before signing off. It doesn't take a whole team. There are lots of mechanisms to produce that evidence. This is just how it works. If two checks aren't sufficient, it becomes three. Or four. Until problems stop making it through.

hinkley 1y ago

General trick in any project management is try to arrange for the work to be done by the group that has the most influence over it. Just as you should never take responsibility over something you are given no power over, you should move responsibility to where the power is (and if they won't take responsibility, you start carving out the edges of their power and hand it over to adjacent groups who will). I learned pretty early how to hack the 5 Why's in order to make sure something actionable but neither trivial nor overwhelming gets chosen as the response. And I often do it early enough in the meeting that I'm difficult to catch doing it. If I don't get invited I will sometimes crash the party, especially if the last analysis resulted only in performative work and no real progress. You get one, maybe two, and then I'm up in your business because mistakes don't mean you're stupid, but learning nothing from them does.

r_thambapillai 1y ago

while its clear that your testing team should have their own 5 Ys as well, I think its reasonable for the manager of your team asking you the question: how do we prevent this in the future? The unfortunate reality of large companies is that sometimes the quality and behaviour of other teams is (to some extent/on some time horizon) out of your control, and so the question for any given team lead is often "what can I do differently". It does seem that in this case there probably was some mechanism that your team could have had its own internal testing process that needn't replicate the full responsibilities of the testing team but could at least have caught this issue.

cyberax 1y ago

> very fundamental problem. ...5 billion years ago, the Earth coalesced from the dust cloud around the Sun...

philbo 1y ago

It sounds like there’s another failure here, which you could have documented. If the test team didn’t understand what they were meant to test, that’s a failure of communication. Simply saying “they were wrong” is not sufficient exploration of the failure so, if that’s the point your manager was making, I agree with them. Blaming a third party for misunderstanding is less useful than seeking to improve the clarity of your own communication.

conover 1y ago

In my experience, you can weaponize processes like the Five Whys or the Amazon Leadership Principles. I don’t think that means they don’t have any value. That being said, in this case, I agree with your manager. Both the QA team and your team had fundamental problems. Your team (I assume) verified the functionality which included X set of changes, and then your team made Y more changes (the flag) which were not verified. Ideally, once the functionality is verified, no more changes would be permitted in order to prevent this type of problem. The fundamental problems on the QA team were…extensive.