6 ms·

Many years ago, I worked at Amazon, and at the time they LOVED "5 Why" analysis. I had to write a couple of them. The second and final one I wrote came back fro

Many years ago, I worked at Amazon, and at the time they LOVED "5 Why" analysis. I had to write a couple of them. The second and final one I wrote came back from my manager, who told me that it was unacceptable for my "5 Whys" to blame something outside our team's control when the most immediate cause was a bug written by our team, and I needed to come up with a different ultimate cause. This was really frustrating, but it was easy enough to completely change the cause without being dishonest. "5 Why" style analysis is really easy to steer toward whatever you want to blame.

dustindiamond 8y ago
Kinda like ‘doing agile’ can mean anything, it seems.
- thnukid 8y ago
  Kinda like `Object Oriented` Programming, which is not about Objects.
  - booleandilemma 8y ago
    OOP has some very clearly defined concepts, object being one of them, and the concepts are almost always baked into the languages themselves.
  - danso 8y ago
    It's "Object Oriented Programming" not "Object Only Programming
awill 8y ago
You can't end a 5 Whys with 'another team I depend on screwed up'. You have to own your software or service. If you depend on something, you need checks, alarms, and whenever possible the ability to handle dependency failure well. Imagine you're built on a public cloud in a single Data Centre. That DC has an issue. Your 5 Whys shouldn't end with 'the Data Centre had an issue' It should continue with 'my service was only in one DC.... Why?' and continue from there.
- CobrastanJorji 8y ago
  Sometimes a problem that affects you is outside your immediate ownership, and the right answer is to escalate and seek to fix it. If you limit yourself to only considering changes that your immediate team can make on its own without consulting others, you're doing your organization a disservice.
  - awill 8y ago
    sure, if your service behaved as well as possible during your dependency's outage, yes.
- perlgeek 8y ago
  > You can't end a 5 Whys with 'another team I depend on screwed up'. You have to own your software or service. Our service was down. Why? Because the platform it runs on went down. Why only one platform? Because management doesn't want us to spend several years engineering for multiple platforms. --- I find it hard to imagine a realistic path of 5 whys that leads us back to a cause within our own team.
  - Karrot_Kream 8y ago
    > Because management doesn't want us to spend several years engineering for multiple platforms. Yes and that's fine. It's okay to end up with an RCA like that and note that as a known risk. If a cost-benefit analysis shows that the risk is dwarfed by the solution, then it's not worth fixing. Knowing the existing risks of your system and reevaluating these risks in the face of changes (more people available, inefficient context switching, etc) is a key component of making progress.
    - awill 8y ago
      True. In a case this simple, you likely wouldn't have to write an RCA. However, most real life situations I've seen show that there's always room to improve. For example, how quickly did your service recover once the underlying platform recovered. Rare events (like DC failure) can be a bad thing for operational training. Teams forget how to recover. No runbooks, no training, or tooling that's infrequently used no longer works.
jimbokun 8y ago
I don't understand, if the bug was written by your team, how could the ultimate cause be something outside of your team? Seems like 5 Whys in that scenario would lead you to look at things like insufficient test coverage, no code review, code reviewers missed the bug because... Those paths of inquiry lead to your team changing things under your direct control that could make it less likely for similar bugs to reach production in the future.
- tedunangst 8y ago
  Team two promised us this service would never fail, so we didn't check for errors...
- espadrine 8y ago
  Broken expectations are a cause. For instance, when ext4 gained traction, some users found a number of zeroed-out files after a crash. Why? Because the programs managing those files updated them by truncating them and then writing them. Why? Because fopen() has no "replace file" mode, even though the "w" mode seems to do just that as long as the system doesn't crash (it does O_WRONLY|O_CREAT|O_TRUNC). Why? Because file systems used to not have delayed allocation when fopen() was created. ext4 introduced the concept to the ext family. Why? Because ext4 was designed to improve performance compared to ext3, and that change does that without breaking POSIX. Why? Because, even with ext3, the only way to correctly implement "replace file" on POSIX is to write the replacement file on the same directory, fsync() the file, close the file descriptor, fsync() the directory, and rename the file. At that point, the whys have shifted the blame through userland programmers, POSIX fopen(), ext4, and back to userland programmers. Depending on where you stop, the solution can be: 1. fix all userland programs, 2. change ext4 to force sync upon closing a truncated file's descriptor (which got implemented), 3. upgrade the POSIX standard to have a dedicated "replace file" function call. The truth is, userland is buggy, and the POSIX standard is arcane, so they both share blame. (But ext4, which was correctly implemented, is the only thing that got patched.) http://dream.thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/ http://dream.thunk.org/tytso/blog/2009/03/12/delayed-allocat...
  - 8y ago
hinkley 8y ago
So you know how in Refactoring you're supposed to be doing bottom-up design, but when the problem gets big it turns into top-down design masquerading as bottom-up? (aside: the Mikado method allows you to fix this) Quite often when I'm doing 5 Why's I find myself socially engineering Why 3-5 to lead us to an action item we can do something about. There are any number of Why #5's someone can generate but a lot of them do not represent progress the team can get behind. The fact that this happens with so many different teams and managers makes me wonder if it's me or there's just an obvious pattern of misuse going on with 5 Whys.
stcredzero 8y ago
"5 Why" style analysis is really easy to steer toward whatever you want to blame. How is this better than "Just So Stories?"
- kochikame 8y ago
  Exactly
kochikame 8y ago
> "5 Why" style analysis is really easy to steer toward whatever you want to blame. This is the big issue. Ideology, bias and self-interest creep in and you can easily end up blaming whatever it is that you personally disagree with. And that's to not even mention the "unknown unknowns", which by their very nature cannot be factored in to a "5 Whys" process, no matter their real world relevance to the problem at hand

dustindiamond 8y ago

Kinda like ‘doing agile’ can mean anything, it seems.

thnukid 8y ago
Kinda like `Object Oriented` Programming, which is not about Objects.
- booleandilemma 8y ago
  OOP has some very clearly defined concepts, object being one of them, and the concepts are almost always baked into the languages themselves.
- danso 8y ago
  It's "Object Oriented Programming" not "Object Only Programming

awill 8y ago

You can't end a 5 Whys with 'another team I depend on screwed up'. You have to own your software or service. If you depend on something, you need checks, alarms, and whenever possible the ability to handle dependency failure well. Imagine you're built on a public cloud in a single Data Centre. That DC has an issue. Your 5 Whys shouldn't end with 'the Data Centre had an issue' It should continue with 'my service was only in one DC.... Why?' and continue from there.

CobrastanJorji 8y ago
Sometimes a problem that affects you is outside your immediate ownership, and the right answer is to escalate and seek to fix it. If you limit yourself to only considering changes that your immediate team can make on its own without consulting others, you're doing your organization a disservice.
- awill 8y ago
  sure, if your service behaved as well as possible during your dependency's outage, yes.
perlgeek 8y ago
> You can't end a 5 Whys with 'another team I depend on screwed up'. You have to own your software or service. Our service was down. Why? Because the platform it runs on went down. Why only one platform? Because management doesn't want us to spend several years engineering for multiple platforms. --- I find it hard to imagine a realistic path of 5 whys that leads us back to a cause within our own team.
- Karrot_Kream 8y ago
  > Because management doesn't want us to spend several years engineering for multiple platforms. Yes and that's fine. It's okay to end up with an RCA like that and note that as a known risk. If a cost-benefit analysis shows that the risk is dwarfed by the solution, then it's not worth fixing. Knowing the existing risks of your system and reevaluating these risks in the face of changes (more people available, inefficient context switching, etc) is a key component of making progress.
  - awill 8y ago
    True. In a case this simple, you likely wouldn't have to write an RCA. However, most real life situations I've seen show that there's always room to improve. For example, how quickly did your service recover once the underlying platform recovered. Rare events (like DC failure) can be a bad thing for operational training. Teams forget how to recover. No runbooks, no training, or tooling that's infrequently used no longer works.

jimbokun 8y ago

I don't understand, if the bug was written by your team, how could the ultimate cause be something outside of your team? Seems like 5 Whys in that scenario would lead you to look at things like insufficient test coverage, no code review, code reviewers missed the bug because... Those paths of inquiry lead to your team changing things under your direct control that could make it less likely for similar bugs to reach production in the future.

tedunangst 8y ago
Team two promised us this service would never fail, so we didn't check for errors...
espadrine 8y ago
Broken expectations are a cause. For instance, when ext4 gained traction, some users found a number of zeroed-out files after a crash. Why? Because the programs managing those files updated them by truncating them and then writing them. Why? Because fopen() has no "replace file" mode, even though the "w" mode seems to do just that as long as the system doesn't crash (it does O_WRONLY|O_CREAT|O_TRUNC). Why? Because file systems used to not have delayed allocation when fopen() was created. ext4 introduced the concept to the ext family. Why? Because ext4 was designed to improve performance compared to ext3, and that change does that without breaking POSIX. Why? Because, even with ext3, the only way to correctly implement "replace file" on POSIX is to write the replacement file on the same directory, fsync() the file, close the file descriptor, fsync() the directory, and rename the file. At that point, the whys have shifted the blame through userland programmers, POSIX fopen(), ext4, and back to userland programmers. Depending on where you stop, the solution can be: 1. fix all userland programs, 2. change ext4 to force sync upon closing a truncated file's descriptor (which got implemented), 3. upgrade the POSIX standard to have a dedicated "replace file" function call. The truth is, userland is buggy, and the POSIX standard is arcane, so they both share blame. (But ext4, which was correctly implemented, is the only thing that got patched.) http://dream.thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/ http://dream.thunk.org/tytso/blog/2009/03/12/delayed-allocat...
- 8y ago

hinkley 8y ago

So you know how in Refactoring you're supposed to be doing bottom-up design, but when the problem gets big it turns into top-down design masquerading as bottom-up? (aside: the Mikado method allows you to fix this) Quite often when I'm doing 5 Why's I find myself socially engineering Why 3-5 to lead us to an action item we can do something about. There are any number of Why #5's someone can generate but a lot of them do not represent progress the team can get behind. The fact that this happens with so many different teams and managers makes me wonder if it's me or there's just an obvious pattern of misuse going on with 5 Whys.

stcredzero 8y ago

"5 Why" style analysis is really easy to steer toward whatever you want to blame. How is this better than "Just So Stories?"

kochikame 8y ago
Exactly

kochikame 8y ago

> "5 Why" style analysis is really easy to steer toward whatever you want to blame. This is the big issue. Ideology, bias and self-interest creep in and you can easily end up blaming whatever it is that you personally disagree with. And that's to not even mention the "unknown unknowns", which by their very nature cannot be factored in to a "5 Whys" process, no matter their real world relevance to the problem at hand