3 ms·

I did a Five Whys for Amazon once. It had been an embarrassing incident for the team. We had put a huge new feature together over a few months, and when we turn

I did a Five Whys for Amazon once. It had been an embarrassing incident for the team. We had put a huge new feature together over a few months, and when we turned it on...nothing happened. The whole feature had been entirely disabled by a typo I made in a configuration change. At the time, we had a test team that was tremendously overworked and tested things almost entirely manually. They asked for a couple extra weeks to test our new feature but eventually greenlit it. My Five Whys basically asked "how did this get to production if it didn't do anything?" My manager handed it back to me and said "this takes the blame off of you for this mistake and puts it on the testing team. Ask different questions." So I wrote another one that was 100% focused on me. "Why did this happen? Because the config was wrong." "Why was the config wrong? Because I made a typo." "Why didn't I notice the typo? Because I didn't run the feature after making the change." "Why didn't I try it out? Because the feature is very difficult to invoke or observe." "Why is the feature hard to invoke directly?" "Because <x,y,z>." What I learned from that is that the problem with Five Whys is that there are many different branches of questions that are worth asking, but that design only allows for exactly one path. Choosing which one is an editorial decision on the part of the author. The outcome of a 5-Whys document is, by its nature, one change. But most of the time, outages at mature tech companies are the result of several different problems coincidentally lining up (if exactly one thing going wrong caused a major problem, that itself is the second problem), and a postmortem process really needs to examine each of the things that went wrong separately.

RealityVoid 5y ago
I has a similar experience where I had to 5 why a issue and... the gist of it was that the why was because I bungled it up. I just.. saw it as an issue while I wrote my stuff, added a TODO and then... just never did it. So then, after the first 2 why's I could only answer in the back of my head, "Because I'm fucking stupid, that's why!" I get the idea with the why's but sometimes, it's just that simple, you can't really go deeper than that. I fucked up.
- chucksmash 5y ago
  I think you can still usefully push past the "I just never did it." If the problem is that you are overcommitted and things are starting to slip, that's a different problem than "I got distracted by something shiny and forgot." In the first case, it might make sense to try to grow the team, for instance.
- necovek 5y ago
  The question then becomes: what could you have done for the "system" to automatically prevent you from forgetting that? Would an end-to-end or smoketest prevented that? Would having a TODO-linter have stopped this? Would singing your country's anthem from the top of a hill prevented this (I dunno, maybe it's a really magic song)? You are looking for improvements to your process that will help you protect yourself from yourself. :)
  - earth_walker 5y ago
    Exactly this. We have a saying in the GxP QA world - look at the process, not the person. Anyone can make mistakes, and sometimes the naive 'why' of something is just that someone made a mistake. But if you focus your questions on the process, it becomes 'why did the process allow this mistake through'. Perhaps a checklist could have helped. Perhaps you need a QC [1] step in the process. Maybe the instructions aren't detailed enough. [1] Though there are many sometimes subtle differences in how these terms are used across industries, generally the difference between Quality Control and Quality Assurance is that QC is part of the process (e.g. a built-in check in the system), while QA is a separate process making sure these checks and balances are working properly. To bring in the car analogy, QC would be your tire pressure sensors, and QA would be your mechanic checking that the pressure sensors are working correctly.
- throwaway984393 5y ago
  You can always go deeper. Why did you fuck up? Because you didn't get enough sleep. Why didn't you get enough sleep? Because you were on-call at 3am. Why were you on-call at 3am? Because the apps keep crashing once a week. Why are the apps crashing once a week? Because nobody has prioritized fixing them. Why has nobody prioritized fixing them? Because the feature work is higher priority. Or: Why did you fuck up? There was no validation of the change before applying it. Why was there no validation of the change? Because we've never prioritized validation. Why have we never prioritized validation? Because we don't think it's important. Why don't we think it's important? Because we never look back at our previous failures to see if there's a pattern of failures from lack of validation. So even from "I fucked up" we find a number of problems and things we can start addressing.
  - raffraffraff 5y ago
    > "Because feature work is higher priority" Major cause of burnout and incidents, yet management will ignore this particular "why", because their promotions are based on features, mostly. Also why it sucks being an SRE unless the company empowers your team.
- usefulcat 5y ago
  The hardest part is usually figuring out a reliable way to avoid making the same mistake in the future. "Don't make mistakes" is never a viable solution to that problem.
ajkjk 5y ago
Yeah, I don't think blaming the typo is the right outcome for this. The right outcome would be identifying where your testing / review / launch / validation process failed. Evidently it failed, or nothing would have gone wrong even when you made a typo. Usually the response to a good root-cause investigation isn't "I'll try to avoid making mistakes" (although sometimes it's just a mistake and no changes is required). The outcome should be a new rule or practice for the organization, some kind of decision that determines how things will be done going forward to make the error less likely. This almost always requires a leader at some level declaring by fiat that things will be done this way going forward; if your managers aren't doing that, they're not 'leading' at all.
didibus 5y ago
I don't think that's an issue per-say. You can't fix everything, but you can at least try to find the most likely path and do something about it. Even if it wasn't the biggest cause, you made progress towards improving some of the variables at play for the issue. There's clearly something to learn and improve in your 5 why's. There was a hard to test feature, typos were allowed to pass through validation in the config, there was only one person making the config change and seemingly no one to review the change, etc. Ok so all these seem like they'd contribute at least somewhat to why this issue occurred. Sure there might have been other deficiencies, like why didn't the test teams also forgot to test the feature or thought they did but didn't, etc. But you can't address everything. So maybe you update the validation code for the config so it is more likely to catch such typo in the future. Maybe you make improvements to your config change management and enforce a 2-eye rule on config changes, and a 2 person review on the change. Maybe you make improvements to your testing capabilities so that such hard to trigger features become easier to write an automated test for. Maybe the leadership learns to inssist on planning more time for dev QA and writing tests when launching the next big feature, etc.
throwaway984393 5y ago
Toyota's implementation of Five Why's actually doesn't stick to just one path, they can branch out or do multiple of them. But you're right about multiple causes and the need for alternative strategies to find and fix causes. The problem with process is that most people follow it as prescribed rather than following the spirit of it.
kevinsundar 5y ago
This is not accurate. Im currently working on a "5 Whys". I am writing it in a nested bullet format which allows multiple branches. Example: * Main Why? * Why 1? * Sub Why 1? * More Whys * Why 2? * Sub Why 2? Its totally fine to have more than one change or path. Theres nothing in the nature of the doc that requires one change or one path, thats just an assumption.
- necovek 5y ago
  The "theory" of 5-whys is that with a proper mindset and the right questions (coming out of that mindset), you can find a root cause to anything in at most 5 linear steps. You are certainly free to adapt it in whatever way you wish, but you don't call that "5 whys" anymore. TBH, your approach looks closer to fishbone diagrams instead.
  - geoduck14 5y ago
    >The "theory" of 5-whys is that with a proper mindset and the right questions (coming out of that mindset), you can find a root cause to anything in at most 5 linear steps. I have never heard of this before. I had heard that the 5 why's was a way to force a person or team to be more than surface level about why a feature is needed or a problem happened. And also, ONE way to represent the work was through a fishbone. But part of the 5 why's is that multiple "why's" converge (e.g. why did this break? because we have no change control... why does X process suck? Also, because we have no change control) My favorite feature of 5 why's is they push you into going deeper. If a process keeps failing, the 1st level reason for the failures can be different (X person changed the data, dates suck, the primary server crashed), but the 3rd or 4th reasons should show some similarities and ideas on how to prioritize (e.g. if we have automated unit tests, it would have prevented 25% of our outages!)
  - kevinsundar 5y ago
    This is wholly incorrect. The internal tool in which you write these 5 why's specifically says: "5 Whys" is a helpful mnemonic but authors are encouraged to use more than five causes as they complete their root cause analysis
freetinker 5y ago
A better way of doing this would be the Ishikawa diagram. https://en.wikipedia.org/wiki/Ishikawa_diagram https://en.wikipedia.org/wiki/Ishikawa_diagram
- zhte415 5y ago
  It isn't better, it's different. There are a plethora of tools that go into a problem analysis, six sigma, lean, or whatever labelled toolbox. The important thing is being able to use the tool, not just do it, as the parent mentions in their comment: It's easy for a manager to say "do a 5 whys". It's another to be able to support fixing what it finds out. Or if not your manager and just you or a small team with seemingly everything in your control to introspect and challenge assumptions or prior work that might need rethinking. That's the hard part.
908B64B197 5y ago
> "Why didn't I try it out? Because the feature is very difficult to invoke or observe." "Why is the feature hard to invoke directly?" "Because <x,y,z>." Now that's engineering. Code or any feature that's hard to get in isolation and test is prone to fail in unexpected ways. And if the failure in prod is big enough that gives you the opportunity to show that it's cheaper to invest engineering time to make this part of the product better instead of picking up the pieces later on.

RealityVoid 5y ago

I has a similar experience where I had to 5 why a issue and... the gist of it was that the why was because I bungled it up. I just.. saw it as an issue while I wrote my stuff, added a TODO and then... just never did it. So then, after the first 2 why's I could only answer in the back of my head, "Because I'm fucking stupid, that's why!" I get the idea with the why's but sometimes, it's just that simple, you can't really go deeper than that. I fucked up.

chucksmash 5y ago
I think you can still usefully push past the "I just never did it." If the problem is that you are overcommitted and things are starting to slip, that's a different problem than "I got distracted by something shiny and forgot." In the first case, it might make sense to try to grow the team, for instance.
necovek 5y ago
The question then becomes: what could you have done for the "system" to automatically prevent you from forgetting that? Would an end-to-end or smoketest prevented that? Would having a TODO-linter have stopped this? Would singing your country's anthem from the top of a hill prevented this (I dunno, maybe it's a really magic song)? You are looking for improvements to your process that will help you protect yourself from yourself. :)
- earth_walker 5y ago
  Exactly this. We have a saying in the GxP QA world - look at the process, not the person. Anyone can make mistakes, and sometimes the naive 'why' of something is just that someone made a mistake. But if you focus your questions on the process, it becomes 'why did the process allow this mistake through'. Perhaps a checklist could have helped. Perhaps you need a QC [1] step in the process. Maybe the instructions aren't detailed enough. [1] Though there are many sometimes subtle differences in how these terms are used across industries, generally the difference between Quality Control and Quality Assurance is that QC is part of the process (e.g. a built-in check in the system), while QA is a separate process making sure these checks and balances are working properly. To bring in the car analogy, QC would be your tire pressure sensors, and QA would be your mechanic checking that the pressure sensors are working correctly.
throwaway984393 5y ago
You can always go deeper. Why did you fuck up? Because you didn't get enough sleep. Why didn't you get enough sleep? Because you were on-call at 3am. Why were you on-call at 3am? Because the apps keep crashing once a week. Why are the apps crashing once a week? Because nobody has prioritized fixing them. Why has nobody prioritized fixing them? Because the feature work is higher priority. Or: Why did you fuck up? There was no validation of the change before applying it. Why was there no validation of the change? Because we've never prioritized validation. Why have we never prioritized validation? Because we don't think it's important. Why don't we think it's important? Because we never look back at our previous failures to see if there's a pattern of failures from lack of validation. So even from "I fucked up" we find a number of problems and things we can start addressing.
- raffraffraff 5y ago
  > "Because feature work is higher priority" Major cause of burnout and incidents, yet management will ignore this particular "why", because their promotions are based on features, mostly. Also why it sucks being an SRE unless the company empowers your team.
usefulcat 5y ago
The hardest part is usually figuring out a reliable way to avoid making the same mistake in the future. "Don't make mistakes" is never a viable solution to that problem.

ajkjk 5y ago

Yeah, I don't think blaming the typo is the right outcome for this. The right outcome would be identifying where your testing / review / launch / validation process failed. Evidently it failed, or nothing would have gone wrong even when you made a typo. Usually the response to a good root-cause investigation isn't "I'll try to avoid making mistakes" (although sometimes it's just a mistake and no changes is required). The outcome should be a new rule or practice for the organization, some kind of decision that determines how things will be done going forward to make the error less likely. This almost always requires a leader at some level declaring by fiat that things will be done this way going forward; if your managers aren't doing that, they're not 'leading' at all.

didibus 5y ago

I don't think that's an issue per-say. You can't fix everything, but you can at least try to find the most likely path and do something about it. Even if it wasn't the biggest cause, you made progress towards improving some of the variables at play for the issue. There's clearly something to learn and improve in your 5 why's. There was a hard to test feature, typos were allowed to pass through validation in the config, there was only one person making the config change and seemingly no one to review the change, etc. Ok so all these seem like they'd contribute at least somewhat to why this issue occurred. Sure there might have been other deficiencies, like why didn't the test teams also forgot to test the feature or thought they did but didn't, etc. But you can't address everything. So maybe you update the validation code for the config so it is more likely to catch such typo in the future. Maybe you make improvements to your config change management and enforce a 2-eye rule on config changes, and a 2 person review on the change. Maybe you make improvements to your testing capabilities so that such hard to trigger features become easier to write an automated test for. Maybe the leadership learns to inssist on planning more time for dev QA and writing tests when launching the next big feature, etc.

throwaway984393 5y ago

Toyota's implementation of Five Why's actually doesn't stick to just one path, they can branch out or do multiple of them. But you're right about multiple causes and the need for alternative strategies to find and fix causes. The problem with process is that most people follow it as prescribed rather than following the spirit of it.

kevinsundar 5y ago

This is not accurate. Im currently working on a "5 Whys". I am writing it in a nested bullet format which allows multiple branches. Example: * Main Why? * Why 1? * Sub Why 1? * More Whys * Why 2? * Sub Why 2? Its totally fine to have more than one change or path. Theres nothing in the nature of the doc that requires one change or one path, thats just an assumption.

necovek 5y ago
The "theory" of 5-whys is that with a proper mindset and the right questions (coming out of that mindset), you can find a root cause to anything in at most 5 linear steps. You are certainly free to adapt it in whatever way you wish, but you don't call that "5 whys" anymore. TBH, your approach looks closer to fishbone diagrams instead.
- geoduck14 5y ago
  >The "theory" of 5-whys is that with a proper mindset and the right questions (coming out of that mindset), you can find a root cause to anything in at most 5 linear steps. I have never heard of this before. I had heard that the 5 why's was a way to force a person or team to be more than surface level about why a feature is needed or a problem happened. And also, ONE way to represent the work was through a fishbone. But part of the 5 why's is that multiple "why's" converge (e.g. why did this break? because we have no change control... why does X process suck? Also, because we have no change control) My favorite feature of 5 why's is they push you into going deeper. If a process keeps failing, the 1st level reason for the failures can be different (X person changed the data, dates suck, the primary server crashed), but the 3rd or 4th reasons should show some similarities and ideas on how to prioritize (e.g. if we have automated unit tests, it would have prevented 25% of our outages!)
- kevinsundar 5y ago
  This is wholly incorrect. The internal tool in which you write these 5 why's specifically says: "5 Whys" is a helpful mnemonic but authors are encouraged to use more than five causes as they complete their root cause analysis

freetinker 5y ago

A better way of doing this would be the Ishikawa diagram. https://en.wikipedia.org/wiki/Ishikawa_diagram https://en.wikipedia.org/wiki/Ishikawa_diagram

zhte415 5y ago
It isn't better, it's different. There are a plethora of tools that go into a problem analysis, six sigma, lean, or whatever labelled toolbox. The important thing is being able to use the tool, not just do it, as the parent mentions in their comment: It's easy for a manager to say "do a 5 whys". It's another to be able to support fixing what it finds out. Or if not your manager and just you or a small team with seemingly everything in your control to introspect and challenge assumptions or prior work that might need rethinking. That's the hard part.

908B64B197 5y ago

> "Why didn't I try it out? Because the feature is very difficult to invoke or observe." "Why is the feature hard to invoke directly?" "Because <x,y,z>." Now that's engineering. Code or any feature that's hard to get in isolation and test is prone to fail in unexpected ways. And if the failure in prod is big enough that gives you the opportunity to show that it's cheaper to invest engineering time to make this part of the product better instead of picking up the pieces later on.