4 ms·

> You can't end a 5 Whys with 'another team I depend on screwed up'. You have to own your software or service. Our service was down. Why? Because the platfor

> You can't end a 5 Whys with 'another team I depend on screwed up'. You have to own your software or service. Our service was down. Why? Because the platform it runs on went down. Why only one platform? Because management doesn't want us to spend several years engineering for multiple platforms. --- I find it hard to imagine a realistic path of 5 whys that leads us back to a cause within our own team.

Karrot_Kream 8y ago
> Because management doesn't want us to spend several years engineering for multiple platforms. Yes and that's fine. It's okay to end up with an RCA like that and note that as a known risk. If a cost-benefit analysis shows that the risk is dwarfed by the solution, then it's not worth fixing. Knowing the existing risks of your system and reevaluating these risks in the face of changes (more people available, inefficient context switching, etc) is a key component of making progress.
- awill 8y ago
  True. In a case this simple, you likely wouldn't have to write an RCA. However, most real life situations I've seen show that there's always room to improve. For example, how quickly did your service recover once the underlying platform recovered. Rare events (like DC failure) can be a bad thing for operational training. Teams forget how to recover. No runbooks, no training, or tooling that's infrequently used no longer works.
  - grogenaut 8y ago
    You actually want to assign to root cause and track that. Them over time you build up data on if it's worth it to stay on one platform or not. Eg doing this let's you build a risk profile to counter the $/time argument against. Then you get to make an informed decision. You don't just shruggie and move in "that ship has sailed" at least if you're doing it right.
msla 8y ago
Alternatively: Our service was down. Why? Because the platform it runs on went down. Why only one platform? Because management declined to approve purchase orders for sufficient hardware to allow redundancy. --- And your career just hit a wall. Passing the buck up the chain tends to do that.
- sfifs 8y ago
  So you don't word it in a way to assign blame. You say previous cost benefit analysis indicated X known trade off from lack of redundancy. This incident either confirms the trade off (finance will usually support you for this) or say additional X factor not previously known needs to be taken into the trade off and revised.
  - therein 8y ago
    That indeed sounds much better.
  - LaGrange 8y ago
    Plot twist: X factor was known but dismissed by management. Management knows this and knows how to read between lines. Your career hit a brick wall and you didn't even get to openly snub the management.
    - roenxi 8y ago
      You're pushing the scenario into foolish territory. If management decisions are the root cause of a problem and they won't accept that, then the quality of the process used to identify the root cause is irrelevant. You could use the best or worst process and it wouldn't matter. If your management is bad, then there are a lot more risks to your career than trying to work with them to improve the process. It is more likely that a given employee has dead-ended their career because they don't know how to explain a problem without blaming people and having the humility to see from other people's point of view; both useful skills for getting a promotion.
      - 8y ago
8note 8y ago
how do make effective action items with that root cause?

Karrot_Kream 8y ago

> Because management doesn't want us to spend several years engineering for multiple platforms. Yes and that's fine. It's okay to end up with an RCA like that and note that as a known risk. If a cost-benefit analysis shows that the risk is dwarfed by the solution, then it's not worth fixing. Knowing the existing risks of your system and reevaluating these risks in the face of changes (more people available, inefficient context switching, etc) is a key component of making progress.

awill 8y ago
True. In a case this simple, you likely wouldn't have to write an RCA. However, most real life situations I've seen show that there's always room to improve. For example, how quickly did your service recover once the underlying platform recovered. Rare events (like DC failure) can be a bad thing for operational training. Teams forget how to recover. No runbooks, no training, or tooling that's infrequently used no longer works.
- grogenaut 8y ago
  You actually want to assign to root cause and track that. Them over time you build up data on if it's worth it to stay on one platform or not. Eg doing this let's you build a risk profile to counter the $/time argument against. Then you get to make an informed decision. You don't just shruggie and move in "that ship has sailed" at least if you're doing it right.

msla 8y ago

Alternatively: Our service was down. Why? Because the platform it runs on went down. Why only one platform? Because management declined to approve purchase orders for sufficient hardware to allow redundancy. --- And your career just hit a wall. Passing the buck up the chain tends to do that.

sfifs 8y ago
So you don't word it in a way to assign blame. You say previous cost benefit analysis indicated X known trade off from lack of redundancy. This incident either confirms the trade off (finance will usually support you for this) or say additional X factor not previously known needs to be taken into the trade off and revised.
- therein 8y ago
  That indeed sounds much better.
- LaGrange 8y ago
  Plot twist: X factor was known but dismissed by management. Management knows this and knows how to read between lines. Your career hit a brick wall and you didn't even get to openly snub the management.
  - roenxi 8y ago
    You're pushing the scenario into foolish territory. If management decisions are the root cause of a problem and they won't accept that, then the quality of the process used to identify the root cause is irrelevant. You could use the best or worst process and it wouldn't matter. If your management is bad, then there are a lot more risks to your career than trying to work with them to improve the process. It is more likely that a given employee has dead-ended their career because they don't know how to explain a problem without blaming people and having the humility to see from other people's point of view; both useful skills for getting a promotion.
    - 8y ago

8note 8y ago

how do make effective action items with that root cause?