3 ms·

Usually another team's failure is covered by their own independent report. That simplifies creating the report since you don't need to collaborate closely, but

Usually another team's failure is covered by their own independent report. That simplifies creating the report since you don't need to collaborate closely, but also prevents shifting the blame on to anyone else (because really, both teams had failures they should have caught independently). E.g. as the last why: Why did the testing team not catch that the feature was not functional? This is covered by LINK

CobrastanJorji 1y ago
If a root cause analysis is not cross team, how deep can the analysis possibly be? "Whoops, that question leads to this other process that our team doesn't directly control, guess we stop thinking about that!"
- lijok 1y ago
  Pretty deep. It forces you to account for failures in other domains
- taeric 1y ago
  If your root cause is cross team, then you wind up having to make some implicit assumptions on what the other team could have done. Is akin to ending with "because the gods got angry." Not really actionable. This is a classic "limit the scope of the feature." You want the document to be written and constrained to someone that is in a position to impact everything they talk about. If you think there was something more holistic, push for that, as well. Note you can discuss what other teams are doing. But do that in a way that is strictly factual. And then ask why that led your team to the failure that your team owns.
  - ghurtado 1y ago
    > on what the other team could have done If you're wondering what anyone "could have done", you've already missed the point of the article completely.
    - taeric 1y ago
      I was just commenting further on why you constrain it to your team/area of control. Should have been more clear that I meant my comment as a plus one to some of the other replies.
      - ghurtado 1y ago
        Ah, thank you and sorry for assuming.
  - hinkley 1y ago
    If the dynamic is cross functional you need to reschedule the post mortem and invite the other team to the meeting. This is literally a "the right people aren't in the room" issue.
- grogers 1y ago
  If your team doesn't control it, should you be thinking about it? Or should the team that owns it also own fixing it? I should also have stated that based on the context I assumed this was talking about in incident report meant to be consumed internally, which I believe should be one per team. Incident reports published externally should be one single document, combining all the RCA from each individual report.
  - mitthrowaway2 1y ago
    Shouldn't you be thinking about it? Whether you can control it or not, if you need to rely on it, you should be thinking about it. Otherwise when an airplane crashes because of a defect in the aluminum, the design team's RCA will have to conclude that the root cause is "a lack of a redundant set of wings in the plane design", because they don't want to pin the blame on the materials quality inspection team mixing up some batch numbers. If you were relying on something not to fail and it failed, your RCA should state as much. At best in the GP's case they could say "it's our fault for trusting the testing team to test the feature".
  - hinkley 1y ago
    This sort of thinking is why the Japanese obliterated Detroit in the late 70's and early 80's. The idea that if it was obvious another team messed up, you'd just ignore the problem until it got audited by a cross-functional team. All of the time and effort and materials spent between the two being wasted because nobody spoke up.
  - lazyasciiart 1y ago
    So what do heartbleed and log4shell RCAs look like for your internal teams? “A necessary source library screwed up, not our problem”?
  - daxfohl 1y ago
    I think that's a valid question, and has plenty of ways you could go. Is the process set up so that it's literally "throw it over the wall and you're done unless the test team contacts you"? Then arguably not. You did your job e2e and there was nothing you could've done. Doing more would've disrupted the process that's in place and taken time from other things you were assigned to. The test team should've contacted you. BUT, well now the director has egg on his face and makes it your problem, so "should" is irrelevant; you will be thinking about it. And you ask yourself, was there something I could've done? And you know the answer is "probably". Then, the more you think about it, you wonder, why on earth is the process set up to be so "throw it over the wall"? Isn't that stupid? All my hard work, and I don't even get to follow its progress through to production? Is this maybe also why my morale is so low? And the morale of everyone else on the team? And why testing always takes so long and misses so many bugs? And then as you start putting things together, you realize that your director didn't assign this to you out of spite. He assigned it to you to make things better. That this isn't a form of punishment, but an opportunity to make a difference. It's something that is ultimately a director-level question. Why is the process set up like it is? The director could put together and solve with adequate time, but at that level time is on short supply, and he's putting his trust in you to analyze and flesh out, what really is the root cause for this incredibly asinine (and frightening) failure, and how can we improve as a result? That said, in an org so broken that something like this could happen, I'm guessing the director is wanting you to do the RCA and the ten other firedrills that you're currently fighting as well, in which case, eh, fuck it. Blame the other team and move on.
- lazyasciiart 1y ago
  I oversaw an RCA once where this is exactly where they stopped. “We didn’t write this code originally so it isn’t our fault that our new feature broke it”. Repeat for 30 minutes whenever anyone says anything. We gave up.
hinkley 1y ago
> root You keep using that word. I do not think it means what you think it means.

CobrastanJorji 1y ago

If a root cause analysis is not cross team, how deep can the analysis possibly be? "Whoops, that question leads to this other process that our team doesn't directly control, guess we stop thinking about that!"

lijok 1y ago
Pretty deep. It forces you to account for failures in other domains
taeric 1y ago
If your root cause is cross team, then you wind up having to make some implicit assumptions on what the other team could have done. Is akin to ending with "because the gods got angry." Not really actionable. This is a classic "limit the scope of the feature." You want the document to be written and constrained to someone that is in a position to impact everything they talk about. If you think there was something more holistic, push for that, as well. Note you can discuss what other teams are doing. But do that in a way that is strictly factual. And then ask why that led your team to the failure that your team owns.
- ghurtado 1y ago
  > on what the other team could have done If you're wondering what anyone "could have done", you've already missed the point of the article completely.
  - taeric 1y ago
    I was just commenting further on why you constrain it to your team/area of control. Should have been more clear that I meant my comment as a plus one to some of the other replies.
    - ghurtado 1y ago
      Ah, thank you and sorry for assuming.
- hinkley 1y ago
  If the dynamic is cross functional you need to reschedule the post mortem and invite the other team to the meeting. This is literally a "the right people aren't in the room" issue.
grogers 1y ago
If your team doesn't control it, should you be thinking about it? Or should the team that owns it also own fixing it? I should also have stated that based on the context I assumed this was talking about in incident report meant to be consumed internally, which I believe should be one per team. Incident reports published externally should be one single document, combining all the RCA from each individual report.
- mitthrowaway2 1y ago
  Shouldn't you be thinking about it? Whether you can control it or not, if you need to rely on it, you should be thinking about it. Otherwise when an airplane crashes because of a defect in the aluminum, the design team's RCA will have to conclude that the root cause is "a lack of a redundant set of wings in the plane design", because they don't want to pin the blame on the materials quality inspection team mixing up some batch numbers. If you were relying on something not to fail and it failed, your RCA should state as much. At best in the GP's case they could say "it's our fault for trusting the testing team to test the feature".
- hinkley 1y ago
  This sort of thinking is why the Japanese obliterated Detroit in the late 70's and early 80's. The idea that if it was obvious another team messed up, you'd just ignore the problem until it got audited by a cross-functional team. All of the time and effort and materials spent between the two being wasted because nobody spoke up.
- lazyasciiart 1y ago
  So what do heartbleed and log4shell RCAs look like for your internal teams? “A necessary source library screwed up, not our problem”?
- daxfohl 1y ago
  I think that's a valid question, and has plenty of ways you could go. Is the process set up so that it's literally "throw it over the wall and you're done unless the test team contacts you"? Then arguably not. You did your job e2e and there was nothing you could've done. Doing more would've disrupted the process that's in place and taken time from other things you were assigned to. The test team should've contacted you. BUT, well now the director has egg on his face and makes it your problem, so "should" is irrelevant; you will be thinking about it. And you ask yourself, was there something I could've done? And you know the answer is "probably". Then, the more you think about it, you wonder, why on earth is the process set up to be so "throw it over the wall"? Isn't that stupid? All my hard work, and I don't even get to follow its progress through to production? Is this maybe also why my morale is so low? And the morale of everyone else on the team? And why testing always takes so long and misses so many bugs? And then as you start putting things together, you realize that your director didn't assign this to you out of spite. He assigned it to you to make things better. That this isn't a form of punishment, but an opportunity to make a difference. It's something that is ultimately a director-level question. Why is the process set up like it is? The director could put together and solve with adequate time, but at that level time is on short supply, and he's putting his trust in you to analyze and flesh out, what really is the root cause for this incredibly asinine (and frightening) failure, and how can we improve as a result? That said, in an org so broken that something like this could happen, I'm guessing the director is wanting you to do the RCA and the ten other firedrills that you're currently fighting as well, in which case, eh, fuck it. Blame the other team and move on.
lazyasciiart 1y ago
I oversaw an RCA once where this is exactly where they stopped. “We didn’t write this code originally so it isn’t our fault that our new feature broke it”. Repeat for 30 minutes whenever anyone says anything. We gave up.

hinkley 1y ago

> root You keep using that word. I do not think it means what you think it means.