6 ms·

You can't end a 5 Whys with 'another team I depend on screwed up'. You have to own your software or service. If you depend on something, you need checks, alarms

• CobrastanJorji 8y ago
  Sometimes a problem that affects you is outside your immediate ownership, and the right answer is to escalate and seek to fix it. If you limit yourself to only considering changes that your immediate team can make on its own without consulting others, you're doing your organization a disservice.
  • awill 8y ago
    sure, if your service behaved as well as possible during your dependency's outage, yes.
• perlgeek 8y ago
  > You can't end a 5 Whys with 'another team I depend on screwed up'. You have to own your software or service. Our service was down. Why? Because the platform it runs on went down. Why only one platform? Because management doesn't want us to spend several years engineering for multiple platforms. --- I find it hard to imagine a realistic path of 5 whys that leads us back to a cause within our own team.
  • Karrot_Kream 8y ago
    > Because management doesn't want us to spend several years engineering for multiple platforms. Yes and that's fine. It's okay to end up with an RCA like that and note that as a known risk. If a cost-benefit analysis shows that the risk is dwarfed by the solution, then it's not worth fixing. Knowing the existing risks of your system and reevaluating these risks in the face of changes (more people available, inefficient context switching, etc) is a key component of making progress.
    • awill 8y ago
      True. In a case this simple, you likely wouldn't have to write an RCA. However, most real life situations I've seen show that there's always room to improve. For example, how quickly did your service recover once the underlying platform recovered. Rare events (like DC failure) can be a bad thing for operational training. Teams forget how to recover. No runbooks, no training, or tooling that's infrequently used no longer works.
      • grogenaut 8y ago
        You actually want to assign to root cause and track that. Them over time you build up data on if it's worth it to stay on one platform or not. Eg doing this let's you build a risk profile to counter the $/time argument against. Then you get to make an informed decision. You don't just shruggie and move in "that ship has sailed" at least if you're doing it right.
  • msla 8y ago
    Alternatively: Our service was down. Why? Because the platform it runs on went down. Why only one platform? Because management declined to approve purchase orders for sufficient hardware to allow redundancy. --- And your career just hit a wall. Passing the buck up the chain tends to do that.
    • sfifs 8y ago
      So you don't word it in a way to assign blame. You say previous cost benefit analysis indicated X known trade off from lack of redundancy. This incident either confirms the trade off (finance will usually support you for this) or say additional X factor not previously known needs to be taken into the trade off and revised.
      • therein 8y ago
        That indeed sounds much better.
      • LaGrange 8y ago
        Plot twist: X factor was known but dismissed by management. Management knows this and knows how to read between lines. Your career hit a brick wall and you didn't even get to openly snub the management.
        • roenxi 8y ago
          You're pushing the scenario into foolish territory. If management decisions are the root cause of a problem and they won't accept that, then the quality of the process used to identify the root cause is irrelevant. You could use the best or worst process and it wouldn't matter. If your management is bad, then there are a lot more risks to your career than trying to work with them to improve the process. It is more likely that a given employee has dead-ended their career because they don't know how to explain a problem without blaming people and having the humility to see from other people's point of view; both useful skills for getting a promotion.
          • 8y ago
  • 8note 8y ago
    how do make effective action items with that root cause?
• rakoo 8y ago
  Where does it stop though? Because in that case all externalities will become a "I trusted that dependency but it failed, why did I trust it and didn't work around it?".
  • kevingadd 8y ago
    the causes of failures that you explicitly enumerate which were under your control aren't always You Did A Bad Thing. For example, "I trusted this dependency which broke" doesn't mean you Did Something Bad, it just means that the cause of the failure was the lack of a way to validate the dependency and fall back on a known-good version. A good example of this is build pipelines relying on the ability to download npm/nuget/debian packages from a remote server instead of vendoring them locally - this is an intrinsically unreliable choice to make. The remote server being unreachable is not your fault, but there are steps you can take to fix this problem in the future. Avoiding external blame in your 5Ys (without ignoring the external factors) encourages you to make local changes. As always you have to apply reason here and not adhere to dogma, but it's a useful dogma.
  • dtech 8y ago
    It's okay to conclude after a root cause analysis that the root cause is not something you want to work to prevent, because the cost-benefit is skewed. For some products a AWS S3 [1] outage might be a influential enough that they add a redundant second storage at another provider, for other it might be "this is too expensive and time consuming to work around for the problems it caused, let's hope Amazon gets their shit together" [1] https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/
• danso 8y ago
  This reminds me of the Twitter [0]/HN [1] thread by someone trying to argue that the Boeing 787 MAX couldn't be blamed on software engineers, because it was the aerodynamic/mechanical/systems engineers and business executives who demanded too much (and also, the pilots for just not being better trained). Yes, that's all technically true. We could even put more blame on the human desire for convenient speedy travel as more of a direct factor. But software engineering is part of systems engineering for any modern project. We should want and expect software engineers to have the agency and bigger-picture thinking to do more than just design for inadequate inputs. [0] https://twitter.com/trevorsumner/status/1106934369158078470 https://twitter.com/trevorsumner/status/1106934369158078470 [1] https://news.ycombinator.com/item?id=19414775 https://news.ycombinator.com/item?id=19414775
  • jdietrich 8y ago
    The buck stops with senior management; the buck always stops with senior management. That is the essential lesson of the 5 whys and the essential problem with trying to use it without being fully committed to the Toyota Production System. If senior management aren't willing to cop the blame for literally everything that ever goes wrong, the whole effort is just a dog-and-pony show. https://en.wikipedia.org/wiki/Toyota_Production_System https://en.wikipedia.org/wiki/Toyota_Production_System
    • dtech 8y ago
      I disagree. If an outage in the product your team managed was caused by a bug your team introduced 5 can point at several root causes and solutions that your team will be able to control and implement. Management only comes into play when causes like "under-funding", "constantly changing requirements" or "too tight deadlines" are identified.
      • vajrabum 8y ago
        Management also has to support the idea of blamelessness and the costs of investigation and remediation and to participate in reviews. Without that support it may work some of the time but often won't.
• danbolt 8y ago
  I think what you're touching on is the crux of the article too. Depending on the "why" that gets delivered, you end up with very different conclusions.
• empath75 8y ago
  But that would be 6 whys.
• BeetleB 8y ago
  >You can't end a 5 Whys with 'another team I depend on screwed up'. You have to own your software or service. If you depend on something, you need checks, alarms, and whenever possible the ability to handle dependency failure well. On the big picture, I agree with GP - different people will come to very different conclusions with the "Why" game - because the world is complex, and many factors come into play when something goes wrong. You don't need to fix all of them, but it does become a game of figuring out the most convenient aspect to fix (and no, another level of "Why" won't fix that). Somewhat related: I had a 2nd level manager who in meetings kept saying "In an ideal world, ..." and using that as a starting point for requirements, problem solving, etc. At one point, I interrupted him and said "Nope. In an ideal world, I would not need to work for a living. Your ideal is already far from it, so let's not make distinctions between ideal and non-ideal."[1] It's easy to use the "whys" to come up with answers like "Because X needs to make a living" and "Because the incentives with the manager/department/org/company create a situation where testing is devalued (despite all claims by management to the contrary)" or "Because we cannot retain talent due to our compensation policies" or "Because people in the team are unwilling to learn version control systems newer than cvs." These are not facetious answers - they should be treated on an equal footing with technical solutions. In my experience, management that does not want to deal with solutions that involve other teams/people/policies are ones where the work has been miserable. It's also been my experience that SW people seem to prefer technical solutions over alternatives, which is tragic. When you study things like negotiation, it's almost inverted. They have their own push for "Why", but the push is usually in the motives direction. The person you're dealing with is presenting a concrete demand that may seem objective, but behind it is almost always a fairly human reason (not looking bad, higher status, etc). And they always emphasize trying to root cause to those and addressing them. Many here know the saying: In the top tech companies, all problems are social ones. They have the talent to achieve anything. If any piece of SW fails, it's not because they didn't have technically capable people. It's because they failed in the social/organizational domain. So when I hear management say they don't want to address that in the 5 Why's, I see management that is solving the wrong problem. I took a systems engineering class once. The thing they emphasized throughout is "If every aspect of a product is designed perfectly as its own unit, you'll get a crappy product that will not win in the marketplace." The idea is that a perfect widget in isolation may not integrate well, and there are budgetary issues as well. There has to be a certain amount of compromise to integrate with other widgets in the product, and the teams involved need to talk to one another and have dependencies on one another. The role of the system engineer is to oversee that this is happening. Insisting that your SW or service must be bulletproof of other services, when both are related and in the same product, is highly suboptimal (as is the other extreme). Often the optimal solution is to say "Our service will work, provided service Y works". [1] I was on good terms with him so I got away with it.
• mikekchar 8y ago
  I think there is an underlying misunderstanding with the application of "5 whys" which is a bit like the misunderstanding that people have with prioritisation. Let me diverge a bit with prioritisation: a lot of people know about the story of how to fit large rocks, medium sized rocks, small rocks and sand into a jar. You start with the largest size and work your way down to the smallest size (sand). The smaller rocks fit in the spaces of the large ones. The moral of the story is to look at the items that are most important as being large rocks and the ones that are being least important as being sand. And by doing the most important things first, you'll get everything done. Because... um... lesser important tasks fit into the spaces of more important tasks??? What??? No. This is a bad analogy that looks like's related, but isn't related in any way! Getting back to the question at hand. "5 whys = 1 how". Because if we do the root cause analysis, we will definitely find a solution to our problem. What???? Because there is always a solution to our problem??? It's that same skipping over of the most important detail: you aren't doing the most important problems first because that way there will be space for the lesser important problems. You are doing the most important problem first because then when you run out of time, you've done the most important thing! Similarly, you aren't finding a solution when you are doing root cause analysis, your are finding problems. 5 whys is a tool for helping search the problem space. That is all. The "5 whys = 1 how" quote is really unfortunate and I think it's taken completely out of context. You don't just do 5 whys once and say, "Oh now we know how to fix our problem". You keep doing the 5 whys over and over and over again so you can identify your problems. And at that point you can start to get a handle on your solutions. "5 whys = 1 how" doesn't mean there is only "1 how" that you need. And just to sum up, if one of your 5 whys ends up with "An external entity failed us", then you might need to find a way that you aren't relying on that entity. However, run some other 5 whys to help you search the problem space a bit more. You may find another "how" that will help you better.