7 ms·

As someone who used 5-whys when I worked at Amazon, I love it and the criticisms here don't resonate. It's not about finding one path of causation, or finding e

• kqr 5y ago
  > As someone who used 5-whys when I worked at Amazon, I love it and the criticisms here don't resonate. It's not about finding one path of causation, or finding everything you can point a finger out. It's about being willing to admit the actual causes and then do something about them. You're right, of course. To do causal analysis at all, you have to do it respectfully and well. That's the first step. When you have the first step down, there are other analysis methods that are not five whys that does this better. One of my favourites is Leveson's causal analysis based on systems theory (CAST). These dig wider as well as deeper, and offer more interesting questions to ask along the way rather than just "what caused this?"
  • ajkjk 5y ago
    imo, the question "Why?" is really supposed to be more like "How could this have happened!??". It really makes a lot more sense that way. [edit: to respond to the dead comment about what else "Why" could mean, the difference is the emotional valence. It's not asking the _factual_ question of how it happened; it's applying the appropriate outrage and embarrassment to the fact that it happened. "My god, we messed up! How can we ensure it never happens again, with urgency?".]
    • kqr 5y ago
      Sure, but even better is to be more explicit: - Which business constraint was violated? - Why is that an important constraint? - What control mechanisms did we have in place to prevent that from happening? Have they prevented the constraint from being violated before? Why where those mechanisms insufficient this time? - What control mechanisms didn't we have? Why not? And so on. You get fan-out of much higher degree that way, but it also, in my experience, is a much more efficient way to search the causal network.
      • ajkjk 5y ago
        I don't really agree. The model you describe presupposes a functional framework for decision making -- constraints are in place and respected; controls exist, etc. Most of the time the result of the Whys process is, in my experience: "oh, we really need a framework for constraints/controls/etc around this". I guess it looks very different depending on the business you're in and how codified everything already is. If you're investigating failures in, say, a manufacturing process, everything is pretty rigid already and you're finding problems with the rules you already have. If you're investigating failures in a hodgepodge software engineering organization, the problem is that there is no rigidity in the first place, and those processes need to be created from scratch. I'm mostly familiar with the latter, and usually the result of whatever you come up with is a place where leadership ought to be directly applied. (I'd hazard to claim that: if you do a 5-Whys in an organization and the leader of the organization doesn't see something that they can go fix directly -- not by delegating but by leading, themselves -- either you did it wrong, or they're a bad leader.)
• adamsmark 5y ago
  I worked at Amazon too. This outcome of a 5 why, "Why?" "Because our culture has product managers demanding things with short deadlines and no concern about the code quality." Does not sound like anything would change. I could chalk up a lot of failures, missed goals or mistakes to this culture of shipping the bare minimum because that's all the time you have. Sometimes it works well, others lead to tech debt that metastasizes because the engineers don't have to use what they built leading to instances like pricing errors that lost significant amounts of money. Working at Amazon sometimes felt like everyone was gaming the system to hit impossible numbers/deadlines - Goodhart's law at scale.
  • shepherdjerred 5y ago
    Hah. I cannot count the number of times we made action items as a result of five COEs and the ultimate result was a SIM issue permanently sitting in our queue because there was always something more important to do.
    • 8note 5y ago
      That makes sense. I remember some principal saying "COEs are a chance for engineers to drive work priority" at a coe review.
  • alisonkisk 5y ago
    Amazon has the most serious "business operations" of the FAANGish companies. probably from their roots as low margin retailer like Walmart. Google is creative and technically smart, but always was drowning in ad revenue so had plenty of room for slack everywhere. Apple had the creativity and attention to detail in the experience. MS could do everything -- product, engineering and legal department -- good enough to protect their monopoly moat.
  • ajkjk 5y ago
    It worked pretty well in my org (fulfillment/supply-chain) but maybe I just got lucky. As you might imagine that org was pretty functional, being the backbone of the retail business.
• stevebmark 5y ago
  To me your message embodies the criticism of the five whys. You don't run a five whys analysis because it's not effective. You run a more open ended root cause analysis with the goal of understanding issues, which is effective.
  • ajkjk 5y ago
    I believe that what I'm talking about is what "five whys" _actually_ is, and what a lot of the critics think "five whys" is, which they are rightfully criticizing, is just not a thing at all -- it would be trivially useless and ineffective, and obviously is so to anyone who does it. In particular 'five whys' when run correctly has neither a set number of whys, nor a particular linear structure, nor a fixation on assigning blame, and involves actually taking actions in response to the root causes, not just creating meaningless action items. If those don't apply, you're not doing five whys, you're doing some bizarre aberration of it.
    • stevebmark 5y ago
      To me that sounds pretty clearly like not the five whys: there aren't five whys, they aren't linear, and you're not asking "why" at every step. The history of the five whys and Toyoda make it clear that it's 5 linear whys. You're performing a root cause analysis, which I suspect is why you find it effective. VS the five whys are not an effective analysis tool. Maybe where we see things differently is you consider the non-five-whys analysis part of the five whys, and I consider it "root cause analysis". For me, an issue with still calling it the "five whys" is watching other folks less familiar with how to do root cause analysis using the low quality and awkward "five whys", without any clear direction that they shouldn't use that strategy because it's not effective.
      • ajkjk 5y ago
        Yeah, I guess what I'm describing is "the thing we did in the Amazon COE process that we called 5 whys". Whatever that was called, it's definitely better than what you're describing.
• _ea1k 5y ago
  > "Why?" "Because our culture has product managers demanding things with short deadlines and no concern about the code quality." I find answers like this to be the real downfall of a five whys approach. Is it really true that product managers have "no concern about the code quality"? At best, that seems like a statement that the product managers will not accept, but even worse, it isn't directly actionable. I'd prefer something like, "Because our teams are not given n days for quality assurance before feature delivery", or "because our estimate of 21 days to deliver was replaced with 14 days without our input". If it isn't measurable, how can you make the needed changes? "Care more" isn't quantifiable.
  • RealityVoid 5y ago
    At that point, it's not actionable either, it's just blame shifting.
  • passivate 5y ago
    Indeed, the five Why's is just one of the starting points for introspection. Each person/team/department/org needs to find ways to generate consensus and actionable items out of that exercise. Reactionary phrases like "because x sucks" aren't very useful.
    • stefan_ 5y ago
      Oh good, we have arrived at the no true scotsman phase. It's not that scrum and agile aren't working, your team just hasn't introspected enough!
      • bumby 5y ago
        Talk to industrial engineers and they’ll tell you these exercises are just a starting point to try and understand the problem. We all like to pretend there are always simple solutions (if I had a nickel for every time somebody told me the root problem was ‘we just need more people’…) but I’m practice, process improvement is often messy and convoluted until you drill deep into the system
      • 5y ago
• CobrastanJorji 5y ago
  I did a Five Whys for Amazon once. It had been an embarrassing incident for the team. We had put a huge new feature together over a few months, and when we turned it on...nothing happened. The whole feature had been entirely disabled by a typo I made in a configuration change. At the time, we had a test team that was tremendously overworked and tested things almost entirely manually. They asked for a couple extra weeks to test our new feature but eventually greenlit it. My Five Whys basically asked "how did this get to production if it didn't do anything?" My manager handed it back to me and said "this takes the blame off of you for this mistake and puts it on the testing team. Ask different questions." So I wrote another one that was 100% focused on me. "Why did this happen? Because the config was wrong." "Why was the config wrong? Because I made a typo." "Why didn't I notice the typo? Because I didn't run the feature after making the change." "Why didn't I try it out? Because the feature is very difficult to invoke or observe." "Why is the feature hard to invoke directly?" "Because <x,y,z>." What I learned from that is that the problem with Five Whys is that there are many different branches of questions that are worth asking, but that design only allows for exactly one path. Choosing which one is an editorial decision on the part of the author. The outcome of a 5-Whys document is, by its nature, one change. But most of the time, outages at mature tech companies are the result of several different problems coincidentally lining up (if exactly one thing going wrong caused a major problem, that itself is the second problem), and a postmortem process really needs to examine each of the things that went wrong separately.
  • RealityVoid 5y ago
    I has a similar experience where I had to 5 why a issue and... the gist of it was that the why was because I bungled it up. I just.. saw it as an issue while I wrote my stuff, added a TODO and then... just never did it. So then, after the first 2 why's I could only answer in the back of my head, "Because I'm fucking stupid, that's why!" I get the idea with the why's but sometimes, it's just that simple, you can't really go deeper than that. I fucked up.
    • chucksmash 5y ago
      I think you can still usefully push past the "I just never did it." If the problem is that you are overcommitted and things are starting to slip, that's a different problem than "I got distracted by something shiny and forgot." In the first case, it might make sense to try to grow the team, for instance.
• necovek 5y ago
  Like any tool, it can easily be misused. The post in question is a good demonstration of how you can misuse the 5-whys, and it gets well deserved criticism for that. It's obviously not getting anywhere 5 levels deep, so it's asking the wrong questions, and honestly, giving very bad answers that don't lead anywhere: 5-whys require a specific mindset first. Thus, the final "root cause" is very suspect and there's one too many "feel" and "think" in there. To illustrate, the answer to #2 of "Because I am constantly tempted to check my email, rather than work on important projects." is lacking in substance to answer "why are you getting distracted" (it says the same thing in slightly different words). You can go on-and-on in the same vein. "Because my email notifications pop up and I check them out", "Because when I check my email notifications out, I read the email and now I've lost my context and got immersed into the email topic instead"... That's not helpful at all. As soon as you get to "I get easily distracted by email", that's already a problem worth solving ("root cause"). And it can have multiple solutions: 1. if you are a blocker for something, move that to an email list of people, a "board" or "forum" discussion; 2. if you are not a blocker, stop notifications and set checkpoints for when you'll read email during the day, and open a side-channel for urgencies if it's ever needed. If that's not the root cause (it is actually expected of you to be responsive even if you are not a blocker to anything), only then is it worth digging deeper to find problems in the team that make this so. Of course, I am working with the assumption that being extremely responsive to email is really hurting this person's work (they were not hired to be responsive to emails, eg. as a technical architect to support a development team).
• beebmam 5y ago
  >"Why?" "Because our culture has product managers demanding things with short deadlines and no concern about the code quality." If you want the 5-whys to not be a joke, you certainly shouldn't be arriving at such strong conclusions without an enormous amount of evidence. There's likely many other explanations. A serious answer to a "why" question requires enough evidence that it could not be explained in any other way. If we just want to arrive at strong conclusions that fit in with our priors, so be it, but at least be honest that you don't care about the truth.
  • bumby 5y ago
    When the 5 whys technique is used for actual process improvement, data collection is part of the process to measure improvement of refute a claim. The next steps usually test multiple theories (sometimes concurrently).
• aflag 5y ago
  Why did you leave Amazon?
  • ajkjk 5y ago
    I was 25 and it felt like a weird thing to be doing in my 20s when other people were traveling the world.
    • aflag 5y ago
      Couldn't you travel while still being employed by Amazon?
      • ajkjk 5y ago
        Not for months at a time, which is what I wanted to do (and did). Also, no, remote work wasn't really a thing at the time anyway. But more importantly I don't believe in being continuously employed your whole adult life. It feels dumb and unhappy.
• snthd 5y ago
  >management is incentivized to care about things other than shipping/deliverables/this quarter's returns. Could someone elaborate on how that's done?
  • ajkjk 5y ago
    Management has to be rewarded by _their_ management for delivering on abstract improvements that make things less bad without necessarily pegging them to metrics and returns. Their management has to be rewarded by the managers above them, etc, all the way up to.. somebody. Ideally the CEO because the board ain't gonna do it. Bezos was great at this: it was always clear that fixing your shit was the first priority, because the C-suite understood that for a company that large to scale its labor, teams have to not be blocking / causing friction for each other. Hence being very forward-thinking about abstracting everything out into service-oriented architecture. (although it took like two decades to actually get there completely. My org, fulfillment, was one of the last to do it, and only finished after I left.)
• timoth3y 5y ago
  The Five Whys are a great as a tool for personal introspection, but in business it seems to become an exercise in chaotic story telling, with the final "answer" highly dependent on the initial assumptions. Let's take this in a different direction: > "Why did our code break?" "Because we habitually ship things without thoroughly testing them." .... or "Because Bob didn't write proper tests for his patch, so the bug it introduced went undetected" It obviously would suck to be Bob here, but the point is there is no single right answer to any of the whys and each choice along the way will lead us down very different narrative paths. You can't achieve greater certainly simply by chaining multiple uncertain statements together.
• 908B64B197 5y ago
  > Then we did none of the things suggested. Well, there's no point. Just quit. You're going to be mediocre forever until the management chain buys in. The reason Amazon scales well and can do so many things is that that their management is incentivized to care about things other than shipping/deliverables/this quarter's returns. It's still about the business overall, but Bezos' long-term-thinking has infected the entire company with long-termism and it's very effective. There's a lot of cargo culting around Amazon, but having a strong engineering culture and technical leadership is almost never what companies choose to emulate. True engineering orgs are quite rare (and will often ship disproportionately compared to their competitors. Just think of pre-merger Boeing for instance). > I am quite sure that Google/Microsoft/etc do not have this quality in the same degree (or they would not have made many of their obvious mistakes). Not that Amazon doesn't make mistakes; they just make different kinds than a lot of other companies I'm interested what are the obvious mistakes you are thinking about.
• paulryanrogers 5y ago
  > The reason Amazon scales well and can do so many things is that that their management is incentivized to care about things other than shipping/deliverables/this quarter's returns. It's still about the business overall, but Bezos' long-term-thinking has infected the entire company with long-termism and it's very effective. Is comingling inventory among sellers long term thinking?
  • ajkjk 5y ago
    I actually worked on that project, and yes, it is. It was a good idea whose implications were not fully understood -- but Amazon will eventually get it right and it will stop causing problems, because they have a good 'make mistakes and fix things' loop. (I think this has already started happening but this moves very slowly.) The main reason for commingling inventory is to be able to ship a product from the closest warehouse if it's the same as something in another warehouse (if you order it on the east coast and the nearest Amazon-owned inventory is on the west coast but an FBA seller has it on the east coast, ideally you could fulfill the order from the east coast). The problem is, nobody anticipated the degree to which sellers began cheating this system when they figured it out. I don't think Amazon was ready to fully moderate the whole system against exploitation. They are no doubt working on it but it's a huge battle.