12 ms·

More than five whys and “layer eight” problems

317 points by zdw 4y ago

quickthrower2 4y ago
If you go below the "tech whys" and into the organizational whys, then you are thinking at the team leader or VP Engineering level. I think you want to be eyes opened to the full list of whys, but be happy that some cannot be changed. After all you are hired by a business, run by people, not a perfectly functional institution.
csours 4y ago
Yup, and for this reason, I will never work on a compliance app again in my life. Compliance is all about reducing risk to an acceptable level, but if you care about it, it is also about reducing risk to as close to zero as possible. Compliance generally affects multiple organizations, and that means convincing a bunch of people to do something that doesn't help them accomplish their primary goals. To be clear, I'm not alleging any wrongdoing whatsoever against any group that I worked with. I'm simply saying that it's hard enough to get people to do the job they accept as their main job; adding 'extra' work is never a rewarding experience - especially as an individual contributor.
- stakhanov 4y ago
  ...yeah, there are several roles/functions that are by definition thankless tasks. In compliance, the best you can produce is nothing, as in no incidents that could have been prevented through better compliance, while causing a ton of cost and frustration across the organization. The worst you can produce is ruining the company and sending people to jail in a way that you were tasked with preventing and could actually have prevented. It sort of reminds me of this thought experiment that Nassim Nicholas Taleb mentions in one of his books: What if a bureaucrat had figured out, pre 9/11, that terrorists flying a commercial airliner in to a building was a risk worth trying to prevent and had passed legislation for all cockpits to be bulletproof cells that could only be opened from the inside, or something like that. 9/11 would have never happened, and that bureaucrat would have caught shit for the rest of his life for championing legislation that is causing a lot of people a lot of cost and frustration and has never had any observably positive impact anywhere in the observable universe.
nl 4y ago
> The car didn't start... because the battery is dead... because the alternator wasn't charging it... because the alternator belt broke... because the belt was beyond its useful life but wasn't replaced... because it wasn't maintained according to recommended schedule. This isn't "five whys" except in the most superficial sense. It can be useful to do this to build reliable technology but it isn't what five whys is trying to solve. > because the battery is dead... because the alternator wasn't charging it... because the alternator belt broke... because the belt was beyond its useful life but wasn't replaced... All of these are a single "why" (call it "technical faults" or something). > It turns out that when you start doing this root-cause analysis and really keep after it, the "squishy human realm" is actually the no-longer-hypothetical "layer eight" from those T-shirts. Is this actually a revelation to anyone? These hard questions are exactly what five whys is designed to address.
debacle 4y ago
I have a socket wrench. It's a really good socket wrench, it would probably cost me close to $200 to replace. I use it for nearly all of the bolts and lags that I need to loosen or tighten. The bolts on the spindles for my tractor deck are massive, they rust up quickly, and my $200 socket wrench doesn't touch them. For those spindles, I have a special tool that can solve the kind of problems that a $200 socket wrench can't solve. Despite my socket wrench not solving all problems, I'm still really glad I have it, because it is helpful most of the time, and I trust myself to be smart enough to know when I need a different tool.
- electromech 4y ago
  > Despite my socket wrench not solving all problems If only, amirite?!
ggm 4y ago
To me, the real layer-8 in a car model is that you probably shouldn't have built out society to depend on cars in the first place. Thats why we have the idea of the 15 minute city: places you can manage without private transport. Applied to s/w this would be "do we really need to make this system, in the first place: How about if we decide not to"
- viburnum 4y ago
  It is so satisfying when you can simply delete a bunch of code rather than fix it.
  - ggm 4y ago
    I've struggled to take my teams (in the past: I don't do PM now) beyond "ship it" to "kill it": the C-suites don't like the idea of stopping doing things. They have a point, we maybe don't have community permission, but we should at least entertain the idea (in the not for profit sector) People rei-ify things. "oh we need to do X we've always done X" when in fact, you could say "doing X is stupid"
- azmarks 4y ago
  Yes. This is a system problem. Why are cars allowed to be near pedestrians? Why were cities/streets designed to make it possible for cars to run over pedestrians? Saying it was an inattentive driver misses the point that we can't expect fallible people to not make mistakes. We should be designing systems to minimize the damage from inattentive drivers and drunk pedestrians
darksaints 4y ago
I’ve always been bothered by the five whys, simply because it’s such a dramatic oversimplification of cause and effect. The fatal flaw is the assumption that causality is linear, when it is more often combinatorial in nature. To illustrate consider a drunk pedestrian jaywalking and being hit by a speeding driver who was texting while driving. Who caused the accident? They both did, together. Or maybe one of them did and the other didn’t. Or maybe the stars aligned just right and both would have been fine if they hadn’t been doing it at the exact same time? I can’t help but feel like anybody who has found the “root cause” for any problem, their answer will reflect their biases far more than will reflect true causality. If you can come up with a single answer for a complex causality problem, something hints to me that you would probably point to the same “cause”for other problems too.
- madrox 4y ago
  There's a common fallacy in your example in that you're asking "who" caused the accident and not "what." This is why blameless retros are important. Whenever you're dealing with who, the responsibility is never binary. There's always some responsibility someone can take for a situation they're a part of. That's good advice for life in general. The intention of the five whys is to find breakdowns at each level...not just the last one, and mitigate each.
- efsavage 4y ago
  > If you can come up with a single answer for a complex causality problem, something hints to me that you would probably point to the same “cause”for other problems too. That's basically the point of the 5 whys approach. You work up the tree/into the graph of contributing factors based on the specific problem and you end up solving lots of problems. If you don't go back far enough, you're likely just putting a band-aid on the problem and some other variation of it will waste more time again. You get "5 whys" back and now you can really start figuring out what went wrong and how to fix/prevent/mitigate/avoid/etc. it for next time.
  - darksaints 4y ago
    If you asked my dad to root cause literally any problem in my life, from my failed marriage all the way to a random nasty smell emanating from my kitchen, he would trace it back to the fundamental root cause that I’m no longer a believing mormon. I don’t have any problem with the idea that you should dig deeper for any form of causal reasoning. My problem is with the inherent linear flow and singular root cause reasoning of the 5-whys methodology. It is just too easy to lead you back to the thing that you are personally biased to think anyway. If you want to solve actual problems, you more than likely have to solve multiple problems that contribute to a complex system than to solve any singular problem.
sokoloff 4y ago
The fourth why is often “because we were lazy”.
- efsavage 4y ago
  This is actually a legitimate cause, though not always expressed that way ;) Many problems don't get solved well (or at all) because nobody in the organization finds them interesting. The only way to fix this that I've seen is to change the composition of the team and/or change the composition of the problems. This is why so many startups struggle, they have a small team and eventually encounter problems nobody is energized by solving and they can't hire someone who is interested. Eventually people burn out doing things they hate and it all falls apart.
- bombcar 4y ago
  The follow-up question should be along the lines of "what could make the right choice easier/the lazy choice?" Often you CAN actually make it so that fixing the issue IS the lazy option, but you may need the right skillset applied to it. E.g, manual process that could have been automated it someone with automation skills had been available.
xyzelement 4y ago
I used to work at a hedge fund known for its diagnosis process, a logical process of clearly defining the suboptimal outcome that occurred, agreeing on how it should have gone and what had happened, and if it's indeed wrong to have gone this way, what to do about it. I gained two key insights from experiencing this several times. First, the really useful diagnosis often transcend realms. Eg, what may have occurred as a technical issue was really the result of a process issue, which may have been a result of a hiring issue, which may have stemmed from the personality of the manager who made the hire. Second, the goal is to necessarily to find the "one true cause" but to drive some improvement. For example, if something was caused by a confluence of factors, if you dove in and improved just one of the factors, you've already made the situation better. You may still chose to fix just the tech or process, but the insight into the deeper causes helped shape our understanding through time.
- MuffinFlavored 4y ago
  This is a really optimistic way to spin the soul sucking reality of "investigate production bugs for clients" full time for a living or else.
  - xyzelement 4y ago
    I actually don't know how you got to this comment from my narrative. Personally, I deeply enjoyed the ability to surface deep insights into the org and the people based on what superficially may have been a technical or process issue - it gave us much better leverage to continuously optimize the org in great ways. It's kinda the opposite of "soul sucking"
    - idiocrat 4y ago
      Yes, if your client is genuinely looking for an improvement. There are bad examples also, especially in cash-rich companies, who tend to be infested by parasite managers. Those are not interested in org optimizations.
      - 4y ago
raldi 4y ago
> If you are given responsibility without the power to make any changes, then you have just become the scapegoat Exactly. Or as I like to put it: With great responsibility (should) come great power.
- aworks 4y ago
  In a more moderate version of this, I believe to be powerful also requires one to be responsible (among other attributes). I won't go into the details but I've found this to be also true in my personal life, when I've been in both the powerful and powerless roles.
idiocrat 4y ago
In the context of big corporations, where I had experience working at, I often recall Aesop's animal tales. The animals have bought all the expensive shiny musical instruments, in order to build an orchestra. But nobody knows how to play (and how to play together). So the only thing they can adjust is the seat order arrangement, stings to the left, brass to the right. One of the companies I worked for had licensed all possible expensive applications, databases, analytical tools. But nobody knew how to make use of those. There were many initiatives, but all efforts were spread thin. Perhaps in German this would be called Gießkannenprinzip (the water sprinkler principle)
- idiocrat 4y ago
  Is there any collection of Aesop animal tales, adapted to the modern corporate environment?
fwlr 4y ago
This post links to an article, “People can read their manager’s mind”, which is truly a gem. It argues that employees have the time and the incentive to figure out what their manager truly values, and the employees will then pay lip service to, or even outright ignore, what the manager requires of them - in favor of doing what they know the manager truly values, because the manager will subconsciously reward them for the “truly valued” stuff more than the “stated goals” stuff. The examples are illustrative: “ A manager truly appreciates original mathematical ideas. The [same] manager requests to rid the code of crash-causing bugs, because customers resent crashes. The most confident people ignore him and spend time coming up with original math. The less confident people spend time chasing bugs, are upset by the lack of recognition, and eventually leave for greener pastures. At any given moment, the code base is ridden by crash-causing bugs. “ A manager enjoys "software architecture", design patterns, and language lawyer type of knowledge. The [same] manager requests to cooperate better with neighboring teams who are upset by missing functionality in the beautifully architected software. People will tend to keep designing more patterns into the program.” It suggests a unique and amusing approach to hiring management: executives should find people who truly value the goals of the executive, hire them as managers, and (perhaps via intentional lack of oversight!) give them leeway to let their subconscious values impact their work, thus making their minds legible to mind-reading employees.
- lmm 4y ago
  > It suggests a unique and amusing approach to hiring management: executives should find people who truly value the goals of the executive, hire them as managers, and (perhaps via intentional lack of oversight!) give them leeway to let their subconscious values impact their work, thus making their minds legible to mind-reading employees. "Hire people who share your values/goals and let them get on with things" is hardly a novel insight. The trouble is that aligning them is hard.
  - fwlr 4y ago
    >”Hire people who share your values/goals and let them get on with things" is hardly a novel insight. Right, because it’s not the same thing. If you hear “hire people who share your values/goals and let them get on with things” and implement it, you’ll likely end up focusing on hiring developers/engineers and giving them resources, rather than hiring managers and giving them fiefdoms to run according to their whims. You’ll also likely end up hiring people whose true values match your true values and whose stated goals match your stated goals, but this argues that you want people whose true values match your stated goals, and you should ignore their stated goals. Despite sounding similar I think it is really quite different. (Arguably, this is WHY we find “alignment” so hard: we are constantly aligning their values with our values and their goals with our goals, and we are even achieving that alignment some of the time, and yet things still don’t work - instead we should be trying to align their values with our goals, because this is the kind of alignment that produces results)
bloomingeek 4y ago
Humans have always had the ability to mess up a one car funeral. The kingdom building manager has always been the problem where I work. He will hire those below him who are woefully inept, so he can insulate himself from his own errors. Those of us not in management, who actually know to job, are ignored. This causes an incredible amount of waste and time. And yes, if you try to go above his head, you will be scapegoated out of your job. One can imagine what this does for morale.
manmal 4y ago
While I find the idea of perfect debuggability appealing, i.e. that I can trace problems back to upper management or the CEO - I think there is way too much entropy involved to do this in a meaningful way. Remove manager M1 or M2, and things might actually get worse. Why? You’ll have no clue.
tunesmith 4y ago
This is about the logic of sufficiency and necessity, and it requires the skill to identify narrowly-scoped syllogisms for each step of the causal graph. For anyone really interested in this stuff, read about Theory of Constraints "Thinking Tools", and supplement with a smattering of simple philosophy, such as Hume's Guillotine and the Munchausen Trilemma. Basically, for any examination of "why did our actions result in this outcome", you're going to come down to a collection of premises, some of which are facts and others of which are values. Many of these values are obvious, widely-shared non-negotiable values, but others are slippery biases or assumptions that have snuck into the system, and focusing on one of these is often identified as the "root cause", even though there are by definition several; any one of which was sufficient to cause the outcome. Really "root cause" just means the identification of one value that could and should be changed, which thereby will propagate its changes throughout the causal network. I will say though, that this is a difficult skill, and it takes practice, because there are so many personal biases that can creep in and lead to a corrupted causal analysis. It's also a bit thankless because once you come up with a decent root-cause analysis, it reads so clearly that it basically comes across as obvious and common-sense to anyone that reads it, leading them to underestimate just how much work it took to get there.
mcqueenjordan 4y ago
I've had times where I stopped the "5 Why's" process short of getting into the human-squishy organizational root causes because I knew the problems were too big to be solved in the context of that particular COE/postmortem. I've also had times where I knew my director/senior manager would've taken issue with the publication of those squishy organizational issues, /even if/ they were true, simply because he/she surmised it wasn't worth burning the political capital for it. (And I've been asked to revise my 5 Whys by leadership exactly because of that reason before.) I've successfully pushed through organizational changes at least a few times from the output of the 5 Whys process too. If the COE/postmortem revolves around a critical issue or has gravitas, it's easier to put your full weight behind it. To be honest, my personal hit rate on these is probably about right. If you insist on it every time, you're probably going to burn out on it, personally or organizationally. Sometimes it's OK to try to address a symptom, if the medicine is cheap, easy to administer, and has little to no side effects. And in tech, the symptoms are often inherently layered -- you can often address entire classes of symptoms in reasonable ways, /even if/ that solution is itself working around a deeper organizational shortcoming / root cause. Spend the friction tokens where it matters most.
- PeterisP 4y ago
  The level of "5 why's" depends on the level of the person doing it, going as far as its needed to reach that level where you should be driving change. In the "car won't start" example of the original article, the motor pool mechanic needs to stop the whys at the broken alternator belt, because that is something they can and should fix; the motor pool manager needs to go on until reaching the practice/policy for how often preventive maintenance is done, and the top management (if the consequences are severe enough to warrant attention) need to go on until the "why do we have poor car maintenance" questions reach things like mid-management forcing people to ignore the maintenance schedules, or staffing policy, budget for whole departments, and/or the choice to outsource/insource that function. But going on much further than your own level is not going to be productive - you can make an actionable suggestion to your boss if some root cause isn't fixable at your level but is fixable there, and that's about it, at least in my experience.
- andruby 4y ago
throwawaaarrgh 4y ago
Tech stopped being interesting to me when I realized that building a widget isn't hard, it's getting 30 people to build a widget that's hard. Yet the way those 30 people are organized is so shitty it's as if nobody had ever done it before. I've worked with so many truly incompetent and stupid teams and organizations, that have millions to billions of dollars. Sometimes I wonder if I've just hallucinated all the books and training I've taken that explain the basic business and industry concepts most people should know. And I wonder how it is an IC could seem to know how things should be working while seemingly nobody else does. Then I go back to my 1/30th of the widget and wait for lunch time.
- gryn 4y ago
  > basic business and industry concepts most people should know got a list of them for reference ?
  - bobsmooth 4y ago
    Capital expenditure isn't a bad thing. Buying equipment before you need it is cheaper than scrambling to buy something when its urgent.
    - MrJohz 4y ago
      I'm not an economist, but I don't believe that's always true. Let's say I can buy replacement server at my own pace for £100, or in a panic when it's urgent for £200 (obviously made up numbers to make the maths easier for me). I can do the former at any time up until the server breaks, when I can only do the latter. If I invest that £100 instead in another part of the business, that returns £150 a year on average. After two years, that's already £225 - enough to pay for the crisis and still have made £25 profit. If I can get a third year without the server breaking, I'm closer to £140 profit, and after four years I can already afford to have two crises off the back of my initial £100 crisis-avoidance money. Obviously this is a very simple example, and there are lots of examples where this wouldn't work out. If my rate of return was only 10%, then it would take eight years before I'd have made back enough money to pay for the crisis situation. And sometimes gambles don't pay off and the crisis happens sooner than you expect, and other times people are just consistently bad at estimating these sorts of odds. But the point is that, even if it's cheaper to do something now then later, if you can find something more useful for that money to be doing right now, it might still be more profitable to take the expensive option. (That said, there's also obviously the human cost - if your team is constantly operating in crisis mode because everything's been used until the last possible moment, then it's unlikely to be an effective team for very long. But then that's kind of the point here: this is all a matter of tradeoffs, and trying to predict the future as best as possible with limited data.)
jimwhite42 4y ago
I like to think about this sort of stuff in terms of strategy. The definition I use is 'strategical goals' are the things you are trying to achieve, and 'strategy' is the 'why, how, what's the rationale for these, when do we revisit the goals and the strategy, etc.'. Strategical goals can be high level or low level. In some cases you can take part in a big picture company strategy process. In other cases, the strategical goals you have as an individual are to support the goals/ways of working of your boss, and the other people you work with, which you can only comprehend in a very blinkered way. In all these situations, it seems like some of the time, and for a critical part of what you do a lot of the time, you have to have strategies that take into account other people having hidden motivations, and particularly people who don't think strategically and cannot be reasoned with on the basis of strategy, or they refuse to be open with these sorts of things with you in particular, or will be evasive, or say they will do things and then not do them, etc.. A lot of the time you cannot fix these things, you can only take them into account and/or find workarounds. And it seems reasonable to say that this is the part of the picture where you have the most leverage to make a difference, or to reduce things like the amount of bullshit you deal with, or the amount of wasted effort you put into things. You can also have the insight that other people will see you the same way - as someone who is obstructive, or fails to be clear, or isn't persuing the right goals, etc..
- erlich 4y ago
  > hidden motivations I guess you just have to ask why people have these motivations and why they hide them. It feels like people would act dishonestly first to seek survival (job security), but then also to seek power (promotion/influence). I wonder what the relationship is between survival and power. I guess at some point survival manifests into the desire to hold onto power. And the stakes get larger as the power accrued increases. I also wonder how the human reward schedule plays into it. People are motivated by consistent rewards but at random intervals. If you take out the corporate power politics, you remove the randomness of the intervals, and it perhaps becomes less motivating. The chance for an early promotion provides motivation to play the game harder.
  - jimwhite42 4y ago
erlich 4y ago
How is it that no one has solved "management" yet with a technological solution? Where are the project/org management AI solutions?
- Atolian 4y ago
  Because management is 95% managing people. And managing people is the hard part of the job.
  - fuzzfactor 4y ago
    That's not even enough, you've got to be able to herd cats.
    - erlich 4y ago
      An AI can probably herd cats.
  - erlich 4y ago
    Tech systems manage people all the time. How many chat conversations between managers and employees do you think are actually unique.
gtirloni 4y ago
I wish I could work on Rachel's team. Such clarity of thought and purpose. It's inspiring to read her articles.
twoquestions 4y ago
This is a common failure in authoritarian organizations. We'll be in a much better place when Corporate America either adopts or is forced to adopt a more egalitarian way of management. This is one of the major reasons why I chose to make less money working at a smaller company, as it isn't big enough to accommodate these "reality distortion fields". I keep wanting to read Moral Mazes, but unfortunately I've yet to get to it. https://en.wikipedia.org/wiki/Moral_Mazes https://en.wikipedia.org/wiki/Moral_Mazes
politician 4y ago
> That might lead into "the entire company is obsessed with hiring even though the tech equivalent of the Drake equation says there is no way they can find anywhere near that many qualified people in the entire world. Fortunately, we don’t have to characterize this equation to quantify bloat in the corporate structure. We just have to wait for layoffs in the next recession. Unfortunately, this is a lagging indicator.
biscuits1 4y ago
A fantastic write up on a metaphorical ultra-intelligent way to lament about manager(s), and link it to kaizen. Bravo!
johngalt 4y ago
An excellent write up of the changes in perspective which occur with experience. No organization is optimal and neither are we. The key is to incorporate these details into planning. For example if the organization can't set durable goals, don't start a 3 year plan. Don't passively accept it, or become fatalistic about dysfunctions. If there are achievable changes to make, try to make them within your sphere of influence. Do what you can to produce positive changes, but don't allow frustration to lead to cynicism. Think of layer eight problems as engineering constraints like any other.
ranger207 4y ago
Some people like writing code. Some people like acquiring power. The latter will tend to consolidate in management positions, the former will tend to wonder why incredibly obvious technical and social problems are being ignored
craigkilgo 4y ago
Sometimes though if you track the problem all the way to layer 8, the origin of the problem has "resolved itself" i.e. left the organization, fired, retired, died, etc. Then you can solve the issue with no pushback.
ogn3rd 4y ago
Isn't it all layer 8 problems these days? Most of us can solve problems at the first seven.
rhacker 4y ago
Pedant: If your "alternator" belt is broken, you are likely not going to have a dead battery - you probably won't be able to steer, and you're likely not going to have a dead battery since you are not driving/using the car since it will also overheat very quickly.

317 points by zdw 4y ago

If you go below the "tech whys" and into the organizational whys, then you are thinking at the team leader or VP Engineering level. I think you want to be eyes opened to the full list of whys, but be happy that some cannot be changed. After all you are hired by a business, run by people, not a perfectly functional institution.

csours 4y ago

Yup, and for this reason, I will never work on a compliance app again in my life. Compliance is all about reducing risk to an acceptable level, but if you care about it, it is also about reducing risk to as close to zero as possible. Compliance generally affects multiple organizations, and that means convincing a bunch of people to do something that doesn't help them accomplish their primary goals. To be clear, I'm not alleging any wrongdoing whatsoever against any group that I worked with. I'm simply saying that it's hard enough to get people to do the job they accept as their main job; adding 'extra' work is never a rewarding experience - especially as an individual contributor.

stakhanov 4y ago
...yeah, there are several roles/functions that are by definition thankless tasks. In compliance, the best you can produce is nothing, as in no incidents that could have been prevented through better compliance, while causing a ton of cost and frustration across the organization. The worst you can produce is ruining the company and sending people to jail in a way that you were tasked with preventing and could actually have prevented. It sort of reminds me of this thought experiment that Nassim Nicholas Taleb mentions in one of his books: What if a bureaucrat had figured out, pre 9/11, that terrorists flying a commercial airliner in to a building was a risk worth trying to prevent and had passed legislation for all cockpits to be bulletproof cells that could only be opened from the inside, or something like that. 9/11 would have never happened, and that bureaucrat would have caught shit for the rest of his life for championing legislation that is causing a lot of people a lot of cost and frustration and has never had any observably positive impact anywhere in the observable universe.

nl 4y ago

> The car didn't start... because the battery is dead... because the alternator wasn't charging it... because the alternator belt broke... because the belt was beyond its useful life but wasn't replaced... because it wasn't maintained according to recommended schedule. This isn't "five whys" except in the most superficial sense. It can be useful to do this to build reliable technology but it isn't what five whys is trying to solve. > because the battery is dead... because the alternator wasn't charging it... because the alternator belt broke... because the belt was beyond its useful life but wasn't replaced... All of these are a single "why" (call it "technical faults" or something). > It turns out that when you start doing this root-cause analysis and really keep after it, the "squishy human realm" is actually the no-longer-hypothetical "layer eight" from those T-shirts. Is this actually a revelation to anyone? These hard questions are exactly what five whys is designed to address.

debacle 4y ago

I have a socket wrench. It's a really good socket wrench, it would probably cost me close to $200 to replace. I use it for nearly all of the bolts and lags that I need to loosen or tighten. The bolts on the spindles for my tractor deck are massive, they rust up quickly, and my $200 socket wrench doesn't touch them. For those spindles, I have a special tool that can solve the kind of problems that a $200 socket wrench can't solve. Despite my socket wrench not solving all problems, I'm still really glad I have it, because it is helpful most of the time, and I trust myself to be smart enough to know when I need a different tool.

electromech 4y ago
> Despite my socket wrench not solving all problems If only, amirite?!

ggm 4y ago

To me, the real layer-8 in a car model is that you probably shouldn't have built out society to depend on cars in the first place. Thats why we have the idea of the 15 minute city: places you can manage without private transport. Applied to s/w this would be "do we really need to make this system, in the first place: How about if we decide not to"

viburnum 4y ago
It is so satisfying when you can simply delete a bunch of code rather than fix it.
- ggm 4y ago
  I've struggled to take my teams (in the past: I don't do PM now) beyond "ship it" to "kill it": the C-suites don't like the idea of stopping doing things. They have a point, we maybe don't have community permission, but we should at least entertain the idea (in the not for profit sector) People rei-ify things. "oh we need to do X we've always done X" when in fact, you could say "doing X is stupid"
azmarks 4y ago
Yes. This is a system problem. Why are cars allowed to be near pedestrians? Why were cities/streets designed to make it possible for cars to run over pedestrians? Saying it was an inattentive driver misses the point that we can't expect fallible people to not make mistakes. We should be designing systems to minimize the damage from inattentive drivers and drunk pedestrians

darksaints 4y ago

I’ve always been bothered by the five whys, simply because it’s such a dramatic oversimplification of cause and effect. The fatal flaw is the assumption that causality is linear, when it is more often combinatorial in nature. To illustrate consider a drunk pedestrian jaywalking and being hit by a speeding driver who was texting while driving. Who caused the accident? They both did, together. Or maybe one of them did and the other didn’t. Or maybe the stars aligned just right and both would have been fine if they hadn’t been doing it at the exact same time? I can’t help but feel like anybody who has found the “root cause” for any problem, their answer will reflect their biases far more than will reflect true causality. If you can come up with a single answer for a complex causality problem, something hints to me that you would probably point to the same “cause”for other problems too.

madrox 4y ago
There's a common fallacy in your example in that you're asking "who" caused the accident and not "what." This is why blameless retros are important. Whenever you're dealing with who, the responsibility is never binary. There's always some responsibility someone can take for a situation they're a part of. That's good advice for life in general. The intention of the five whys is to find breakdowns at each level...not just the last one, and mitigate each.
efsavage 4y ago
> If you can come up with a single answer for a complex causality problem, something hints to me that you would probably point to the same “cause”for other problems too. That's basically the point of the 5 whys approach. You work up the tree/into the graph of contributing factors based on the specific problem and you end up solving lots of problems. If you don't go back far enough, you're likely just putting a band-aid on the problem and some other variation of it will waste more time again. You get "5 whys" back and now you can really start figuring out what went wrong and how to fix/prevent/mitigate/avoid/etc. it for next time.
- darksaints 4y ago
  If you asked my dad to root cause literally any problem in my life, from my failed marriage all the way to a random nasty smell emanating from my kitchen, he would trace it back to the fundamental root cause that I’m no longer a believing mormon. I don’t have any problem with the idea that you should dig deeper for any form of causal reasoning. My problem is with the inherent linear flow and singular root cause reasoning of the 5-whys methodology. It is just too easy to lead you back to the thing that you are personally biased to think anyway. If you want to solve actual problems, you more than likely have to solve multiple problems that contribute to a complex system than to solve any singular problem.

sokoloff 4y ago

The fourth why is often “because we were lazy”.

efsavage 4y ago
This is actually a legitimate cause, though not always expressed that way ;) Many problems don't get solved well (or at all) because nobody in the organization finds them interesting. The only way to fix this that I've seen is to change the composition of the team and/or change the composition of the problems. This is why so many startups struggle, they have a small team and eventually encounter problems nobody is energized by solving and they can't hire someone who is interested. Eventually people burn out doing things they hate and it all falls apart.
bombcar 4y ago
The follow-up question should be along the lines of "what could make the right choice easier/the lazy choice?" Often you CAN actually make it so that fixing the issue IS the lazy option, but you may need the right skillset applied to it. E.g, manual process that could have been automated it someone with automation skills had been available.

xyzelement 4y ago

I used to work at a hedge fund known for its diagnosis process, a logical process of clearly defining the suboptimal outcome that occurred, agreeing on how it should have gone and what had happened, and if it's indeed wrong to have gone this way, what to do about it. I gained two key insights from experiencing this several times. First, the really useful diagnosis often transcend realms. Eg, what may have occurred as a technical issue was really the result of a process issue, which may have been a result of a hiring issue, which may have stemmed from the personality of the manager who made the hire. Second, the goal is to necessarily to find the "one true cause" but to drive some improvement. For example, if something was caused by a confluence of factors, if you dove in and improved just one of the factors, you've already made the situation better. You may still chose to fix just the tech or process, but the insight into the deeper causes helped shape our understanding through time.

MuffinFlavored 4y ago
This is a really optimistic way to spin the soul sucking reality of "investigate production bugs for clients" full time for a living or else.
- xyzelement 4y ago
  I actually don't know how you got to this comment from my narrative. Personally, I deeply enjoyed the ability to surface deep insights into the org and the people based on what superficially may have been a technical or process issue - it gave us much better leverage to continuously optimize the org in great ways. It's kinda the opposite of "soul sucking"
  - idiocrat 4y ago
    Yes, if your client is genuinely looking for an improvement. There are bad examples also, especially in cash-rich companies, who tend to be infested by parasite managers. Those are not interested in org optimizations.
    - 4y ago

raldi 4y ago

> If you are given responsibility without the power to make any changes, then you have just become the scapegoat Exactly. Or as I like to put it: With great responsibility (should) come great power.

aworks 4y ago
In a more moderate version of this, I believe to be powerful also requires one to be responsible (among other attributes). I won't go into the details but I've found this to be also true in my personal life, when I've been in both the powerful and powerless roles.

idiocrat 4y ago

In the context of big corporations, where I had experience working at, I often recall Aesop's animal tales. The animals have bought all the expensive shiny musical instruments, in order to build an orchestra. But nobody knows how to play (and how to play together). So the only thing they can adjust is the seat order arrangement, stings to the left, brass to the right. One of the companies I worked for had licensed all possible expensive applications, databases, analytical tools. But nobody knew how to make use of those. There were many initiatives, but all efforts were spread thin. Perhaps in German this would be called Gießkannenprinzip (the water sprinkler principle)

idiocrat 4y ago
Is there any collection of Aesop animal tales, adapted to the modern corporate environment?

fwlr 4y ago

This post links to an article, “People can read their manager’s mind”, which is truly a gem. It argues that employees have the time and the incentive to figure out what their manager truly values, and the employees will then pay lip service to, or even outright ignore, what the manager requires of them - in favor of doing what they know the manager truly values, because the manager will subconsciously reward them for the “truly valued” stuff more than the “stated goals” stuff. The examples are illustrative: “ A manager truly appreciates original mathematical ideas. The [same] manager requests to rid the code of crash-causing bugs, because customers resent crashes. The most confident people ignore him and spend time coming up with original math. The less confident people spend time chasing bugs, are upset by the lack of recognition, and eventually leave for greener pastures. At any given moment, the code base is ridden by crash-causing bugs. “ A manager enjoys "software architecture", design patterns, and language lawyer type of knowledge. The [same] manager requests to cooperate better with neighboring teams who are upset by missing functionality in the beautifully architected software. People will tend to keep designing more patterns into the program.” It suggests a unique and amusing approach to hiring management: executives should find people who truly value the goals of the executive, hire them as managers, and (perhaps via intentional lack of oversight!) give them leeway to let their subconscious values impact their work, thus making their minds legible to mind-reading employees.

lmm 4y ago
> It suggests a unique and amusing approach to hiring management: executives should find people who truly value the goals of the executive, hire them as managers, and (perhaps via intentional lack of oversight!) give them leeway to let their subconscious values impact their work, thus making their minds legible to mind-reading employees. "Hire people who share your values/goals and let them get on with things" is hardly a novel insight. The trouble is that aligning them is hard.
- fwlr 4y ago
  >”Hire people who share your values/goals and let them get on with things" is hardly a novel insight. Right, because it’s not the same thing. If you hear “hire people who share your values/goals and let them get on with things” and implement it, you’ll likely end up focusing on hiring developers/engineers and giving them resources, rather than hiring managers and giving them fiefdoms to run according to their whims. You’ll also likely end up hiring people whose true values match your true values and whose stated goals match your stated goals, but this argues that you want people whose true values match your stated goals, and you should ignore their stated goals. Despite sounding similar I think it is really quite different. (Arguably, this is WHY we find “alignment” so hard: we are constantly aligning their values with our values and their goals with our goals, and we are even achieving that alignment some of the time, and yet things still don’t work - instead we should be trying to align their values with our goals, because this is the kind of alignment that produces results)

bloomingeek 4y ago

Humans have always had the ability to mess up a one car funeral. The kingdom building manager has always been the problem where I work. He will hire those below him who are woefully inept, so he can insulate himself from his own errors. Those of us not in management, who actually know to job, are ignored. This causes an incredible amount of waste and time. And yes, if you try to go above his head, you will be scapegoated out of your job. One can imagine what this does for morale.

manmal 4y ago

While I find the idea of perfect debuggability appealing, i.e. that I can trace problems back to upper management or the CEO - I think there is way too much entropy involved to do this in a meaningful way. Remove manager M1 or M2, and things might actually get worse. Why? You’ll have no clue.

tunesmith 4y ago

This is about the logic of sufficiency and necessity, and it requires the skill to identify narrowly-scoped syllogisms for each step of the causal graph. For anyone really interested in this stuff, read about Theory of Constraints "Thinking Tools", and supplement with a smattering of simple philosophy, such as Hume's Guillotine and the Munchausen Trilemma. Basically, for any examination of "why did our actions result in this outcome", you're going to come down to a collection of premises, some of which are facts and others of which are values. Many of these values are obvious, widely-shared non-negotiable values, but others are slippery biases or assumptions that have snuck into the system, and focusing on one of these is often identified as the "root cause", even though there are by definition several; any one of which was sufficient to cause the outcome. Really "root cause" just means the identification of one value that could and should be changed, which thereby will propagate its changes throughout the causal network. I will say though, that this is a difficult skill, and it takes practice, because there are so many personal biases that can creep in and lead to a corrupted causal analysis. It's also a bit thankless because once you come up with a decent root-cause analysis, it reads so clearly that it basically comes across as obvious and common-sense to anyone that reads it, leading them to underestimate just how much work it took to get there.

mcqueenjordan 4y ago

I've had times where I stopped the "5 Why's" process short of getting into the human-squishy organizational root causes because I knew the problems were too big to be solved in the context of that particular COE/postmortem. I've also had times where I knew my director/senior manager would've taken issue with the publication of those squishy organizational issues, /even if/ they were true, simply because he/she surmised it wasn't worth burning the political capital for it. (And I've been asked to revise my 5 Whys by leadership exactly because of that reason before.) I've successfully pushed through organizational changes at least a few times from the output of the 5 Whys process too. If the COE/postmortem revolves around a critical issue or has gravitas, it's easier to put your full weight behind it. To be honest, my personal hit rate on these is probably about right. If you insist on it every time, you're probably going to burn out on it, personally or organizationally. Sometimes it's OK to try to address a symptom, if the medicine is cheap, easy to administer, and has little to no side effects. And in tech, the symptoms are often inherently layered -- you can often address entire classes of symptoms in reasonable ways, /even if/ that solution is itself working around a deeper organizational shortcoming / root cause. Spend the friction tokens where it matters most.

PeterisP 4y ago
The level of "5 why's" depends on the level of the person doing it, going as far as its needed to reach that level where you should be driving change. In the "car won't start" example of the original article, the motor pool mechanic needs to stop the whys at the broken alternator belt, because that is something they can and should fix; the motor pool manager needs to go on until reaching the practice/policy for how often preventive maintenance is done, and the top management (if the consequences are severe enough to warrant attention) need to go on until the "why do we have poor car maintenance" questions reach things like mid-management forcing people to ignore the maintenance schedules, or staffing policy, budget for whole departments, and/or the choice to outsource/insource that function. But going on much further than your own level is not going to be productive - you can make an actionable suggestion to your boss if some root cause isn't fixable at your level but is fixable there, and that's about it, at least in my experience.
andruby 4y ago

throwawaaarrgh 4y ago

Tech stopped being interesting to me when I realized that building a widget isn't hard, it's getting 30 people to build a widget that's hard. Yet the way those 30 people are organized is so shitty it's as if nobody had ever done it before. I've worked with so many truly incompetent and stupid teams and organizations, that have millions to billions of dollars. Sometimes I wonder if I've just hallucinated all the books and training I've taken that explain the basic business and industry concepts most people should know. And I wonder how it is an IC could seem to know how things should be working while seemingly nobody else does. Then I go back to my 1/30th of the widget and wait for lunch time.

gryn 4y ago
> basic business and industry concepts most people should know got a list of them for reference ?
- bobsmooth 4y ago
  Capital expenditure isn't a bad thing. Buying equipment before you need it is cheaper than scrambling to buy something when its urgent.
  - MrJohz 4y ago
    I'm not an economist, but I don't believe that's always true. Let's say I can buy replacement server at my own pace for £100, or in a panic when it's urgent for £200 (obviously made up numbers to make the maths easier for me). I can do the former at any time up until the server breaks, when I can only do the latter. If I invest that £100 instead in another part of the business, that returns £150 a year on average. After two years, that's already £225 - enough to pay for the crisis and still have made £25 profit. If I can get a third year without the server breaking, I'm closer to £140 profit, and after four years I can already afford to have two crises off the back of my initial £100 crisis-avoidance money. Obviously this is a very simple example, and there are lots of examples where this wouldn't work out. If my rate of return was only 10%, then it would take eight years before I'd have made back enough money to pay for the crisis situation. And sometimes gambles don't pay off and the crisis happens sooner than you expect, and other times people are just consistently bad at estimating these sorts of odds. But the point is that, even if it's cheaper to do something now then later, if you can find something more useful for that money to be doing right now, it might still be more profitable to take the expensive option. (That said, there's also obviously the human cost - if your team is constantly operating in crisis mode because everything's been used until the last possible moment, then it's unlikely to be an effective team for very long. But then that's kind of the point here: this is all a matter of tradeoffs, and trying to predict the future as best as possible with limited data.)

jimwhite42 4y ago

I like to think about this sort of stuff in terms of strategy. The definition I use is 'strategical goals' are the things you are trying to achieve, and 'strategy' is the 'why, how, what's the rationale for these, when do we revisit the goals and the strategy, etc.'. Strategical goals can be high level or low level. In some cases you can take part in a big picture company strategy process. In other cases, the strategical goals you have as an individual are to support the goals/ways of working of your boss, and the other people you work with, which you can only comprehend in a very blinkered way. In all these situations, it seems like some of the time, and for a critical part of what you do a lot of the time, you have to have strategies that take into account other people having hidden motivations, and particularly people who don't think strategically and cannot be reasoned with on the basis of strategy, or they refuse to be open with these sorts of things with you in particular, or will be evasive, or say they will do things and then not do them, etc.. A lot of the time you cannot fix these things, you can only take them into account and/or find workarounds. And it seems reasonable to say that this is the part of the picture where you have the most leverage to make a difference, or to reduce things like the amount of bullshit you deal with, or the amount of wasted effort you put into things. You can also have the insight that other people will see you the same way - as someone who is obstructive, or fails to be clear, or isn't persuing the right goals, etc..

erlich 4y ago
> hidden motivations I guess you just have to ask why people have these motivations and why they hide them. It feels like people would act dishonestly first to seek survival (job security), but then also to seek power (promotion/influence). I wonder what the relationship is between survival and power. I guess at some point survival manifests into the desire to hold onto power. And the stakes get larger as the power accrued increases. I also wonder how the human reward schedule plays into it. People are motivated by consistent rewards but at random intervals. If you take out the corporate power politics, you remove the randomness of the intervals, and it perhaps becomes less motivating. The chance for an early promotion provides motivation to play the game harder.
- jimwhite42 4y ago

erlich 4y ago

How is it that no one has solved "management" yet with a technological solution? Where are the project/org management AI solutions?

Atolian 4y ago
Because management is 95% managing people. And managing people is the hard part of the job.
- fuzzfactor 4y ago
  That's not even enough, you've got to be able to herd cats.
  - erlich 4y ago
    An AI can probably herd cats.
- erlich 4y ago
  Tech systems manage people all the time. How many chat conversations between managers and employees do you think are actually unique.

gtirloni 4y ago

I wish I could work on Rachel's team. Such clarity of thought and purpose. It's inspiring to read her articles.

twoquestions 4y ago

This is a common failure in authoritarian organizations. We'll be in a much better place when Corporate America either adopts or is forced to adopt a more egalitarian way of management. This is one of the major reasons why I chose to make less money working at a smaller company, as it isn't big enough to accommodate these "reality distortion fields". I keep wanting to read Moral Mazes, but unfortunately I've yet to get to it. https://en.wikipedia.org/wiki/Moral_Mazes https://en.wikipedia.org/wiki/Moral_Mazes

politician 4y ago

> That might lead into "the entire company is obsessed with hiring even though the tech equivalent of the Drake equation says there is no way they can find anywhere near that many qualified people in the entire world. Fortunately, we don’t have to characterize this equation to quantify bloat in the corporate structure. We just have to wait for layoffs in the next recession. Unfortunately, this is a lagging indicator.

biscuits1 4y ago

A fantastic write up on a metaphorical ultra-intelligent way to lament about manager(s), and link it to kaizen. Bravo!

johngalt 4y ago

An excellent write up of the changes in perspective which occur with experience. No organization is optimal and neither are we. The key is to incorporate these details into planning. For example if the organization can't set durable goals, don't start a 3 year plan. Don't passively accept it, or become fatalistic about dysfunctions. If there are achievable changes to make, try to make them within your sphere of influence. Do what you can to produce positive changes, but don't allow frustration to lead to cynicism. Think of layer eight problems as engineering constraints like any other.

ranger207 4y ago

Some people like writing code. Some people like acquiring power. The latter will tend to consolidate in management positions, the former will tend to wonder why incredibly obvious technical and social problems are being ignored

craigkilgo 4y ago

Sometimes though if you track the problem all the way to layer 8, the origin of the problem has "resolved itself" i.e. left the organization, fired, retired, died, etc. Then you can solve the issue with no pushback.

ogn3rd 4y ago

Isn't it all layer 8 problems these days? Most of us can solve problems at the first seven.

rhacker 4y ago

Pedant: If your "alternator" belt is broken, you are likely not going to have a dead battery - you probably won't be able to steer, and you're likely not going to have a dead battery since you are not driving/using the car since it will also overheat very quickly.