5 ms·

I used to work at a hedge fund known for its diagnosis process, a logical process of clearly defining the suboptimal outcome that occurred, agreeing on how it s

I used to work at a hedge fund known for its diagnosis process, a logical process of clearly defining the suboptimal outcome that occurred, agreeing on how it should have gone and what had happened, and if it's indeed wrong to have gone this way, what to do about it. I gained two key insights from experiencing this several times. First, the really useful diagnosis often transcend realms. Eg, what may have occurred as a technical issue was really the result of a process issue, which may have been a result of a hiring issue, which may have stemmed from the personality of the manager who made the hire. Second, the goal is to necessarily to find the "one true cause" but to drive some improvement. For example, if something was caused by a confluence of factors, if you dove in and improved just one of the factors, you've already made the situation better. You may still chose to fix just the tech or process, but the insight into the deeper causes helped shape our understanding through time.

MuffinFlavored 4y ago
This is a really optimistic way to spin the soul sucking reality of "investigate production bugs for clients" full time for a living or else.
- xyzelement 4y ago
  I actually don't know how you got to this comment from my narrative. Personally, I deeply enjoyed the ability to surface deep insights into the org and the people based on what superficially may have been a technical or process issue - it gave us much better leverage to continuously optimize the org in great ways. It's kinda the opposite of "soul sucking"
  - idiocrat 4y ago
    Yes, if your client is genuinely looking for an improvement. There are bad examples also, especially in cash-rich companies, who tend to be infested by parasite managers. Those are not interested in org optimizations.
    - wpietri 4y ago
      I'll note that he was talking about a hedge fund. It's been a long time since I worked in finance, as I don't find zero-sum games rewarding. But despite the number of assholes in the industry, they were also very focused on maximizing gains, so they could also be very thoughtful and clear-minded. I found that as long as I could explain the value to them in dollar terms, they were very effective at improvement.
- seanhunter 4y ago
  If the parent is talking about who I'm pretty sure they are talking about (Bridgewater Associates) it's absolutely not just about investigating production bugs for them. It's about finding the truth of a given situation and working to improve the processes and understanding that led to the problem. You can find out more about the philosophy behind this by reading Ray Dalio's Principles, the published version of which is close enough to the version that everyone at Bridgewater gets a copy of when they walk in the door and is treated as almost a bible[1]. In particular read up about the 5-step process which is what they are talking about here unless I have completely misread. I didn't work for Bridgewater but I spent some time working at Bridgewater (for a vendor). Edit to add: Bridgewater's diagnosis process typically goes waaaaaaaay deeper than your standard 5-whys. Like the difference between the doctor just taking your blood pressure and the full physical where they put on the rubber gloves. [1] Like literally. People will quote principles to you by number if they think you have violated some of them.
- ReactiveJelly 4y ago
  From a metaphysical standpoint, is not every problem we ever solve, as humans, technically a production bug for some client?
rsanheim 4y ago
Regarding your first insight one of my favorite quotes came to mind: "The Second Law of Consulting: No matter how it looks at first, it's always a people problem." - Gerald Weinberg - The Secrets of Consulting
- barking_biscuit 4y ago
  Which is precisely why a lot of problems never seem to go away.
- wpietri 4y ago
  An especially painful truth in software. Years back I heard a talk by Kent Beck, one of the originators of Test-Driven Development and Extreme Programming. In response to some audience question, he said, "I got into programming because I didn't like dealing with people, and now I'm a @#$&% family therapist for software teams." That comes back to me often.
  - fragmede 4y ago
    Family therapist, for a family of people that don't deal well with other people. Maybe a psychology degree would be a better background to pull engineering managers from.
    - quietcomments 4y ago
      Psychologists don't make for good relationships. They break their own rules and ethics as much as anyone else. Believable relationships are scarce in the workplace.
    - PeterisP 4y ago
      Management study programs tend to include quite a few psychology-related courses, because these skills are indeed quite useful in practice.
      - disgruntledphd2 4y ago
idiocrat 4y ago
> useful diagnosis often transcend realms In big corporations many problems occur because of process/data siloing and impedance mismatch in the inter-departmental communication. Everyone is building their own, no organic intensives to share and cooperate.
msla 4y ago
Both of those things are good, useful versions of the "holistic" school of thought that engineers/westerners/males are supposed to be bad at, if you believe dumb pop psychology books. It's also a good first step to intersectionality, if you really want to get everyone to try to jump on your head. /s
donavanm 4y ago
>the goal is to ... drive some improvement. One aspect that I like to emphasise is the systemic or second order aspect. Yes, doing a retro/rca/post mortem/coe to prevent reoccurance of the same problem is good. Better is to prevent similar problems, or even classes of the problem, across the system in the future. A quick litmus test is "Do the lessons learned and action items apply to the observed problem, or do they apply to similar problems."
- kqr 4y ago
  Similar to this I look at it through Meadow's leverage points. Does the solution improve information flows in the system? Does it change the time scale of feedback loops? Does it improve material flows? A solution that improves information flow solves problems you don't even know existed, yet! This is one way to deal with unknown unknowns. (Or black swans, depending on whose terminology you prefer.)
srcreigh 4y ago
What type of hiring issue are we talking about?
jimmySixDOF 4y ago
I don't for a second doubt the insights derived from many of the Lesson Learned Retrospective Hot Wash Exit Interview type structured exercises of which this is one of the better, but there is an air of the Maginot Line preparing for tactics of a previous conflict while any ensuing false sense of accomplishment can increase exposure to new risk factors. The exception is if you can identify patterns over time across samples.
henrikeh 4y ago
> I used to work at a hedge fund known for its diagnosis process, a logical process of clearly defining the suboptimal outcome that occurred, agreeing on how it should have gone and what had happened, and if it's indeed wrong to have gone this way, what to do about it. Do you know of some publicly available material on this? I’d love to hear about their approach.
- jsty 4y ago
  It sounds like Bridgewater - and if so Dalio has written quite widely, e.g. https://www.principles.com/principles/85cda87a-6491-46bd-9cf3-f7918ad9b5a3/ https://www.principles.com/principles/85cda87a-6491-46bd-9cf...
jerf 4y ago
I think of "five whys" more as an example than an actual methodology. The goal in my opinion is: 1. Dig into the root causes. Five Whys is at least helpful in giving the idea that the root causes may be several layers away; you should definitely go past 1 or 2. Untrained humans find it very cognitively tempting to stop at exactly 1; this must be overcome and the Five Whys meme is a good entry point for that. 2. Don't be stuck on Five Whys literally, as essentially a linked list of exactly length five. Causes can be trees. (I mean, technically nothing stops them from being full graphs, at least from a high-level view that ignores temporal dependencies, but generally you need to tree-ify them for this discussion or you'll literally loop around the graph in your postmortem and analysis steps just like a computer would!) A 5-element linked list is merely a particular shape it can have. Management encouraging "Five Whys"-like thinking is a good thing; management mandating that an incident come with a Five Whys analysis with exactly five whys in a linked list would be missing the point in that oh-so-classic management methodology way. 3. Once you have your root cause tree, you examine it with the same engineering/business cost/benefit eyes as everything else. "The entire company's organization is inimical to this sort of quality" may be true, but extremely expensive to fix on its own terms. Even more so when you look up and consider that the company may in fact have other good reasons to be organized that way; there is no perfect organization that solves all problems optimally, and it may be a net loss to even consider re-orging the company to solve one particular thing that in the global context is an annoyance, if it would in the process break the organization that is producing actual value. But you may find a process tweak in some other layer that will mitigate 80% of the problem for hardly more than the cost of a meeting. It is perfectly valid to come up with a portfolio of solutions; one pattern I end up seeing a lot is the cheap short-term bandaid and the longer term fix. Cynicism aside, yes we often do end up with the long term fixes getting put into place... the key there is that they are usually themselves a portfolio of other fixes we needed that long-term fix for, not just this one issue that we've put a bandaid over. I'm yet to ever participate in an analysis like this that didn't produce some good actionable item. Often at least one of them is indeed quite cheap, even if fixing the root causes is expensive.
- malfist 4y ago
  The company I work for has a postmortem process that asks for the 5 whys. I'm part of a group that "levels up" other team's postmortems and the 5 whys are always the most difficult. 5 Whys are used a lot of time to shift blame away from the team where the issue originated, or someone does the 5 whys dogmatically and ends in a non-actionable final why. One of the things I teach is "5" is a suggestion, not a requirement. Trees are great. Each root node in the tree MUST be actionable. Asking people to do better next time isn't actionable. Blame isn't the goal of the document.

MuffinFlavored 4y ago

This is a really optimistic way to spin the soul sucking reality of "investigate production bugs for clients" full time for a living or else.

xyzelement 4y ago
I actually don't know how you got to this comment from my narrative. Personally, I deeply enjoyed the ability to surface deep insights into the org and the people based on what superficially may have been a technical or process issue - it gave us much better leverage to continuously optimize the org in great ways. It's kinda the opposite of "soul sucking"
- idiocrat 4y ago
  Yes, if your client is genuinely looking for an improvement. There are bad examples also, especially in cash-rich companies, who tend to be infested by parasite managers. Those are not interested in org optimizations.
  - wpietri 4y ago
    I'll note that he was talking about a hedge fund. It's been a long time since I worked in finance, as I don't find zero-sum games rewarding. But despite the number of assholes in the industry, they were also very focused on maximizing gains, so they could also be very thoughtful and clear-minded. I found that as long as I could explain the value to them in dollar terms, they were very effective at improvement.
seanhunter 4y ago
If the parent is talking about who I'm pretty sure they are talking about (Bridgewater Associates) it's absolutely not just about investigating production bugs for them. It's about finding the truth of a given situation and working to improve the processes and understanding that led to the problem. You can find out more about the philosophy behind this by reading Ray Dalio's Principles, the published version of which is close enough to the version that everyone at Bridgewater gets a copy of when they walk in the door and is treated as almost a bible[1]. In particular read up about the 5-step process which is what they are talking about here unless I have completely misread. I didn't work for Bridgewater but I spent some time working at Bridgewater (for a vendor). Edit to add: Bridgewater's diagnosis process typically goes waaaaaaaay deeper than your standard 5-whys. Like the difference between the doctor just taking your blood pressure and the full physical where they put on the rubber gloves. [1] Like literally. People will quote principles to you by number if they think you have violated some of them.
ReactiveJelly 4y ago
From a metaphysical standpoint, is not every problem we ever solve, as humans, technically a production bug for some client?

rsanheim 4y ago

Regarding your first insight one of my favorite quotes came to mind: "The Second Law of Consulting: No matter how it looks at first, it's always a people problem." - Gerald Weinberg - The Secrets of Consulting

barking_biscuit 4y ago
Which is precisely why a lot of problems never seem to go away.
wpietri 4y ago
An especially painful truth in software. Years back I heard a talk by Kent Beck, one of the originators of Test-Driven Development and Extreme Programming. In response to some audience question, he said, "I got into programming because I didn't like dealing with people, and now I'm a @#$&% family therapist for software teams." That comes back to me often.
- fragmede 4y ago
  Family therapist, for a family of people that don't deal well with other people. Maybe a psychology degree would be a better background to pull engineering managers from.
  - quietcomments 4y ago
    Psychologists don't make for good relationships. They break their own rules and ethics as much as anyone else. Believable relationships are scarce in the workplace.
  - PeterisP 4y ago
    Management study programs tend to include quite a few psychology-related courses, because these skills are indeed quite useful in practice.
    - disgruntledphd2 4y ago

idiocrat 4y ago

> useful diagnosis often transcend realms In big corporations many problems occur because of process/data siloing and impedance mismatch in the inter-departmental communication. Everyone is building their own, no organic intensives to share and cooperate.

msla 4y ago

Both of those things are good, useful versions of the "holistic" school of thought that engineers/westerners/males are supposed to be bad at, if you believe dumb pop psychology books. It's also a good first step to intersectionality, if you really want to get everyone to try to jump on your head. /s

donavanm 4y ago

>the goal is to ... drive some improvement. One aspect that I like to emphasise is the systemic or second order aspect. Yes, doing a retro/rca/post mortem/coe to prevent reoccurance of the same problem is good. Better is to prevent similar problems, or even classes of the problem, across the system in the future. A quick litmus test is "Do the lessons learned and action items apply to the observed problem, or do they apply to similar problems."

kqr 4y ago
Similar to this I look at it through Meadow's leverage points. Does the solution improve information flows in the system? Does it change the time scale of feedback loops? Does it improve material flows? A solution that improves information flow solves problems you don't even know existed, yet! This is one way to deal with unknown unknowns. (Or black swans, depending on whose terminology you prefer.)

srcreigh 4y ago

What type of hiring issue are we talking about?

jimmySixDOF 4y ago

I don't for a second doubt the insights derived from many of the Lesson Learned Retrospective Hot Wash Exit Interview type structured exercises of which this is one of the better, but there is an air of the Maginot Line preparing for tactics of a previous conflict while any ensuing false sense of accomplishment can increase exposure to new risk factors. The exception is if you can identify patterns over time across samples.

henrikeh 4y ago

> I used to work at a hedge fund known for its diagnosis process, a logical process of clearly defining the suboptimal outcome that occurred, agreeing on how it should have gone and what had happened, and if it's indeed wrong to have gone this way, what to do about it. Do you know of some publicly available material on this? I’d love to hear about their approach.

jsty 4y ago
It sounds like Bridgewater - and if so Dalio has written quite widely, e.g. https://www.principles.com/principles/85cda87a-6491-46bd-9cf3-f7918ad9b5a3/ https://www.principles.com/principles/85cda87a-6491-46bd-9cf...

jerf 4y ago

I think of "five whys" more as an example than an actual methodology. The goal in my opinion is: 1. Dig into the root causes. Five Whys is at least helpful in giving the idea that the root causes may be several layers away; you should definitely go past 1 or 2. Untrained humans find it very cognitively tempting to stop at exactly 1; this must be overcome and the Five Whys meme is a good entry point for that. 2. Don't be stuck on Five Whys literally, as essentially a linked list of exactly length five. Causes can be trees. (I mean, technically nothing stops them from being full graphs, at least from a high-level view that ignores temporal dependencies, but generally you need to tree-ify them for this discussion or you'll literally loop around the graph in your postmortem and analysis steps just like a computer would!) A 5-element linked list is merely a particular shape it can have. Management encouraging "Five Whys"-like thinking is a good thing; management mandating that an incident come with a Five Whys analysis with exactly five whys in a linked list would be missing the point in that oh-so-classic management methodology way. 3. Once you have your root cause tree, you examine it with the same engineering/business cost/benefit eyes as everything else. "The entire company's organization is inimical to this sort of quality" may be true, but extremely expensive to fix on its own terms. Even more so when you look up and consider that the company may in fact have other good reasons to be organized that way; there is no perfect organization that solves all problems optimally, and it may be a net loss to even consider re-orging the company to solve one particular thing that in the global context is an annoyance, if it would in the process break the organization that is producing actual value. But you may find a process tweak in some other layer that will mitigate 80% of the problem for hardly more than the cost of a meeting. It is perfectly valid to come up with a portfolio of solutions; one pattern I end up seeing a lot is the cheap short-term bandaid and the longer term fix. Cynicism aside, yes we often do end up with the long term fixes getting put into place... the key there is that they are usually themselves a portfolio of other fixes we needed that long-term fix for, not just this one issue that we've put a bandaid over. I'm yet to ever participate in an analysis like this that didn't produce some good actionable item. Often at least one of them is indeed quite cheap, even if fixing the root causes is expensive.

malfist 4y ago
The company I work for has a postmortem process that asks for the 5 whys. I'm part of a group that "levels up" other team's postmortems and the 5 whys are always the most difficult. 5 Whys are used a lot of time to shift blame away from the team where the issue originated, or someone does the 5 whys dogmatically and ends in a non-actionable final why. One of the things I teach is "5" is a suggestion, not a requirement. Trees are great. Each root node in the tree MUST be actionable. Asking people to do better next time isn't actionable. Blame isn't the goal of the document.