7 ms·
I worked at a company once where each bug had a really interesting field: root cause I wish I could remember the values you could fill in, they were very intel
by m463 7y ago
I worked at a company once where each bug had a really interesting field: root cause
I wish I could remember the values you could fill in, they were very intelligently chosen.
What I learned: if you didn't know what the root cause was, you probably didn't fix anything.
- finnthehuman 7y agoWas it like an list of predefined values? Where I work they do root cause analysis for everything, but with freeform answers so what you describe might be different from what I'm used to. In general, I'm so used to RCA and layered mitigations (what one of our greybeards calls "belt and suspenders") that I don't know how quality happens without it. I'm a convert to the idea that if you can't fix a problem directly, the fix has to isolate or be as close to the problem as possible. Otherwise the bad state just ripples outward as complexity.
- m463 7y agoUnfortunately this was probably 20 years ago. I know it was a list of predefined values in a drop-down, but I'm not sure if there was a other/write-in field. The gist was that the causes were appropriate and educational. Folks couldn't choose "user is an idiot", instead having to choose "the interface was confusing".
- baud147258 7y agoWell "the interface was confusing" doesn't really rule out "user is an idiot", but most likely will make matters worse.
- JohnFen 7y agoI like having a set of broad predefined values (it helps with standardization, which helps with searching in the future). But a freeform report is also necessary. How else are you going to adequately explain what, where, why, etc., the root cause was?
- devdas 7y agoThere is no root cause. https://www.kitchensoap.com/2012/02/10/each-necessary-but-only-jointly-sufficient/ https://www.kitchensoap.com/2012/02/10/each-necessary-but-on... https://blog.acolyer.org/2016/02/10/how-complex-systems-fail/ https://blog.acolyer.org/2016/02/10/how-complex-systems-fail... http://web.mit.edu/2.75/resources/random/How%20Complex%20Systems%20Fail.pdf http://web.mit.edu/2.75/resources/random/How%20Complex%20Sys...
- astine 7y agoThe OP is talking about bugs not complex system failures. These are not the same thing. It might take multiple bugs to cause a complex system to fail, because complex systems often have enormous amounts of complexity built in, but a single bug is often a fairly unidimensional thing. You can identify the root cause of a bug.
- inflatableDodo 7y agoAnother way to look at that is that a root cause can be a set.
- hnick 7y agoI've had more than one case at work where it would come down to bad requirements. Both systems worked exactly as specified and were bug free (for the issue in scope). They just had a different understanding of reality by design. Root cause here is some mixture of poor understanding of the problem domain by various staff.
- windexh8er 7y ago1000% this. I've been in many orgs where root cause was either completely missing or completely missed the point. Recently I quit a company that thought they were doing RCA. The fact they sent out an email that there was an outage and then N number of hours or days later another email to say it was "fixed" and that they scaled or thought they found the problem and don't worry about it anymore. Literally we had weeks where the exact same outage occurred multiple times. And with every outage the exact same response. So... I asked simple questions of leadership as to why RCA process was not implemented. Why RCA did not require a standardized template to be filled out as part of a production outage. Why a "5 Whys" approach wasn't being considered to truly expose the actual root cause. Why there wasn't any accountability. At the end of the day failure cause is not root cause and many struggle conflating the two. Honestly when an organization doesn't hold true RCA as a critical part of engineering process I, personally, feel as though that organization will innevitably hit a glass ceiling. Among other problem areas the disconnect with RCA, for me, was why I couldn't stay at that company anymore. It was embarrassing watching from the inside as the same mistakes were made over and over with nobody the wiser.
- zaptheimpaler 7y agoSo how did the leadership respond to your questions? Did they agree, and did things change?
- windexh8er 7y agoThey "took it as feedback". The problem was the leadership was convinced they had a real RCA process in place. They didn't.
- JohnFen 7y ago> if you didn't know what the root cause was, you probably didn't fix anything. Yes! If you don't know root cause, then you don't know what went wrong. Not only do you not know what to fix, any shotgun debugging is likely to have only fixed one symptom, leaving the actual malfunction in place. Almost all of the companies I've owned or worked for have recognized this with a simple rule: if you haven't found (and proven) root cause, then the bug cannot be closed as fixed. Any company that doesn't have a variation of this going on is a company whose products you can't trust (and a company I would prefer not to work for).