3 ms·
The weirdest “likelihood” conversation I ever had, the putative team lead didn’t want to change priorities to fix a bug because, “how often does that happen?”
by hinkley 3y ago
The weirdest “likelihood” conversation I ever had, the putative team lead didn’t want to change priorities to fix a bug because, “how often does that happen?”
My reply was, “it happens to every user, the first time they use the app.” And then something about how frequency has nothing to do with it. Every single user was going to encounter this bug. Even if they only used the app once.
I already had a toolbox full of conversations about how bad we are at statistics, but that one opened up a whole new avenue of things to worry about. One that was reinforced by later articles about the uselessness of p95 stats - particularly where 3% of your users are experiencing 100% outage.
But the one that is more apropos to the linked question, vs HN in general, is how people are bad at calculating the probability that “nothing bad happens” when there are fifty low probability things that can go wrong. Especially as the number of opportunities go up.
And the way that, if we do something risky and nothing bad happens, we estimate down the probability of future calamity instead of counting ourselves lucky and backing away.
- opportune 3y agoI've seen this exact same fallacy happen several times throughout my career, which isn't even very long. I think in many cases it boils down to some subtype not being identified and evaluated on its own. As in your case it's especially impactful, and yet IME also usually where these kinds of things get improperly prioritized, when it's a user's first impression or when it occurs in a way that causes a user to have to just sit and wait on the other end as these are often "special" cases with different logic in your application code. OTOH sometimes users try to weird/wrong/adversarial shit and so their high failure rate is working as intended. But it pollutes your stats such that it can hide real issues with similar symptoms and skew distributions.
- itronitron 3y agoYeah, and I'd also add that the total # of bugs in an application will always be greater than the total # of 'known' bugs. Tracking down and fixing the oddball bugs usually prevents a larger set of related issues from popping up later.
- hinkley 3y agoSome bugs have multiple symptoms, it’s true.
- NoPicklez 3y agoThis is interesting, I wonder if there are principles from other types of engineering, civil, structural, aeronautical etc which provide some additional thoughts around probabilities of failure and now to deal with them Particularly where you have a lot of low probability bugs