3 ms·

I once worked at a firm that ended up spending > 10 MM dollars because a very senior person did not understand statistical noise. There was a metric composed of

by lumost 4y ago

I once worked at a firm that ended up spending > 10 MM dollars because a very senior person did not understand statistical noise. There was a metric composed of ~600 human created data points with an error rate of 2%. Every week if the errors doubled very expensive ICs/managers would need to investigate. So the firm started spending more on trying to stabilize the metric - multiple applied scientists were hired to develop new statistical models. Engineering teams were funded to improve data collection. At its peak, there were probably 15+ engineers and scientists working on making sure that 12 errors out of 600 didn't become 24 errors out of 600 a couple weeks out of the year. This problem was compounded by trying to measure ~30 different segments with 600 measures per segment per week. To this day I'm shocked at how much time that ate up. Expecting leaders to understand statistics was unlikely to work, and we shouldn't be surprised that 5 statisticians paid handsomely to solve the problem came up with ways to occupy their time.

whatever1 4y ago
I don’t have full context but why it does not make sense to invest in bringing down the error in measuring a very important metric ? I used to barbecue using my experience(eyeballing it), and many times I was getting food cooked pretty well, but it was very unpredictable. After I bought remote thermometer I consistently cook my food to perfect temperature.
- pilotneko 4y ago
  Because the measurement itself was noisy, and unless the operational metric was updated, it would never be stable. I’ve experienced this hell, where a client is “red” or “yellow” based on an a phone call with an account manager. That assessment is one of many inputs to a flawed operational metric that swings wildly. To extend your metaphor, imagine if you were constrained to only visually asses the food. Your fancy new remote monitoring system is simply a camera that allows a stupidly expensive pitmaster view your food and make an expert assessment. The result might get a little better, but it’s (probably) never going to be as good as using an actual thermometer.
listenallyall 4y ago
Your story sounds outrageous initially (10 million dollars!) but if this is a 5, 50, maybe 500 billion company, that number isn't so crazy. If it's related to quality control of a manufactured product, large changes in errors would seem to be a significant problem. Moreover, if 12 errors can become 24 in a week and nobody did anything about it, what would prevent 24 from becoming 48, or even more, until hundreds of your metrics were out of line and in total chaos? I'll also note you don't really explain "error rate of 2%." Do you mean, all 600 data measures are continuous variables, of which, each measured result may be off by 2% (or 2% is the std dev?), or are these discrete or binary measures, of which 2% are entirely wrong?

by lumost 4y ago

whatever1 4y ago

I don’t have full context but why it does not make sense to invest in bringing down the error in measuring a very important metric ? I used to barbecue using my experience(eyeballing it), and many times I was getting food cooked pretty well, but it was very unpredictable. After I bought remote thermometer I consistently cook my food to perfect temperature.

pilotneko 4y ago
Because the measurement itself was noisy, and unless the operational metric was updated, it would never be stable. I’ve experienced this hell, where a client is “red” or “yellow” based on an a phone call with an account manager. That assessment is one of many inputs to a flawed operational metric that swings wildly. To extend your metaphor, imagine if you were constrained to only visually asses the food. Your fancy new remote monitoring system is simply a camera that allows a stupidly expensive pitmaster view your food and make an expert assessment. The result might get a little better, but it’s (probably) never going to be as good as using an actual thermometer.

listenallyall 4y ago

Your story sounds outrageous initially (10 million dollars!) but if this is a 5, 50, maybe 500 billion company, that number isn't so crazy. If it's related to quality control of a manufactured product, large changes in errors would seem to be a significant problem. Moreover, if 12 errors can become 24 in a week and nobody did anything about it, what would prevent 24 from becoming 48, or even more, until hundreds of your metrics were out of line and in total chaos? I'll also note you don't really explain "error rate of 2%." Do you mean, all 600 data measures are continuous variables, of which, each measured result may be off by 2% (or 2% is the std dev?), or are these discrete or binary measures, of which 2% are entirely wrong?