5 ms·

I think a lot of companies hopped on the "data science" bandwagon when they were really looking for data monkeys. A great data scientist may have more questions

I think a lot of companies hopped on the "data science" bandwagon when they were really looking for data monkeys. A great data scientist may have more questions than answers, challenging assumptions and sharpening domain knowledge. But leadership/PMs aren't used to having to really think hard about problems, they're used to some person fixing the spreadsheet or "go get me number... plz ". I hope as the field builds more hierarchy and data scientists have more organization decision making power, things could improve.

lumost 4y ago
I once worked at a firm that ended up spending > 10 MM dollars because a very senior person did not understand statistical noise. There was a metric composed of ~600 human created data points with an error rate of 2%. Every week if the errors doubled very expensive ICs/managers would need to investigate. So the firm started spending more on trying to stabilize the metric - multiple applied scientists were hired to develop new statistical models. Engineering teams were funded to improve data collection. At its peak, there were probably 15+ engineers and scientists working on making sure that 12 errors out of 600 didn't become 24 errors out of 600 a couple weeks out of the year. This problem was compounded by trying to measure ~30 different segments with 600 measures per segment per week. To this day I'm shocked at how much time that ate up. Expecting leaders to understand statistics was unlikely to work, and we shouldn't be surprised that 5 statisticians paid handsomely to solve the problem came up with ways to occupy their time.
- whatever1 4y ago
  I don’t have full context but why it does not make sense to invest in bringing down the error in measuring a very important metric ? I used to barbecue using my experience(eyeballing it), and many times I was getting food cooked pretty well, but it was very unpredictable. After I bought remote thermometer I consistently cook my food to perfect temperature.
  - pilotneko 4y ago
    Because the measurement itself was noisy, and unless the operational metric was updated, it would never be stable. I’ve experienced this hell, where a client is “red” or “yellow” based on an a phone call with an account manager. That assessment is one of many inputs to a flawed operational metric that swings wildly. To extend your metaphor, imagine if you were constrained to only visually asses the food. Your fancy new remote monitoring system is simply a camera that allows a stupidly expensive pitmaster view your food and make an expert assessment. The result might get a little better, but it’s (probably) never going to be as good as using an actual thermometer.
- listenallyall 4y ago
  Your story sounds outrageous initially (10 million dollars!) but if this is a 5, 50, maybe 500 billion company, that number isn't so crazy. If it's related to quality control of a manufactured product, large changes in errors would seem to be a significant problem. Moreover, if 12 errors can become 24 in a week and nobody did anything about it, what would prevent 24 from becoming 48, or even more, until hundreds of your metrics were out of line and in total chaos? I'll also note you don't really explain "error rate of 2%." Do you mean, all 600 data measures are continuous variables, of which, each measured result may be off by 2% (or 2% is the std dev?), or are these discrete or binary measures, of which 2% are entirely wrong?
keithalewis 4y ago
An Indian chief was asked by his tribe, "Winter is coming, what should we do?" He told them to collect firewood. When they brought that back they asked, "Is this enough?" The chief had forgotten the wisdom of his ancestors so he called the weather service. "Is it going to be cold this winter?" he asked. "Yes!" they told him. So he instructed his tribe to collect more firewood. When they asked again "Is this enough?" He called up the weather service. "So when you say it is going to be cold, exactly how cold?" They told him "All indications are pointing to a very cold winter." He told his tribe to collect even more firewood. When they asked yet again if it was enough, he called the weather service again. "Can you tell me how you know this winter will be so cold?" They told him "It's got to be. The Indians are collecting firewood like crazy."
- Eddy_Viscosity2 4y ago
  This reminds me of an actual story about book that was being priced at hundreds of thousands of dollars (I looked for a link to this story but couldn't find it). The gist was that a feedback loop from two online book sellers, neither of which had a copy of the book. They automated the pricing such it was based on the other's listing plus a small markup; the idea being that if someone wanted this book, they could buy it from the other seller, add the markup and pass it on the customer. But since both were doing this, one would see the price add a markup and set it as their price. After a while, the other would then see this price and do the same thing. Over time the price just grew and grew.

lumost 4y ago

I once worked at a firm that ended up spending > 10 MM dollars because a very senior person did not understand statistical noise. There was a metric composed of ~600 human created data points with an error rate of 2%. Every week if the errors doubled very expensive ICs/managers would need to investigate. So the firm started spending more on trying to stabilize the metric - multiple applied scientists were hired to develop new statistical models. Engineering teams were funded to improve data collection. At its peak, there were probably 15+ engineers and scientists working on making sure that 12 errors out of 600 didn't become 24 errors out of 600 a couple weeks out of the year. This problem was compounded by trying to measure ~30 different segments with 600 measures per segment per week. To this day I'm shocked at how much time that ate up. Expecting leaders to understand statistics was unlikely to work, and we shouldn't be surprised that 5 statisticians paid handsomely to solve the problem came up with ways to occupy their time.

whatever1 4y ago
I don’t have full context but why it does not make sense to invest in bringing down the error in measuring a very important metric ? I used to barbecue using my experience(eyeballing it), and many times I was getting food cooked pretty well, but it was very unpredictable. After I bought remote thermometer I consistently cook my food to perfect temperature.
- pilotneko 4y ago
  Because the measurement itself was noisy, and unless the operational metric was updated, it would never be stable. I’ve experienced this hell, where a client is “red” or “yellow” based on an a phone call with an account manager. That assessment is one of many inputs to a flawed operational metric that swings wildly. To extend your metaphor, imagine if you were constrained to only visually asses the food. Your fancy new remote monitoring system is simply a camera that allows a stupidly expensive pitmaster view your food and make an expert assessment. The result might get a little better, but it’s (probably) never going to be as good as using an actual thermometer.
listenallyall 4y ago
Your story sounds outrageous initially (10 million dollars!) but if this is a 5, 50, maybe 500 billion company, that number isn't so crazy. If it's related to quality control of a manufactured product, large changes in errors would seem to be a significant problem. Moreover, if 12 errors can become 24 in a week and nobody did anything about it, what would prevent 24 from becoming 48, or even more, until hundreds of your metrics were out of line and in total chaos? I'll also note you don't really explain "error rate of 2%." Do you mean, all 600 data measures are continuous variables, of which, each measured result may be off by 2% (or 2% is the std dev?), or are these discrete or binary measures, of which 2% are entirely wrong?

keithalewis 4y ago

An Indian chief was asked by his tribe, "Winter is coming, what should we do?" He told them to collect firewood. When they brought that back they asked, "Is this enough?" The chief had forgotten the wisdom of his ancestors so he called the weather service. "Is it going to be cold this winter?" he asked. "Yes!" they told him. So he instructed his tribe to collect more firewood. When they asked again "Is this enough?" He called up the weather service. "So when you say it is going to be cold, exactly how cold?" They told him "All indications are pointing to a very cold winter." He told his tribe to collect even more firewood. When they asked yet again if it was enough, he called the weather service again. "Can you tell me how you know this winter will be so cold?" They told him "It's got to be. The Indians are collecting firewood like crazy."

Eddy_Viscosity2 4y ago
This reminds me of an actual story about book that was being priced at hundreds of thousands of dollars (I looked for a link to this story but couldn't find it). The gist was that a feedback loop from two online book sellers, neither of which had a copy of the book. They automated the pricing such it was based on the other's listing plus a small markup; the idea being that if someone wanted this book, they could buy it from the other seller, add the markup and pass it on the customer. But since both were doing this, one would see the price add a markup and set it as their price. After a while, the other would then see this price and do the same thing. Over time the price just grew and grew.