4 ms·

> Do you want to be a Ford Factory Worker? maybe not but the implication is that there are others who will do it Unfortunately the vast majority of data scient

> Do you want to be a Ford Factory Worker? maybe not but the implication is that there are others who will do it Unfortunately the vast majority of data scientists I've worked with are already more than half way there. Very few data scientists know how to actually model a problem. They only know how to take data, turn it into a matrix and run it through a grid search algorithm on a bucket of SKlearn models and parameters. In fact I'm sure there are data scientists reading this saying to themselves "wait, isn't that modeling?". The frustrating part of this is, in my experience, there are a ton of important modeling problems to be solved at most companies, and yet the huge number of data scientists hired out there don't even see modeling problems that don't fit the form y = f(x). A good quick sanity check of this is to ask a data scientist how they would model churn. An overwhelming majority will say "I'll take N days of data in the past, label who churned there, build a model and then predict who will churn in the next N days", which is absolutely the incorrect way to model churn. If you ask these people about censored data they'll just shrug their shoulders, completely ignorant of the fact that for decades people have been solving this exact same problem in the medical research community using survival analysis. In reality there is only a small subset of real business problems that can be quickly modeled in SKlearn, and yet this type of "modeling" is the large majority of what data scientists are doing today.

jwilber 4y ago
Your argument reads as, “data scientists only know machine learning! Not me, I use popular models from statistics too!” Not trying to sound offensive, but it’s the same aggressive stance repeated on twitter by every other stats/econ phd. And it’s hardly ever true at any big tech company, where most data scientists (especially those writing the design docs) have PhDs in fields ripe with popular non-ml methods. I’ve seen far more e.g. fixed-effect models in production than neural networks. Far more synthetic controls and d-in-ds in research papers than ml models. Far more survey sampling methodologies and power studies than multi-armed bandits. All from the “very few” data scientists who apparently haven’t heard of these methods. There’s a big difference between data scientists working in industry (at least what I’ve seen) and the “learn ml in 10 minutes!” blog post authors on medium.

jwilber 4y ago

Your argument reads as, “data scientists only know machine learning! Not me, I use popular models from statistics too!” Not trying to sound offensive, but it’s the same aggressive stance repeated on twitter by every other stats/econ phd. And it’s hardly ever true at any big tech company, where most data scientists (especially those writing the design docs) have PhDs in fields ripe with popular non-ml methods. I’ve seen far more e.g. fixed-effect models in production than neural networks. Far more synthetic controls and d-in-ds in research papers than ml models. Far more survey sampling methodologies and power studies than multi-armed bandits. All from the “very few” data scientists who apparently haven’t heard of these methods. There’s a big difference between data scientists working in industry (at least what I’ve seen) and the “learn ml in 10 minutes!” blog post authors on medium.