3 ms·

My first job as a "Data Scientist" (it wasn't called that, but the work was the same) was for a small gaming shop, around 2011. It involved applying econometri

My first job as a "Data Scientist" (it wasn't called that, but the work was the same) was for a small gaming shop, around 2011. It involved applying econometric analysis and doing simple statistical testing on the player data sets. I realized quickly that knowing how to do statistical testing was only a very small portion of what it took to create value in such a role. At the time, I didn't even know (but learned) SQL. Everything I wanted to do involved teaming up with a developer, which wasn't efficient in a small operation. So I learned to program. I continue to enjoy skilling-up, most recently learning cloud-tech to enable me to deploy data tools I develop. The most valuable people in the data chain will be those that can take idea to near-production. Running ML libraries over clean datasets is overrated. The fact is, 80% of the value of "Data Science" comes from KPIs and basic stuff.

hardtke 6y ago
> The most valuable people in the data chain will be those that can take idea to near-production. Having hired many data scientists/ML engineers over the years, people that build robust automated intelligence directly into products are extremely rare. I've estimated a maximum of 10K people in the entire world. They also command the highest salaries, not coincidentally. Very few people have both the statistics and engineering backgrounds as well as temperament to be successful, particularly when the problem requires new data sources or new types of models. There are some real simple practical hurdles such as the need to implement robust tracking that allows data snapshotting at the time when a decision needs to be made without affecting product performance, as well as figuring out how to gather data on users/situations that are actually important for moving the needle (first time users, casual users, etc.) There is also a mismatch between the best frameworks for prototyping/research and implementation which (at least at the companies I've worked for) can be summarized as "Java is good for application development, not ML. Python is good for ML, not application development."

hardtke 6y ago

> The most valuable people in the data chain will be those that can take idea to near-production. Having hired many data scientists/ML engineers over the years, people that build robust automated intelligence directly into products are extremely rare. I've estimated a maximum of 10K people in the entire world. They also command the highest salaries, not coincidentally. Very few people have both the statistics and engineering backgrounds as well as temperament to be successful, particularly when the problem requires new data sources or new types of models. There are some real simple practical hurdles such as the need to implement robust tracking that allows data snapshotting at the time when a decision needs to be made without affecting product performance, as well as figuring out how to gather data on users/situations that are actually important for moving the needle (first time users, casual users, etc.) There is also a mismatch between the best frameworks for prototyping/research and implementation which (at least at the companies I've worked for) can be summarized as "Java is good for application development, not ML. Python is good for ML, not application development."