5 ms·

I feel like it was also a classic case of running before we could crawl. Jumping from A to Z before we could go from 0 to 1. I work at an Residential IoT compa

I feel like it was also a classic case of running before we could crawl. Jumping from A to Z before we could go from 0 to 1. I work at an Residential IoT company, there are quite a few really valid use cases for Big Data and even ML. (Think about predictive failure). We hired more than one expensive data scientist in the past few years, and had big strategies more than once. But at the end of the day it's still "hard" to ask a question such as "if I give you a MAC Address give me the runtime for the last 6 months". We're trying to shoot for the moon, when all I've ever asked is I want an API to show me indoor temp for particular device over a long period.

mywittyname 6y ago
This is absolutely right. And when you think about it, the reason behind has been staring us in the face: people who want to do machine learning approach everything as a machine learning problem. It's really common to see people handwave away the "easy stuff" because they want to get credit for doing the "hard stuff." It's not just the data scientists fault. I once heard our chief data scientist point out that they don't want to hand off a linear regression as a machine learning model -- as if a delivered solution to a problem has a minimal complexity. She absolutely had a point. Clients are paying for a Ph.D. to solve problems in a Ph.D way. If we delivered the client a simple, yet effective solution, there's the risk of blow-back from the client for being too rudimentary. I'm certain this extends attitude extends to in-house data scientists as well. Nobody wants to be the data "scientist" who delivers the work of a data "analyst." Even when the best solution is a simple SQL query. Our company kind of sidesteps this problem by having a tiered approach, where companies are paying for engineering, analysis, visualization, and data science work for all projects. So if a client is at the simple analysis level, we deliver at that level, with the understanding that this is the foundational work for more advanced features. It turns out to be a winning strategy, because while every client wants to land on the moon, most of them figure out that they are perfectly happy to with a Cessna once they have one.
- RhysU 6y ago
  > Clients are paying for a Ph.D. to solve problems in a Ph.D way. Ideally, "in a PhD way" is with careful attention to problem framing, understanding prior art, and well-structured research roadmaps. I worry about PhD graduates who seemingly never spent much time hanging out with postdocs. Advisors teach a lot, but some approach considerations can be gleaned more easily from postdocs gunning for academic posts.
- pm90 6y ago
  How good are data scientists in building reliable, scalable systems? My anecdotal experience has been that many don’t bother or care to learn good software development practices, so the systems they build almost always work well for specific use cases but are hard to productionize.
pbourke 6y ago
Everyone wants to fire up Tensorflow, Keras and PyTorch these days. Fewer people want to work in Airflow and SSIS, spend days tuning ETL, etc. This is the domain of data engineering, which bridges software engineering and data science with a dash of devops. I’ve been working in this field for a couple of years and it’s clear to me that data engineering is a necessary foundation and impact multiplier for data science.
- berzerk 6y ago
  I'm finding myself really enjoying this type of work and I think I would like to specialize in it. Any good learning resources you used on your path to where you are now?
- tajd 6y ago
  How might you recommend moving into this field more?
- stainforth 6y ago
  Any formalized paths I could take to enter this field?
  - castlecrasher2 6y ago
    I got into data engineering by starting in ETL (DataStage) and learning about cloud services (AWS) on my own and getting my next job in a cloud-based SaaS startup.
- deleted 6y ago
  [deleted]
- jnwatson 6y ago
  Don't forget data cleaning. A huge issue I've seen is just getting sufficient data of a high enough quality. Also, (for supervised classification problems) labelling is a big problem. It is almost as if we need a "data janitor" title.
  - 6y ago
throwaway7281 6y ago
My impression too. I earn my money turning your mess into a data "landscape" - I saw people wanting to jump on the ML wagon, who did not even heard of version control for code before. Not a winter, no, but a long bumpy road ahead.
- teraku 6y ago
  This is my job, too. And it involves so many politics, since the first thing companies do is hiring data scientists and afterwards figuring out they need clean data but they already have all these "products" that data scientists build that are a mess built ontop of a mess...

mywittyname 6y ago

This is absolutely right. And when you think about it, the reason behind has been staring us in the face: people who want to do machine learning approach everything as a machine learning problem. It's really common to see people handwave away the "easy stuff" because they want to get credit for doing the "hard stuff." It's not just the data scientists fault. I once heard our chief data scientist point out that they don't want to hand off a linear regression as a machine learning model -- as if a delivered solution to a problem has a minimal complexity. She absolutely had a point. Clients are paying for a Ph.D. to solve problems in a Ph.D way. If we delivered the client a simple, yet effective solution, there's the risk of blow-back from the client for being too rudimentary. I'm certain this extends attitude extends to in-house data scientists as well. Nobody wants to be the data "scientist" who delivers the work of a data "analyst." Even when the best solution is a simple SQL query. Our company kind of sidesteps this problem by having a tiered approach, where companies are paying for engineering, analysis, visualization, and data science work for all projects. So if a client is at the simple analysis level, we deliver at that level, with the understanding that this is the foundational work for more advanced features. It turns out to be a winning strategy, because while every client wants to land on the moon, most of them figure out that they are perfectly happy to with a Cessna once they have one.

RhysU 6y ago
> Clients are paying for a Ph.D. to solve problems in a Ph.D way. Ideally, "in a PhD way" is with careful attention to problem framing, understanding prior art, and well-structured research roadmaps. I worry about PhD graduates who seemingly never spent much time hanging out with postdocs. Advisors teach a lot, but some approach considerations can be gleaned more easily from postdocs gunning for academic posts.
pm90 6y ago
How good are data scientists in building reliable, scalable systems? My anecdotal experience has been that many don’t bother or care to learn good software development practices, so the systems they build almost always work well for specific use cases but are hard to productionize.

pbourke 6y ago

Everyone wants to fire up Tensorflow, Keras and PyTorch these days. Fewer people want to work in Airflow and SSIS, spend days tuning ETL, etc. This is the domain of data engineering, which bridges software engineering and data science with a dash of devops. I’ve been working in this field for a couple of years and it’s clear to me that data engineering is a necessary foundation and impact multiplier for data science.

berzerk 6y ago
I'm finding myself really enjoying this type of work and I think I would like to specialize in it. Any good learning resources you used on your path to where you are now?
tajd 6y ago
How might you recommend moving into this field more?
stainforth 6y ago
Any formalized paths I could take to enter this field?
- castlecrasher2 6y ago
  I got into data engineering by starting in ETL (DataStage) and learning about cloud services (AWS) on my own and getting my next job in a cloud-based SaaS startup.
deleted 6y ago
[deleted]
jnwatson 6y ago
Don't forget data cleaning. A huge issue I've seen is just getting sufficient data of a high enough quality. Also, (for supervised classification problems) labelling is a big problem. It is almost as if we need a "data janitor" title.
- 6y ago

throwaway7281 6y ago

My impression too. I earn my money turning your mess into a data "landscape" - I saw people wanting to jump on the ML wagon, who did not even heard of version control for code before. Not a winter, no, but a long bumpy road ahead.

teraku 6y ago
This is my job, too. And it involves so many politics, since the first thing companies do is hiring data scientists and afterwards figuring out they need clean data but they already have all these "products" that data scientists build that are a mess built ontop of a mess...