4 ms·

For interest, the raw data is published here: https://www.kaggle.com/kaggle/kaggle-survey-2017 https://www.kaggle.com/kaggle/kaggle-survey-2017 And some early

For interest, the raw data is published here: https://www.kaggle.com/kaggle/kaggle-survey-2017 https://www.kaggle.com/kaggle/kaggle-survey-2017 And some early analysis from our community here: https://www.kaggle.com/crawford/analyzing-the-analyzers https://www.kaggle.com/crawford/analyzing-the-analyzers Some things that jumped out at me: 1. more people learn data science and ML from MOOCs than university courses 2. Tensorflow the tech people most want to learn in the next year 3. 40% of people survey spend >1-2 hours per week searching for another job. Surprising given all companies complain about the difficulty in finding data scientists/machine learners.

PeachPlum 9y ago
Re 3. "There's a skills shortage (at the price we want to pay)"
- technologia 9y ago
  I wish some of these companies would embrace having offices in places other than those with very high cost of living.
- antgoldbloom 9y ago
  Yeh. The median salary for a machine learning engineer, which is definitely higher than what most companies are used to paying (even for software eng roles). My argument is that machine learning also higher leverage than most roles. One algorithm written by one machine learner can generate a huge ROI. Think of an algorithm to predict loan defaults or customer churn for a bank. That algorithm in the hands of a great machine learner can generate a huge ROI.
rndmwlk 9y ago
>3. 40% of people survey spend >1-2 hours per week searching for another job. Surprising given all companies complain about the difficulty in finding data scientists/machine learners. Not too surprised about this. I think a lot of people who go into the field want to do more interesting things than what they find being used in the field. I don't think pay is necessarily the gap here, as others have pointed out, so much as interesting work (or at least the intersection of interesting work and pay).
huac 9y ago
> 1. more people learn data science and ML from MOOCs than university courses More people, out of the subset of people on Kaggle. Lot of selection bias there!
apohn 9y ago
>40% of people survey spend >1-2 hours per week searching for another job. Surprising given all companies complain about the difficulty in finding data scientists/machine learners. I've hired data scientists in the past. One thing I found is that a lot of interviewees want to talk about all the algorithms (e.g. Gradient Boosting) they've used and are not able to describe how they thought through the problems before they applied the algorithms. It's easier to find somebody who downloaded some mostly clean data, then copy/pasted some code than a person actually thinks through the quantification of a problem. There are a lot of buzzword artists out there. This is important because in a lot of organizations the business problems have not yet been quantified in a way that lends itself to getting meaningful and valid results from an algorithm. The Data Scientist has to be able to work with others to quantify a problem. Or at a minimum, recognize that there are issues with the current way the problem is quantified and think of ways to improve it. It's much easier to teach somebody to run a data algorithm than it is to actually understand a business problem. There are issues with people doing the hiring as well. In my last job (not a software company), the VP of the group had pushed to get headcount for a data science team and was fearful of making the wrong hire because he didn't want to say "We hired a data scientist at 2X-3X the cost of a Business Analyst and that was a bad hire." The end result was a massive amount of paralysis, an insanely long and convoluted job description, and complaints about the hiring pipeline.
- zebrafish 9y ago
  I agree with your assessment that a lot of times the business problems have been put into a form that lends itself to exploitation by machine learning. Sometimes a company has a lot of data that's actually useless. Most of the time, I've found that business people do not understand the value of data. Often I hear, "we have this data set, let's unleash the data scientist on this to tell us something." or "we have this data set but what are the so-whats here?". I spend a lot of my time explaining that there must first be a business objective, a key question, or hypothesis that can then be understood through data. I cannot take a haystack and find the needle that is interesting to you. And if I do find that needle, many times there are no resulting changes made to our strategy. I think we're still in a place where the value in a data scientist is not that she knows how to write: fit <- lm(target ~ ., data = customers) The value exists when she can take a problem from the business, understand how to find a solution with data, and then convey that back to the business in a meaningful way that allows them to easily understand how they can make changes to positively impact the bottom line.
sidlls 9y ago
All of these things coupled with the top complaints data make a lot of sense if one views the items you list as identifying a set of people who are mostly self-taught, have a perhaps insufficiently broad and deep quantitative background, and are looking to merely catch the "data science wave".

PeachPlum 9y ago

Re 3. "There's a skills shortage (at the price we want to pay)"

technologia 9y ago
I wish some of these companies would embrace having offices in places other than those with very high cost of living.
antgoldbloom 9y ago
Yeh. The median salary for a machine learning engineer, which is definitely higher than what most companies are used to paying (even for software eng roles). My argument is that machine learning also higher leverage than most roles. One algorithm written by one machine learner can generate a huge ROI. Think of an algorithm to predict loan defaults or customer churn for a bank. That algorithm in the hands of a great machine learner can generate a huge ROI.

rndmwlk 9y ago

>3. 40% of people survey spend >1-2 hours per week searching for another job. Surprising given all companies complain about the difficulty in finding data scientists/machine learners. Not too surprised about this. I think a lot of people who go into the field want to do more interesting things than what they find being used in the field. I don't think pay is necessarily the gap here, as others have pointed out, so much as interesting work (or at least the intersection of interesting work and pay).

huac 9y ago

> 1. more people learn data science and ML from MOOCs than university courses More people, out of the subset of people on Kaggle. Lot of selection bias there!

apohn 9y ago

>40% of people survey spend >1-2 hours per week searching for another job. Surprising given all companies complain about the difficulty in finding data scientists/machine learners. I've hired data scientists in the past. One thing I found is that a lot of interviewees want to talk about all the algorithms (e.g. Gradient Boosting) they've used and are not able to describe how they thought through the problems before they applied the algorithms. It's easier to find somebody who downloaded some mostly clean data, then copy/pasted some code than a person actually thinks through the quantification of a problem. There are a lot of buzzword artists out there. This is important because in a lot of organizations the business problems have not yet been quantified in a way that lends itself to getting meaningful and valid results from an algorithm. The Data Scientist has to be able to work with others to quantify a problem. Or at a minimum, recognize that there are issues with the current way the problem is quantified and think of ways to improve it. It's much easier to teach somebody to run a data algorithm than it is to actually understand a business problem. There are issues with people doing the hiring as well. In my last job (not a software company), the VP of the group had pushed to get headcount for a data science team and was fearful of making the wrong hire because he didn't want to say "We hired a data scientist at 2X-3X the cost of a Business Analyst and that was a bad hire." The end result was a massive amount of paralysis, an insanely long and convoluted job description, and complaints about the hiring pipeline.

zebrafish 9y ago
I agree with your assessment that a lot of times the business problems have been put into a form that lends itself to exploitation by machine learning. Sometimes a company has a lot of data that's actually useless. Most of the time, I've found that business people do not understand the value of data. Often I hear, "we have this data set, let's unleash the data scientist on this to tell us something." or "we have this data set but what are the so-whats here?". I spend a lot of my time explaining that there must first be a business objective, a key question, or hypothesis that can then be understood through data. I cannot take a haystack and find the needle that is interesting to you. And if I do find that needle, many times there are no resulting changes made to our strategy. I think we're still in a place where the value in a data scientist is not that she knows how to write: fit <- lm(target ~ ., data = customers) The value exists when she can take a problem from the business, understand how to find a solution with data, and then convey that back to the business in a meaningful way that allows them to easily understand how they can make changes to positively impact the bottom line.

sidlls 9y ago

All of these things coupled with the top complaints data make a lot of sense if one views the items you list as identifying a set of people who are mostly self-taught, have a perhaps insufficiently broad and deep quantitative background, and are looking to merely catch the "data science wave".