5 ms·

Besides the demands of agile and corporate culture, is there other reasons why data science folks are leaving (if true)? For instance lack of cooperation, or no

Besides the demands of agile and corporate culture, is there other reasons why data science folks are leaving (if true)? For instance lack of cooperation, or not using the insights/recommendations?

felideon 9y ago
Our data scientist left because instead of data science, he ended up doing more ETL and data engineering support than what he originally had signed up for. A full-time data engineer should probably have been hired. I wonder if this leads to an analogous discussion as http://wiki.c2.com/?ArchitectsDontCode http://wiki.c2.com/?ArchitectsDontCode.
- crescentfresh 9y ago
  Not too familiar with the data-this, data-that space. What does a data engineer do? What does a data scientist do? Does either code?
  - amrrs 9y ago
    Yes, Ideally, A data scientist works on creating the model or logic while a Data Engineer works on the part of deploying the same.
  - ct0 9y ago
    They both code. The engineer would pull the data from various sources and structure it in a way that the data scientist is happy with. (Tidy format) The DS would the code the models based on the engineer's work, which takes considerably less time.
  - altstar 9y ago
    In our environment, the scientist builds the models, the engineer makes them run. Both can code (the scientist to build models, the engineer to execute models).
  - RHSman2 9y ago
    One does math based hypothesis’s and the other moves data around and compute’s hypothesis’. Making a dashboard is not data science. At all.
- ct0 9y ago
  I know someone who is hired as a data scientist and is mostly doing the same ETL work. The science is literally 10% of the work anyway, and all DS should know that.
- js8 9y ago
  Dammit, Gartner, make up your mind! You told me last year that one data scientist is gonna solve all my problems. Now you're telling me I need to babysit him with five data engineers, because he is not going to get his hands dirty with data??
- megaman22 9y ago
  Nobody ever gets to do the fun parts of their job. Who would sign up if they told you up front that your main contributions would be shuffling email, filling out TPS reports, attending pointless meetings, and being roadblocked every time you tried to automate, eliminate, or improve that drudgery? You can smash your head against the wall trying to make things better, but all that ends up doing is pissing you off and giving you a headache.
apohn 9y ago
Until recently I led a Data Science team (manager and IC) at a non-software Fortune 10. I, and everybody else on the team, left within a year of joining. The division where I was in hired me to grow both Data Science practice and the team. The company was enormous, so there were data science teams all across the company, all operating in silos. The biggest problem can be summarized like this: If you want to run your business differently, let the new people you hire do something different. If you want to do the same old thing and just act you are doing something differently, just teach the old people to talk like the new people. I've got plenty of anecdotes from the year, but here's a simple one. The division had an existing "Research and Analytics" group that used linear regression in Excel to develop models. They would develop models on awful (read: highly-multicollinear) data with 100 rows and get "R-squared of .98." Large long-term business decisions were made on these models. Over the course of the year I fought this constantly. Why do you need more/better data when the other team is okay with the current data? Why are you struggling to get your R-squared to >.90 (output of any type of model had to be R-squared) when the other team can do it so easily? Why do you need to speak to so many people to understand the data better? What I realized over the course of the year was that nobody actually cared about the models unless they matched the "gut" of senior management. So it was easy for the other team to p-hack until they got a slidedeck people would like. I think there are plenty of problems that lend themselves to good Data Science. These are ones where you can properly identify what a prediction or better understanding of data (e.g. variable importance, coefficient confidence intervals) means in terms of resources (e.g. people, money, time). You can also identify what it means to not have that data insight or model. There are also plenty of business areas where the link between resource investment and long term success is less clear (e.g. Marketing, Sales Incentives) and more gut and smart work than science. A lot of people are trying to clear up that grey with data science, but I go back to what I said before: you need to let people think and try different things over time to see the value of those experiments. Comically enough, the Research and Analysis team picked up all the lingo we used and became a huge success: work the old way, talk in the new way. Management decided their Data Science adventure was a failed experiment and we just hadn't lived up to the hype.
- Maven911 9y ago
  Wow sorry to hear but thank you for this!

Besides the demands of agile and corporate culture, is there other reasons why data science folks are leaving (if true)? For instance lack of cooperation, or not using the insights/recommendations?

felideon 9y ago

Our data scientist left because instead of data science, he ended up doing more ETL and data engineering support than what he originally had signed up for. A full-time data engineer should probably have been hired. I wonder if this leads to an analogous discussion as http://wiki.c2.com/?ArchitectsDontCode http://wiki.c2.com/?ArchitectsDontCode.

crescentfresh 9y ago
Not too familiar with the data-this, data-that space. What does a data engineer do? What does a data scientist do? Does either code?
- amrrs 9y ago
  Yes, Ideally, A data scientist works on creating the model or logic while a Data Engineer works on the part of deploying the same.
- ct0 9y ago
  They both code. The engineer would pull the data from various sources and structure it in a way that the data scientist is happy with. (Tidy format) The DS would the code the models based on the engineer's work, which takes considerably less time.
- altstar 9y ago
  In our environment, the scientist builds the models, the engineer makes them run. Both can code (the scientist to build models, the engineer to execute models).
- RHSman2 9y ago
  One does math based hypothesis’s and the other moves data around and compute’s hypothesis’. Making a dashboard is not data science. At all.
ct0 9y ago
I know someone who is hired as a data scientist and is mostly doing the same ETL work. The science is literally 10% of the work anyway, and all DS should know that.
js8 9y ago
Dammit, Gartner, make up your mind! You told me last year that one data scientist is gonna solve all my problems. Now you're telling me I need to babysit him with five data engineers, because he is not going to get his hands dirty with data??
megaman22 9y ago
Nobody ever gets to do the fun parts of their job. Who would sign up if they told you up front that your main contributions would be shuffling email, filling out TPS reports, attending pointless meetings, and being roadblocked every time you tried to automate, eliminate, or improve that drudgery? You can smash your head against the wall trying to make things better, but all that ends up doing is pissing you off and giving you a headache.

apohn 9y ago

Until recently I led a Data Science team (manager and IC) at a non-software Fortune 10. I, and everybody else on the team, left within a year of joining. The division where I was in hired me to grow both Data Science practice and the team. The company was enormous, so there were data science teams all across the company, all operating in silos. The biggest problem can be summarized like this: If you want to run your business differently, let the new people you hire do something different. If you want to do the same old thing and just act you are doing something differently, just teach the old people to talk like the new people. I've got plenty of anecdotes from the year, but here's a simple one. The division had an existing "Research and Analytics" group that used linear regression in Excel to develop models. They would develop models on awful (read: highly-multicollinear) data with 100 rows and get "R-squared of .98." Large long-term business decisions were made on these models. Over the course of the year I fought this constantly. Why do you need more/better data when the other team is okay with the current data? Why are you struggling to get your R-squared to >.90 (output of any type of model had to be R-squared) when the other team can do it so easily? Why do you need to speak to so many people to understand the data better? What I realized over the course of the year was that nobody actually cared about the models unless they matched the "gut" of senior management. So it was easy for the other team to p-hack until they got a slidedeck people would like. I think there are plenty of problems that lend themselves to good Data Science. These are ones where you can properly identify what a prediction or better understanding of data (e.g. variable importance, coefficient confidence intervals) means in terms of resources (e.g. people, money, time). You can also identify what it means to not have that data insight or model. There are also plenty of business areas where the link between resource investment and long term success is less clear (e.g. Marketing, Sales Incentives) and more gut and smart work than science. A lot of people are trying to clear up that grey with data science, but I go back to what I said before: you need to let people think and try different things over time to see the value of those experiments. Comically enough, the Research and Analysis team picked up all the lingo we used and became a huge success: work the old way, talk in the new way. Management decided their Data Science adventure was a failed experiment and we just hadn't lived up to the hype.

Maven911 9y ago
Wow sorry to hear but thank you for this!