6 ms·

This is an awesome analysis of the situation. Some things I have noticed as a data scientist of 4 years so far: - Increasingly, data scientist is just being us

This is an awesome analysis of the situation. Some things I have noticed as a data scientist of 4 years so far: - Increasingly, data scientist is just being used in place of senior analyst because it attracts more applications. - At the firms I've worked that are software tech companies, there was an outsized interest in mid-level software engineers wanting to be data scientists, mostly because the career development prospects at that stage are grim and data science usually means a pay bump. This demand has had the opposite effect - software shops are leery of promoting engineers to data scientists for fear of inciting contention among the ranks. - Building on the data scientist usually means senior analyst, it has also come to mean analyst that can build their sql query into a scheduled ETL or daily process of some sort. You work in pandas instead of excel sorta thing. - I have personally gotten all my data science jobs from talking about the business side of things. I think engineers approaching the field from a hard-skills perspective is totally wrong. My last technical take home was in a language I had never used before and likewise my execution was shitty, but I was able to well explain the problem, how the data could be used to predict the variation, and how the data science product fit into the business. I got an offer before I left the building.

learc83 7y ago
>as an outsized interest in mid-level software engineers wanting to be data scientists, mostly because the career development prospects at that stage are grim What are you basing this on? Senior software engineer jobs are a lot easier to come by than data scientist jobs and from what I've seen, pay better than the average data scientist job as well.
- tjpaudio 7y ago
  I am just going on my experience, but according to glassdoor data scientists of the same years of experience earn about 20k more than senior software engineers here in Boston.
  - touristtam 7y ago
    Could be really depending on location. Current Software Engineer, for full disclosure, pondering how to move away from the prevalent web dev job market where I currently reside. Data science seems to pay on par.
  - codingslave 7y ago
    Glassdoor is not accurate
    - treypitt 7y ago
      source?
  - sidlls 7y ago
    Depends on the company. I fought the senior engineering leadership and HR to get data scientists on the same scale as engineers (they had previously been lower--much lower).
  - learc83 7y ago
    Even if that's true, the OP was talking about mid level software engineers, so they would be leaving a field where they had 5 or so years of experience and moving into one where they had 0.
- codingslave 7y ago
  I agree with you, but I think data science has the perception of more money and prestige. While senior software engineers often get I paid more, I do believe that data scientists have more opportunity to be involved in strategy and business decisions, which can help one get more exposure to high ranking employees. But this is not always the case.
  - Breza 7y ago
    Data scientists do have opportunities to work with management, but the realities of working with MBAs can be surprising. I found myself reading corporate finance textbooks to be able to fully participate when I started in data science.
- panzagl 7y ago
  This is truer outside of the Bay area- non-tech focused companies that see IT as a cost center, but have somehow decided they need to do 'Big Data Analytics'.
  - 8ytecoder 7y ago
    It’s the “it” thing right now. Everyone wants to “leverage” their data and projects in that space get readily funded. It’s also seen slightly better than the typical IT-is-cost-center mentality because of the immediate potential benefit or the risk of losing to a competitor.
    - turk73 7y ago
      It's the "it" but 90% of it is sh-it.
- wil421 7y ago
  OP had I right in the first sentence. Senior Analysts not Senior Software Engineers.
- alexgmcm 7y ago
  I went into Data Science after grad school 5 years ago. If it had been as difficult then as it is now I would probably have chosen software engineering. Honestly, I feel software engineering might be better anyway as it's much easier to demonstrate value building features and shipping products rather than endless analysis and questionable models.
minimaxir 7y ago
The rise of Medium thought pieces/MOOCs has created the conception that data science jobs are a 40-hours-a-week Kaggle competition, whereas the reality is much, much different/less exciting to write thought pieces about. (I wrote a blog post last year about that phenomena: https://minimaxir.com/2018/10/data-science-protips/ https://minimaxir.com/2018/10/data-science-protips/ )
- goatinaboat 7y ago
  That is not limited to data science. I remember the horror of an intern when he realised that real work wasn’t at all like solving neat little puzzles on HackerRank. He discovered that he didn’t really want to be a programmer after all...
  - theflyinghorse 7y ago
    I mean, hackerrank puzzles IS how most companies interview so it isn't unreasonable to assume that that's what the job is like. Presumably orgs would ask questions related to the problems they are solving during an interview
  - Aperocky 7y ago
    It would be a true horror if real work is like solving hackerrank puzzles --- everything is predefined and you're given a set task for a set amount of time, surely a robot could do the same?
- collyw 7y ago
  To be fair that is the same with many jobs. It always sounds like you will be building the next great thing from scratch, but end up spending far more time fixing other people's crappy code.
  - vmurthy 7y ago
    Ask any Postdoc or PhD candidate :-). Even they aren't spared from the "discover the next great thing" phenomenon. Perhaps the best thing for students/new employees to realize is that while they may be on the path to building the next great thing, that path is full of potholes and grunt work. Will save a lot of disappointment when they hit the road.
throwaway713 7y ago
> data science usually means a pay bump That’s interesting. I’m at one of F/G and a lot of the data scientists want to go the other direction to software engineering because we receive about 60% of the RSUs that they do. A few people on my team actually did switch; they said they found the data science work more interesting but an additional $40-100k per year can make a really big difference over the long run.
- chibg10 7y ago
  I think this is dependent on what someone means when they say "data scientist" (mentioned as Type A vs Type B in the article). Facebook and Google "data scientists" (meaning those who hold the title) are really more like analysts -- they analyze data to inform decisions and use a lot of SQL. They make prototype models (usually based on less cutting-edge techniques) that get passed to engineering teams if they become worthwhile to scale/formalize. These folks get paid less than SDEs usually. The other type of "data scientist" is basically an SDE (maybe SDE-lite) with research-level ML skills. These get paid similarly (or higher in some cases) than SDEs. I believe Facebook and Google call these SDEs. Sometimes the term "applied scientist" is used to describe these at other companies as well.
  - throwaway713 7y ago
    I have a research background and do pretty heavy duty machine learning. It’s not analyst work (the internal job family is “applied scientist” while the external facing title is “data scientist”). It’s still not SWE compensation. As far as I’m aware, there is only one team of data scientists that makes the same as SWEs at my company.
    - laichzeit0 7y ago
      Does whatever you build go into production or do you need someone else / team “take care of that part”. That is where the “data science is just Statistics” people’s wheels come off when they realise production ML needs senior software engineering background.
perl4ever 7y ago
"analyst that can build their sql query into a scheduled ETL or daily process of some sort. You work in pandas instead of excel sorta thing" In my experience from my old employer...clients like Google get billed $120/hr for SQL analysts' services; ten years ago, staff earned $20/hr or a little more, and today, they've replaced almost everyone with offshore employees making $3-4/hr.
Aperocky 7y ago
I used to work as a data scientist. This title is not what it should be, I witnessed first hand how business thought data science is the magic that they would be able to rely on to consistently deliver impossible work. There are so much junk that gets shuffled our way and have the expectation that gold be coming out of them. It feels like the financial bubble of 2008, except it's with the inflation of the position and the 'clout' of data science. There's nothing wrong in data science itself, just like there's nothing wrong with mortgage. But the current trend of software engineer/ non-software engineer moving into data-science is not sustainable. Things will break before it's fixed again, I've always considered myself a software engineer first and foremost, just with some extra machine learning/stats knowledge, and I'm glad to be out of that position now as it looks like we're in for a reckoning soon.
Latteland 7y ago
Where is the place that mid level software engineers think there are not good career prospects and they'd get paid more? I can only guess it's a place where there aren't many dev jobs. My experience is in the Seattle area and we are begging for people to even apply for jobs. There are 10,000 jobs easily in Seattle. My company would love to grow its dev force 50% and we can only get people by hiring them away from another company (perhaps an obvious comment :-)), and by hiring new college grads. If your job is working you too hard, with not enough pay, then people here get another job. It seems harder to high people with some experience at my company anyway. New college grads make 120k+ at top companies (we are a startup but not a unicorn, we pay a little more than that).
- blackkat 7y ago
  Pardon the question: What skills are you looking for that define "mid level software engineer"? I'm a long-time engineer at a single company. I've been more and more tempted to strike out elsewhere as I feel like there is nowhere else for me to move into position/pay wise where I am. I need help to determine how to frame the skill level that I have and/or where to focus on so that I can claim/apply with a certain level of skill.
  - therealdrag0 7y ago
    Title placement is often just a rough estimate based on years of experience. If you've been at a single company for a while, the usual advice is to break your time up into roles/projects on your resume.
    - blackkat 7y ago
      Thank you

learc83 7y ago

>as an outsized interest in mid-level software engineers wanting to be data scientists, mostly because the career development prospects at that stage are grim What are you basing this on? Senior software engineer jobs are a lot easier to come by than data scientist jobs and from what I've seen, pay better than the average data scientist job as well.

tjpaudio 7y ago
I am just going on my experience, but according to glassdoor data scientists of the same years of experience earn about 20k more than senior software engineers here in Boston.
- touristtam 7y ago
  Could be really depending on location. Current Software Engineer, for full disclosure, pondering how to move away from the prevalent web dev job market where I currently reside. Data science seems to pay on par.
- codingslave 7y ago
  Glassdoor is not accurate
  - treypitt 7y ago
    source?
- sidlls 7y ago
  Depends on the company. I fought the senior engineering leadership and HR to get data scientists on the same scale as engineers (they had previously been lower--much lower).
- learc83 7y ago
  Even if that's true, the OP was talking about mid level software engineers, so they would be leaving a field where they had 5 or so years of experience and moving into one where they had 0.
codingslave 7y ago
I agree with you, but I think data science has the perception of more money and prestige. While senior software engineers often get I paid more, I do believe that data scientists have more opportunity to be involved in strategy and business decisions, which can help one get more exposure to high ranking employees. But this is not always the case.
- Breza 7y ago
  Data scientists do have opportunities to work with management, but the realities of working with MBAs can be surprising. I found myself reading corporate finance textbooks to be able to fully participate when I started in data science.
panzagl 7y ago
This is truer outside of the Bay area- non-tech focused companies that see IT as a cost center, but have somehow decided they need to do 'Big Data Analytics'.
- 8ytecoder 7y ago
  It’s the “it” thing right now. Everyone wants to “leverage” their data and projects in that space get readily funded. It’s also seen slightly better than the typical IT-is-cost-center mentality because of the immediate potential benefit or the risk of losing to a competitor.
  - turk73 7y ago
    It's the "it" but 90% of it is sh-it.
wil421 7y ago
OP had I right in the first sentence. Senior Analysts not Senior Software Engineers.
alexgmcm 7y ago
I went into Data Science after grad school 5 years ago. If it had been as difficult then as it is now I would probably have chosen software engineering. Honestly, I feel software engineering might be better anyway as it's much easier to demonstrate value building features and shipping products rather than endless analysis and questionable models.

minimaxir 7y ago

The rise of Medium thought pieces/MOOCs has created the conception that data science jobs are a 40-hours-a-week Kaggle competition, whereas the reality is much, much different/less exciting to write thought pieces about. (I wrote a blog post last year about that phenomena: https://minimaxir.com/2018/10/data-science-protips/ https://minimaxir.com/2018/10/data-science-protips/ )

goatinaboat 7y ago
That is not limited to data science. I remember the horror of an intern when he realised that real work wasn’t at all like solving neat little puzzles on HackerRank. He discovered that he didn’t really want to be a programmer after all...
- theflyinghorse 7y ago
  I mean, hackerrank puzzles IS how most companies interview so it isn't unreasonable to assume that that's what the job is like. Presumably orgs would ask questions related to the problems they are solving during an interview
- Aperocky 7y ago
  It would be a true horror if real work is like solving hackerrank puzzles --- everything is predefined and you're given a set task for a set amount of time, surely a robot could do the same?
collyw 7y ago
To be fair that is the same with many jobs. It always sounds like you will be building the next great thing from scratch, but end up spending far more time fixing other people's crappy code.
- vmurthy 7y ago
  Ask any Postdoc or PhD candidate :-). Even they aren't spared from the "discover the next great thing" phenomenon. Perhaps the best thing for students/new employees to realize is that while they may be on the path to building the next great thing, that path is full of potholes and grunt work. Will save a lot of disappointment when they hit the road.

throwaway713 7y ago

> data science usually means a pay bump That’s interesting. I’m at one of F/G and a lot of the data scientists want to go the other direction to software engineering because we receive about 60% of the RSUs that they do. A few people on my team actually did switch; they said they found the data science work more interesting but an additional $40-100k per year can make a really big difference over the long run.

chibg10 7y ago
I think this is dependent on what someone means when they say "data scientist" (mentioned as Type A vs Type B in the article). Facebook and Google "data scientists" (meaning those who hold the title) are really more like analysts -- they analyze data to inform decisions and use a lot of SQL. They make prototype models (usually based on less cutting-edge techniques) that get passed to engineering teams if they become worthwhile to scale/formalize. These folks get paid less than SDEs usually. The other type of "data scientist" is basically an SDE (maybe SDE-lite) with research-level ML skills. These get paid similarly (or higher in some cases) than SDEs. I believe Facebook and Google call these SDEs. Sometimes the term "applied scientist" is used to describe these at other companies as well.
- throwaway713 7y ago
  I have a research background and do pretty heavy duty machine learning. It’s not analyst work (the internal job family is “applied scientist” while the external facing title is “data scientist”). It’s still not SWE compensation. As far as I’m aware, there is only one team of data scientists that makes the same as SWEs at my company.
  - laichzeit0 7y ago
    Does whatever you build go into production or do you need someone else / team “take care of that part”. That is where the “data science is just Statistics” people’s wheels come off when they realise production ML needs senior software engineering background.

perl4ever 7y ago

"analyst that can build their sql query into a scheduled ETL or daily process of some sort. You work in pandas instead of excel sorta thing" In my experience from my old employer...clients like Google get billed $120/hr for SQL analysts' services; ten years ago, staff earned $20/hr or a little more, and today, they've replaced almost everyone with offshore employees making $3-4/hr.

Aperocky 7y ago

I used to work as a data scientist. This title is not what it should be, I witnessed first hand how business thought data science is the magic that they would be able to rely on to consistently deliver impossible work. There are so much junk that gets shuffled our way and have the expectation that gold be coming out of them. It feels like the financial bubble of 2008, except it's with the inflation of the position and the 'clout' of data science. There's nothing wrong in data science itself, just like there's nothing wrong with mortgage. But the current trend of software engineer/ non-software engineer moving into data-science is not sustainable. Things will break before it's fixed again, I've always considered myself a software engineer first and foremost, just with some extra machine learning/stats knowledge, and I'm glad to be out of that position now as it looks like we're in for a reckoning soon.

Latteland 7y ago

Where is the place that mid level software engineers think there are not good career prospects and they'd get paid more? I can only guess it's a place where there aren't many dev jobs. My experience is in the Seattle area and we are begging for people to even apply for jobs. There are 10,000 jobs easily in Seattle. My company would love to grow its dev force 50% and we can only get people by hiring them away from another company (perhaps an obvious comment :-)), and by hiring new college grads. If your job is working you too hard, with not enough pay, then people here get another job. It seems harder to high people with some experience at my company anyway. New college grads make 120k+ at top companies (we are a startup but not a unicorn, we pay a little more than that).

blackkat 7y ago
Pardon the question: What skills are you looking for that define "mid level software engineer"? I'm a long-time engineer at a single company. I've been more and more tempted to strike out elsewhere as I feel like there is nowhere else for me to move into position/pay wise where I am. I need help to determine how to frame the skill level that I have and/or where to focus on so that I can claim/apply with a certain level of skill.
- therealdrag0 7y ago
  Title placement is often just a rough estimate based on years of experience. If you've been at a single company for a while, the usual advice is to break your time up into roles/projects on your resume.
  - blackkat 7y ago
    Thank you