6 ms·

I'm trying to switch careers into "Data Engineering" now, as a full stack developer who is more interested in ML, and I've found almost no traction internally a

I'm trying to switch careers into "Data Engineering" now, as a full stack developer who is more interested in ML, and I've found almost no traction internally at my company or externally. It looks like I may just accept a full stack position at a good company that does a lot of data science for now, but though I would ask - Where are all these jobs?

PaulHoule 10y ago
"Data Engineering" is most of the work that needs to be done, but I think companies haven't identified it as a category. From my P.O.V., "Full Stack Engineer" is a place you don't want to be because it means putting out fires with whatever junk javascript is in the front end. It seems like everybody who's built a serious javascript application has invented their own Virtual DOM because none of the popular Virtual DOM libraries are good for much other than wasting time and CPU cycles. "Data Scientist" is a bad title in it's own way, in the sense that "Computer Science" is bad, but worse. To a lot of people there is a Brahmin kind of attitude associated with "Scientist" -- i.e. an aversion to getting your hands dirty. Real world data is pretty dirty and you aren't going to get far in getting value out of it unless you spend 80-90% of your time dealing with the dirt.
- pjmlp 10y ago
  There are "Full Stack Engineer" doing pure native applications, which is what I have been doing the last three years after escaping the web back into native land.
- mattnewton 10y ago
  You are correct. I thought full stack meant before building the app start to finish, but the reality is often closer to putting out other people's fires in every layer. It does pay well though and you learn a lot of what can go wrong.
  - PaulHoule 10y ago
    The fact that it pays well makes it a job you're likely to get laid off from. Most managers would rather hire two junior developers so they can screw it up faster or better yet hire some people in another country who are really fast and cheap at screwing it up.
    - mattnewton 10y ago
      That may be true but I'm not worried about that, I worry more about getting comfortable doing useless work. If I got fired it would be so much easier to go back to school, as the dream of lots of money while learning on the side would evaporate.
willis77 10y ago
We (Kaggle) run a data science jobs board (https://www.kaggle.com/jobs https://www.kaggle.com/jobs) that gets a few data engineer listings from time to time. Not all of these are active, but you may find a few interested companies via - https://www.google.com/#q=site:https://www.kaggle.com/jobs+%22data+engineer%22 https://www.google.com/#q=site:https://www.kaggle.com/jobs+%...
- mattnewton 10y ago
  Thank you guys! Doing Kaggle competitions is what got me interested in seriously pursuing ML in the first place. You are all seriously awesome. I'll look again at the board but, I didn't see anything there before that wanted software engineering skills (which I have with industry experience), and not a graduate degree (which I don't), and happened to be commutable from my place just south of the bay. But I will keep looking!
minimaxir 10y ago
ML falls more under a Data Science role than Data Engineering, although ML is much more difficult without proper Data Engineering.
ironchef 10y ago
I see tons of them. If you're interested in ML, you're probably more looking towards data science. Data engineering (in general) is more about getting the data in a state where it can be used (extracted, cleaned, moved, transformed, etc.) at least from what i've commonly seen in the industry. A decent breakdown is here: https://blog.insightdatascience.com/data-science-vs-data-engineering-62da7678adaa https://blog.insightdatascience.com/data-science-vs-data-eng...
alexbeloi 10y ago
You might want to look at "Machine Learning Engineer" positions if you want to do ML in practice, it's starting to be a title I see somewhat often now. As others have pointed out Data Engineering is more about building data pipelines, making architecture decisions for your ML stack, things like that. Less about model building, prototyping and training, which is what I think of when somebody says they 'do' ML.
- mattnewton 10y ago
  Right, I'm not picky about the title. I'm looking at those positions too. The main thing is, I want to be able to contribute using my existing software engineering skills from day 1, while picking up the ML stuff. It's been really hard to basically work an unrelated job during the day and go home and do kaggles for practice, so I am hoping to get more of an intersection as a launching place. Anything touching the data or the models will do :)
achompas 10y ago
My official title is "Data Scientist" although I'm closer to the "ML Engineer" someone else mentions in a child comment. Frankly speaking, if your company doesn't need a data engineer, it won't hire one or move you into that role. They likely don't, either, if you're experiencing this pushback -- data engineers often develop ETL pipelines or data warehouses, both of which are very useful if your company has a data team and very useless if it does not. That said, you may want to move closer to my role. There's actually a shortage of data-savvy people who can also write production software, and you would nicely complement a more research-inclined data scientist or analyst -- someone with far more experience with research/analysis than development.
- mattnewton 10y ago
  I think the company does need data engineers but wants someone with a graduate degree from Stanford or CMU in that position, even though the actual work is in building up infrastructure for those people. And I understand. I've only really got software engineering skills to contribute at this point and I'm picking up the ML from kaggles on the side; I am looking for a position that can increase my overlap between those, because learning at home while working on unrelated stuff is making me move slowly and painfully. Your experience sounds exactly like what I'm looking for - data-savvy writing production code, complementing a research-heavy team I can learn from. How did you get started in that?
  - achompas 10y ago
    I honestly fell into it by luck. I moved to NYC, studied machine learning in grad school, networked my ass off, and landed an internship. From there I went full time as something of an ML engineer at a company with a strong tech culture, and learned as much as I could in both tech and ML/statistics. The rest is history (although I'm by no means a rockstar or whatever). My path is hard to reproduce -- it starts with being in NYC or SF at a specific point in time, before the labor market became saturated with data science bootcamps and PhDs furiously learning Python while working on their dissertations. Your best bet at this point is to produce a few data-related projects (maybe work on open source like scikit-learn and pandas?) and network like crazy. Someone somewhere will have a need for someone like you.
bcbrown 10y ago
You should put your email in your profile. If you're in Seattle, send me an email.

PaulHoule 10y ago

"Data Engineering" is most of the work that needs to be done, but I think companies haven't identified it as a category. From my P.O.V., "Full Stack Engineer" is a place you don't want to be because it means putting out fires with whatever junk javascript is in the front end. It seems like everybody who's built a serious javascript application has invented their own Virtual DOM because none of the popular Virtual DOM libraries are good for much other than wasting time and CPU cycles. "Data Scientist" is a bad title in it's own way, in the sense that "Computer Science" is bad, but worse. To a lot of people there is a Brahmin kind of attitude associated with "Scientist" -- i.e. an aversion to getting your hands dirty. Real world data is pretty dirty and you aren't going to get far in getting value out of it unless you spend 80-90% of your time dealing with the dirt.

pjmlp 10y ago
There are "Full Stack Engineer" doing pure native applications, which is what I have been doing the last three years after escaping the web back into native land.
mattnewton 10y ago
You are correct. I thought full stack meant before building the app start to finish, but the reality is often closer to putting out other people's fires in every layer. It does pay well though and you learn a lot of what can go wrong.
- PaulHoule 10y ago
  The fact that it pays well makes it a job you're likely to get laid off from. Most managers would rather hire two junior developers so they can screw it up faster or better yet hire some people in another country who are really fast and cheap at screwing it up.
  - mattnewton 10y ago
    That may be true but I'm not worried about that, I worry more about getting comfortable doing useless work. If I got fired it would be so much easier to go back to school, as the dream of lots of money while learning on the side would evaporate.

willis77 10y ago

We (Kaggle) run a data science jobs board (https://www.kaggle.com/jobs https://www.kaggle.com/jobs) that gets a few data engineer listings from time to time. Not all of these are active, but you may find a few interested companies via - https://www.google.com/#q=site:https://www.kaggle.com/jobs+%22data+engineer%22 https://www.google.com/#q=site:https://www.kaggle.com/jobs+%...

mattnewton 10y ago
Thank you guys! Doing Kaggle competitions is what got me interested in seriously pursuing ML in the first place. You are all seriously awesome. I'll look again at the board but, I didn't see anything there before that wanted software engineering skills (which I have with industry experience), and not a graduate degree (which I don't), and happened to be commutable from my place just south of the bay. But I will keep looking!

minimaxir 10y ago

ML falls more under a Data Science role than Data Engineering, although ML is much more difficult without proper Data Engineering.

ironchef 10y ago

I see tons of them. If you're interested in ML, you're probably more looking towards data science. Data engineering (in general) is more about getting the data in a state where it can be used (extracted, cleaned, moved, transformed, etc.) at least from what i've commonly seen in the industry. A decent breakdown is here: https://blog.insightdatascience.com/data-science-vs-data-engineering-62da7678adaa https://blog.insightdatascience.com/data-science-vs-data-eng...

alexbeloi 10y ago

You might want to look at "Machine Learning Engineer" positions if you want to do ML in practice, it's starting to be a title I see somewhat often now. As others have pointed out Data Engineering is more about building data pipelines, making architecture decisions for your ML stack, things like that. Less about model building, prototyping and training, which is what I think of when somebody says they 'do' ML.

mattnewton 10y ago
Right, I'm not picky about the title. I'm looking at those positions too. The main thing is, I want to be able to contribute using my existing software engineering skills from day 1, while picking up the ML stuff. It's been really hard to basically work an unrelated job during the day and go home and do kaggles for practice, so I am hoping to get more of an intersection as a launching place. Anything touching the data or the models will do :)

achompas 10y ago

My official title is "Data Scientist" although I'm closer to the "ML Engineer" someone else mentions in a child comment. Frankly speaking, if your company doesn't need a data engineer, it won't hire one or move you into that role. They likely don't, either, if you're experiencing this pushback -- data engineers often develop ETL pipelines or data warehouses, both of which are very useful if your company has a data team and very useless if it does not. That said, you may want to move closer to my role. There's actually a shortage of data-savvy people who can also write production software, and you would nicely complement a more research-inclined data scientist or analyst -- someone with far more experience with research/analysis than development.

mattnewton 10y ago
I think the company does need data engineers but wants someone with a graduate degree from Stanford or CMU in that position, even though the actual work is in building up infrastructure for those people. And I understand. I've only really got software engineering skills to contribute at this point and I'm picking up the ML from kaggles on the side; I am looking for a position that can increase my overlap between those, because learning at home while working on unrelated stuff is making me move slowly and painfully. Your experience sounds exactly like what I'm looking for - data-savvy writing production code, complementing a research-heavy team I can learn from. How did you get started in that?
- achompas 10y ago
  I honestly fell into it by luck. I moved to NYC, studied machine learning in grad school, networked my ass off, and landed an internship. From there I went full time as something of an ML engineer at a company with a strong tech culture, and learned as much as I could in both tech and ML/statistics. The rest is history (although I'm by no means a rockstar or whatever). My path is hard to reproduce -- it starts with being in NYC or SF at a specific point in time, before the labor market became saturated with data science bootcamps and PhDs furiously learning Python while working on their dissertations. Your best bet at this point is to produce a few data-related projects (maybe work on open source like scikit-learn and pandas?) and network like crazy. Someone somewhere will have a need for someone like you.

bcbrown 10y ago

You should put your email in your profile. If you're in Seattle, send me an email.