4 ms·

In this case, though, there are way more than 6,600 people in the US that would be able to get do that data engineering job, including: 1. physicists 2. Wall

by trendia 10y ago

In this case, though, there are way more than 6,600 people in the US that would be able to get do that data engineering job, including: 1. physicists 2. Wall St. quants 3. game programmers 4. PhD statisticians So, the problem is not that there aren't 6,600 people in the US that can do it, it's that the companies can't pay or don't want to pay the $200,000 + that would be required to hire them.

blahi 10y ago
phd statistician can write ETLs and data infrastructure?
- ronald_raygun 10y ago
  There can be a lot of trickiness in this. I worked on A/B testing framework at one of the big software cos and me and all my team had a masters or phd in math or stats. While 95+% of our job was data infrastructure and ETLs there is another dimension to making it work and be correct from a statistical point of view.
Declanomous 10y ago
This comment will sound a bit self-serving, but it supports your point. I have most of the skills necessary to be a data engineer. My degree in biology, but I nearly got a double major in computer science with a minor in math. (I wanted to work in bioinformatics, but it's nearly impossible to make more than a pittance without a PhD) I didn't pursue the double major because I felt taking classes outside of those three fields was more useful to my development. Instead of working as a data engineer, I'm working at a non-profit doing pretty much everything involving data for them, as well as running their appeals, and doing almost all of the analysis. I'll lead off by saying the biggest downside of working for this particular non-profit is the salary. However, there are a lot of things I like about this job: 1) Location: I want to be located in Chicago. I have 0 interest in moving out to the West Coast. I'm up in the air about working remotely, because I feel like there is a lot of value in working with people in person. 2) The role is very broad. I get to do a lot of exciting things with data, but it is also a marketing and communication role as well. I am included in nearly every strategic discussion, not just those pertaining to data or technology. 3) Work life balance is very good. I am never expected to work more than 40 hours a week. My boss makes sure that everyone is focused on their lives, to the point where he basically kicked me out of the office for a week because I was waffling about taking a vacation. He makes sure that people know they aren't expected to check their email or do work on the VPN during off-hours. 4) The work I do makes a difference. Not in a "I make something people use" difference, but in a "my work has rescued people from being homeless and fed starving kids" difference. My first couple of jobs out of college were totally lacking this aspect, and I didn't realize how much it meant to me until I started working at a place like this. I've been here a few years now, and so it's approaching the time where I should start looking for a new job if I want to continue to grow, but I'm having trouble visualizing what that would be. From my perspective, the problem with hiring is that job listings really focus on titles rather than roles, even in smaller organizations. I think my best bet of finding an organization matches the first two points, if not all four, is through my network rather than through job postings. So, to your point, the only way I see myself in a narrow-title role like a "data engineer" is if I really need money.
- infinite8s 10y ago
  > (I wanted to work in bioinformatics, but it's nearly impossible to make more than a pittance without a PhD) Just wanted to comment on this part - sadly it's difficult to make more than a pittance even with a PhD.
  - Declanomous 10y ago
    Yeah, totally. The difference is a bioinformaticist with a PhD generally gets to choose what they research, whereas a bioinformaticist without a PhD has to work under someone else's grant. Biology actually has the lowest pay of any major for people working in their field with a four-year degree. You are lucky to make more than minimum wage with a four-year degree, especially if your interest is field ecology or something similar. If you want to talk about a shortage of labor where it would matter, biology as a field is probably hurting way more for talented software engineers than any company that needs a data engineer. There are so many great applications for programming in biology, and unlike other sciences, say physics, researchers don't tend to pick up on any amount of programming skill on their way to their PhD. I've tried getting involved in bioinformatics on the side, but it's really difficult to keep up with the field if you don't have thousands of dollars to drop on journal subscriptions. It's also really hard to get access to the data researchers use in general (in any field), but it is made even harder when dealing with researchers involving people due to concerns about privacy. I don't think a focus on privacy is a bad thing, but a lot of publicly available data is sanitized to the point where your sample size would need to be in the billions to draw any inferences. You can request access to less general data, but good luck doing that without the support of a research organization. Anyways, unless you have a martyr complex, there really isn't any reason to go into bioinformatics.
    - x0x0 10y ago
      I happily worked in a wetlab writing stats software to support breast cancer research. I now do better ad targeting. My salary tripled.
      - deleted 10y ago
huac 10y ago
I'm going to guess most of those people couldn't set up and scale a Hadoop cluster. Are they smart enough that they could learn this stuff? Sure! But there's still a skill mismatch here.
- walshemj 10y ago
  so you find some real full stack devs ie layers 1-7 or you buckle down and do a CCNA or similar is there a CCIE track for big data? And I would bet across all of the tech workers in the USA there are well more than 6k that could do this.
  - huac 10y ago
    So, obviously a LinkedIn search for exact title of "data engineer" isn't exhaustive. And as I understand your point, there's certainly no agreed upon group of skills/certification that qualify someone to be a data engineer (or data scientist, or software engineer, for that matter). But the GP was particularly amusing to me because of its assertion that 'smart, quantitative people, regardless of industry, can build data infrastructures for startups.' I guess we could also say, there's little incentive to pay to train them (or for them to pay to train) to become a data engineer.
    - walshemj 10y ago
      Ah the British Disease ie don't want to pay for training and we don't want those uppity engineers getting above themselves :-(
ISL 10y ago
Or, offer the comparably-compensated part-time job that an academic physicist would accept in parallel with continuing to work in academia. Source: Am physicist who'd love to find sustainable part-time work at market rates.
- __derek__ 10y ago
  This is on-point. For comparison's sake, look at the number of economists, political scientists, and business professors who have side gigs in consulting.

by trendia 10y ago

blahi 10y ago

phd statistician can write ETLs and data infrastructure?

ronald_raygun 10y ago
There can be a lot of trickiness in this. I worked on A/B testing framework at one of the big software cos and me and all my team had a masters or phd in math or stats. While 95+% of our job was data infrastructure and ETLs there is another dimension to making it work and be correct from a statistical point of view.

Declanomous 10y ago

This comment will sound a bit self-serving, but it supports your point. I have most of the skills necessary to be a data engineer. My degree in biology, but I nearly got a double major in computer science with a minor in math. (I wanted to work in bioinformatics, but it's nearly impossible to make more than a pittance without a PhD) I didn't pursue the double major because I felt taking classes outside of those three fields was more useful to my development. Instead of working as a data engineer, I'm working at a non-profit doing pretty much everything involving data for them, as well as running their appeals, and doing almost all of the analysis. I'll lead off by saying the biggest downside of working for this particular non-profit is the salary. However, there are a lot of things I like about this job: 1) Location: I want to be located in Chicago. I have 0 interest in moving out to the West Coast. I'm up in the air about working remotely, because I feel like there is a lot of value in working with people in person. 2) The role is very broad. I get to do a lot of exciting things with data, but it is also a marketing and communication role as well. I am included in nearly every strategic discussion, not just those pertaining to data or technology. 3) Work life balance is very good. I am never expected to work more than 40 hours a week. My boss makes sure that everyone is focused on their lives, to the point where he basically kicked me out of the office for a week because I was waffling about taking a vacation. He makes sure that people know they aren't expected to check their email or do work on the VPN during off-hours. 4) The work I do makes a difference. Not in a "I make something people use" difference, but in a "my work has rescued people from being homeless and fed starving kids" difference. My first couple of jobs out of college were totally lacking this aspect, and I didn't realize how much it meant to me until I started working at a place like this. I've been here a few years now, and so it's approaching the time where I should start looking for a new job if I want to continue to grow, but I'm having trouble visualizing what that would be. From my perspective, the problem with hiring is that job listings really focus on titles rather than roles, even in smaller organizations. I think my best bet of finding an organization matches the first two points, if not all four, is through my network rather than through job postings. So, to your point, the only way I see myself in a narrow-title role like a "data engineer" is if I really need money.

infinite8s 10y ago
> (I wanted to work in bioinformatics, but it's nearly impossible to make more than a pittance without a PhD) Just wanted to comment on this part - sadly it's difficult to make more than a pittance even with a PhD.
- Declanomous 10y ago
  Yeah, totally. The difference is a bioinformaticist with a PhD generally gets to choose what they research, whereas a bioinformaticist without a PhD has to work under someone else's grant. Biology actually has the lowest pay of any major for people working in their field with a four-year degree. You are lucky to make more than minimum wage with a four-year degree, especially if your interest is field ecology or something similar. If you want to talk about a shortage of labor where it would matter, biology as a field is probably hurting way more for talented software engineers than any company that needs a data engineer. There are so many great applications for programming in biology, and unlike other sciences, say physics, researchers don't tend to pick up on any amount of programming skill on their way to their PhD. I've tried getting involved in bioinformatics on the side, but it's really difficult to keep up with the field if you don't have thousands of dollars to drop on journal subscriptions. It's also really hard to get access to the data researchers use in general (in any field), but it is made even harder when dealing with researchers involving people due to concerns about privacy. I don't think a focus on privacy is a bad thing, but a lot of publicly available data is sanitized to the point where your sample size would need to be in the billions to draw any inferences. You can request access to less general data, but good luck doing that without the support of a research organization. Anyways, unless you have a martyr complex, there really isn't any reason to go into bioinformatics.
  - x0x0 10y ago
    I happily worked in a wetlab writing stats software to support breast cancer research. I now do better ad targeting. My salary tripled.
    - deleted 10y ago

huac 10y ago

I'm going to guess most of those people couldn't set up and scale a Hadoop cluster. Are they smart enough that they could learn this stuff? Sure! But there's still a skill mismatch here.

walshemj 10y ago
so you find some real full stack devs ie layers 1-7 or you buckle down and do a CCNA or similar is there a CCIE track for big data? And I would bet across all of the tech workers in the USA there are well more than 6k that could do this.
- huac 10y ago
  So, obviously a LinkedIn search for exact title of "data engineer" isn't exhaustive. And as I understand your point, there's certainly no agreed upon group of skills/certification that qualify someone to be a data engineer (or data scientist, or software engineer, for that matter). But the GP was particularly amusing to me because of its assertion that 'smart, quantitative people, regardless of industry, can build data infrastructures for startups.' I guess we could also say, there's little incentive to pay to train them (or for them to pay to train) to become a data engineer.
  - walshemj 10y ago
    Ah the British Disease ie don't want to pay for training and we don't want those uppity engineers getting above themselves :-(

ISL 10y ago

Or, offer the comparably-compensated part-time job that an academic physicist would accept in parallel with continuing to work in academia. Source: Am physicist who'd love to find sustainable part-time work at market rates.

__derek__ 10y ago
This is on-point. For comparison's sake, look at the number of economists, political scientists, and business professors who have side gigs in consulting.