5 ms·
This argument comes up all the time on HN, but I don't think it means anything. It seems to me that the ability to fill an opening by offering more salary can't
by snowwrestler 10y ago
This argument comes up all the time on HN, but I don't think it means anything. It seems to me that the ability to fill an opening by offering more salary can't disprove a talent shortage, because it is always possible to do so.
Thought experiment: If 100 companies had openings for a skill set that only one person could deliver, all 100 companies could eventually fill their openings by sequentially outbidding each other for the services of that one person.
So how would we know if a talent shortage really exists for a certain job? I can think of a couple potential hints: if starting salaries are going up much faster than the national average, or if the unemployment rate for that job is much lower than the national unemployment rate. Either would seem to indicate that, relative to the job market as a whole, there was a greater demand than supply for that particular job.
- trendia 10y agoIn this case, though, there are way more than 6,600 people in the US that would be able to get do that data engineering job, including: 1. physicists 2. Wall St. quants 3. game programmers 4. PhD statisticians So, the problem is not that there aren't 6,600 people in the US that can do it, it's that the companies can't pay or don't want to pay the $200,000 + that would be required to hire them.
- blahi 10y agophd statistician can write ETLs and data infrastructure?
- ronald_raygun 10y agoThere can be a lot of trickiness in this. I worked on A/B testing framework at one of the big software cos and me and all my team had a masters or phd in math or stats. While 95+% of our job was data infrastructure and ETLs there is another dimension to making it work and be correct from a statistical point of view.
- Declanomous 10y agoThis comment will sound a bit self-serving, but it supports your point. I have most of the skills necessary to be a data engineer. My degree in biology, but I nearly got a double major in computer science with a minor in math. (I wanted to work in bioinformatics, but it's nearly impossible to make more than a pittance without a PhD) I didn't pursue the double major because I felt taking classes outside of those three fields was more useful to my development. Instead of working as a data engineer, I'm working at a non-profit doing pretty much everything involving data for them, as well as running their appeals, and doing almost all of the analysis. I'll lead off by saying the biggest downside of working for this particular non-profit is the salary. However, there are a lot of things I like about this job: 1) Location: I want to be located in Chicago. I have 0 interest in moving out to the West Coast. I'm up in the air about working remotely, because I feel like there is a lot of value in working with people in person. 2) The role is very broad. I get to do a lot of exciting things with data, but it is also a marketing and communication role as well. I am included in nearly every strategic discussion, not just those pertaining to data or technology. 3) Work life balance is very good. I am never expected to work more than 40 hours a week. My boss makes sure that everyone is focused on their lives, to the point where he basically kicked me out of the office for a week because I was waffling about taking a vacation. He makes sure that people know they aren't expected to check their email or do work on the VPN during off-hours. 4) The work I do makes a difference. Not in a "I make something people use" difference, but in a "my work has rescued people from being homeless and fed starving kids" difference. My first couple of jobs out of college were totally lacking this aspect, and I didn't realize how much it meant to me until I started working at a place like this. I've been here a few years now, and so it's approaching the time where I should start looking for a new job if I want to continue to grow, but I'm having trouble visualizing what that would be. From my perspective, the problem with hiring is that job listings really focus on titles rather than roles, even in smaller organizations. I think my best bet of finding an organization matches the first two points, if not all four, is through my network rather than through job postings. So, to your point, the only way I see myself in a narrow-title role like a "data engineer" is if I really need money.
- infinite8s 10y ago> (I wanted to work in bioinformatics, but it's nearly impossible to make more than a pittance without a PhD) Just wanted to comment on this part - sadly it's difficult to make more than a pittance even with a PhD.
- huac 10y agoI'm going to guess most of those people couldn't set up and scale a Hadoop cluster. Are they smart enough that they could learn this stuff? Sure! But there's still a skill mismatch here.
- walshemj 10y agoso you find some real full stack devs ie layers 1-7 or you buckle down and do a CCNA or similar is there a CCIE track for big data? And I would bet across all of the tech workers in the USA there are well more than 6k that could do this.
- huac 10y agoSo, obviously a LinkedIn search for exact title of "data engineer" isn't exhaustive. And as I understand your point, there's certainly no agreed upon group of skills/certification that qualify someone to be a data engineer (or data scientist, or software engineer, for that matter). But the GP was particularly amusing to me because of its assertion that 'smart, quantitative people, regardless of industry, can build data infrastructures for startups.' I guess we could also say, there's little incentive to pay to train them (or for them to pay to train) to become a data engineer.
- walshemj 10y agoAh the British Disease ie don't want to pay for training and we don't want those uppity engineers getting above themselves :-(
- ISL 10y agoOr, offer the comparably-compensated part-time job that an academic physicist would accept in parallel with continuing to work in academia. Source: Am physicist who'd love to find sustainable part-time work at market rates.
- __derek__ 10y agoThis is on-point. For comparison's sake, look at the number of economists, political scientists, and business professors who have side gigs in consulting.
- deleted 10y ago[deleted]
- snovv_crash 10y agoBullshit. I have the skills for an intermediate-level data engineer, but I find it bland and I'd rather work in computer vision. However, offer me enough and I may reconsider, and I don't think I'm alone in this.
- Declanomous 10y agoI basically wrote the same thing as a reply to your sibling comment. Data engineering would have to pay a lot more than I currently make for it to be an option, and even then I'd probably change fields once I paid off my student loans and saved some money.
- snowwrestler 10y agoYou're not alone, but you're just reiterating that it's always possible to fill an opening by running the salary up high enough.
- snovv_crash 10y agoBut the pool of people who wouldn't otherwise take the job grows as the salary increases, pulling the people away from careers where they clearly aren't providing as much value to their employers. Put it this way, the company isn't going to pay the employee more than the value they provide. That is the ceiling on salary. So until that ceiling is reached it is indeed a case of higher bidder takes all, as your thought experiment demonstrated. But once that ceiling is neared the company will make the decision not to bid higher, thus reducing the demand. Thus, there is no shortage, just a shortage to work at the lower salary of companies with lower ceilings, because they aren't capable of leveraging the employee's talents sufficiently to draw from fields with related skill sets.
- JamesBarney 10y agoShortage has a specific term in economics, which pretty much only happens because of price controls. However if people start liking kale, and the price goes up 20% and you start telling people about the massive kale shortage people will think you're being a little histrionic.