11 ms·

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where we start to part ways. If you are really serious about a shortage, you should be really serious about making offers that can be competitive, but I keep seeing the same $150k offers. That isn't a "shortage" kind of offer.

tobyjsullivan 10y ago
"we're in the middle of a talent shortage [and don't believe in upskilling]."
- ThomPete 10y ago
  Upskilling is one of the most ineffective costly ways to try and "re-program" workers and it mostly doesn't work because it's not about skills it's about talent.
  - linkregister 10y ago
    Talent that occurs through the genetic/epigenetic process of having attained a Data Science masters degree after earning a Computer Science degree? I am a believer in inherent talent but Data Engineering is a skill set.
    - ThomPete 10y ago
      Engineering is a talent skill there is a world of difference between teaching someone starting from scratch and then starting someone first having to unlearn what they learned to then learn perhaps a completely new way of thinking. Most of the reskill programs I have heard of failed miserably exactly because the skill isn't enough.
  - duaneb 10y ago
    How would you distinguish talent from experience?
    - ThomPete 10y ago
      I would say that talent needs experience to be valuable.
  - crispyambulance 10y ago
    I sort of agree in that orgs can't simply create massive education programs to re-purpose skill-sets/talent. That might have been possible "back in the day" before project managers were breathing down people's necks, but not today. But the brightside is that talented people will find a way to "upskill" themselves in whatever environment they find themselves in. It is then up to the candidates to sell themselves and for the potential employers to be flexible about considering different backgrounds and nurturing the development of cross-functional skills that are needed for so-called data-engineers. The skills listed in the article are all fairly common but its hard to find enough of these skills within individuals. For example, its not hard to find folks who can do the care and feeding of sql-server databases, or skilled programmers, or analysts who understand the business domain intimately. The problem is getting all of these together in one individual in a "know-enough-to-be-dangerous" level.
    - ThomPete 10y ago
      Yeah if up-skilling means "I used to do front end development now I do back end development" the up-skilling is fairly easy to do. But if you used to work as a plumber and want to up-skill to data analyst (or vice versa) it's not that simple.
  - etjossem 10y ago
    That's not always the case. Talent doesn't exist in a vacuum. Someone with a natural talent for picking up new development skills will still learn data engineering far faster when provided with proper resources and strong internal mentorship. I can see how you might make this observation after observing a poorly conducted training program.
    - ThomPete 10y ago
      The problem is that there aren't that many people with natural talents. They exist but it's very hard to sync them up with where demand is. Also this is not just one poorly conducted training program. Denmark spent billions up-skilling parts of their work force. The results where simply no there. Something like 6 out of every 1000 person or something like that.
snowwrestler 10y ago
This argument comes up all the time on HN, but I don't think it means anything. It seems to me that the ability to fill an opening by offering more salary can't disprove a talent shortage, because it is always possible to do so. Thought experiment: If 100 companies had openings for a skill set that only one person could deliver, all 100 companies could eventually fill their openings by sequentially outbidding each other for the services of that one person. So how would we know if a talent shortage really exists for a certain job? I can think of a couple potential hints: if starting salaries are going up much faster than the national average, or if the unemployment rate for that job is much lower than the national unemployment rate. Either would seem to indicate that, relative to the job market as a whole, there was a greater demand than supply for that particular job.
- trendia 10y ago
  In this case, though, there are way more than 6,600 people in the US that would be able to get do that data engineering job, including: 1. physicists 2. Wall St. quants 3. game programmers 4. PhD statisticians So, the problem is not that there aren't 6,600 people in the US that can do it, it's that the companies can't pay or don't want to pay the $200,000 + that would be required to hire them.
  - blahi 10y ago
    phd statistician can write ETLs and data infrastructure?
    - ronald_raygun 10y ago
      There can be a lot of trickiness in this. I worked on A/B testing framework at one of the big software cos and me and all my team had a masters or phd in math or stats. While 95+% of our job was data infrastructure and ETLs there is another dimension to making it work and be correct from a statistical point of view.
  - Declanomous 10y ago
    This comment will sound a bit self-serving, but it supports your point. I have most of the skills necessary to be a data engineer. My degree in biology, but I nearly got a double major in computer science with a minor in math. (I wanted to work in bioinformatics, but it's nearly impossible to make more than a pittance without a PhD) I didn't pursue the double major because I felt taking classes outside of those three fields was more useful to my development. Instead of working as a data engineer, I'm working at a non-profit doing pretty much everything involving data for them, as well as running their appeals, and doing almost all of the analysis. I'll lead off by saying the biggest downside of working for this particular non-profit is the salary. However, there are a lot of things I like about this job: 1) Location: I want to be located in Chicago. I have 0 interest in moving out to the West Coast. I'm up in the air about working remotely, because I feel like there is a lot of value in working with people in person. 2) The role is very broad. I get to do a lot of exciting things with data, but it is also a marketing and communication role as well. I am included in nearly every strategic discussion, not just those pertaining to data or technology. 3) Work life balance is very good. I am never expected to work more than 40 hours a week. My boss makes sure that everyone is focused on their lives, to the point where he basically kicked me out of the office for a week because I was waffling about taking a vacation. He makes sure that people know they aren't expected to check their email or do work on the VPN during off-hours. 4) The work I do makes a difference. Not in a "I make something people use" difference, but in a "my work has rescued people from being homeless and fed starving kids" difference. My first couple of jobs out of college were totally lacking this aspect, and I didn't realize how much it meant to me until I started working at a place like this. I've been here a few years now, and so it's approaching the time where I should start looking for a new job if I want to continue to grow, but I'm having trouble visualizing what that would be. From my perspective, the problem with hiring is that job listings really focus on titles rather than roles, even in smaller organizations. I think my best bet of finding an organization matches the first two points, if not all four, is through my network rather than through job postings. So, to your point, the only way I see myself in a narrow-title role like a "data engineer" is if I really need money.
achompas 10y ago
Yeah, this is a selling problem. It feels like you're far more likely to gain traction starting a data team than taking an IC-track DE role. It's easier for companies to justify $200k+ for your skillset in that case, even if it takes you away from pure engineering. Alternatively, you can just join a large tech org. Netflix etc. have no problem paying good DEs north of $200k in total comp.
sdoowpilihp 10y ago
This has been my experience with any "senior" engineering / BI / DS role. There is a particularly high level of price sensitivity to anything above 200k.
tuna-piano 10y ago
Of course this is more or less always true - there are only shortages or excesses of things when prices don't or can't adjust freely. If there was 1 gallon of water left on earth, Bill gates would buy that gallon for $50 billion, and everyone else would die of dehydration. There has always been a shortage of maids willing to do all my house work for $10. And there is a shortage of data engineers at $x, but there wouldn't be a shortage at $1M/year (because less companies would want one, and more people would be willing to do the work).
- beachstartup 10y ago
  > If there was 1 gallon of water left on earth, Bill gates would buy that gallon for $50 billion, and everyone else would die of dehydration. really? who would sell the last gallon of water on earth?
  - spoonie 10y ago
    Someone who just drank the last 2nd last gallon. ;)
    - qaq 10y ago
      and what would he use the money for :)?
      - atombath 10y ago
        bullets =)
        beachstartup 10y ago
        all the bullets would have been fired by then.
  - wickedlogic 10y ago
    If they needed water to prime the last pump on earth?
bobosha 10y ago
>Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." That's true for just about anything. "there is no epipen crisis, only a crisis at what you are willing to pay" "There is no poverty , only poverty at a given income level" *"there is no crime problem, only crime problem at a given crime level" what you are saying is self-contradictory. If you (or others) are able to turn down 150K offers...you know what you are.
- wavefunction 10y ago
  You must admit that the price of epipens is an artificially inflated one only possible due to government imposed monopoly, not one driving by true market forces. Poverty is simply a description wealth and is always comparative. We can define poverty as any level we so desire. One might argue that any crime is a problem, as long as it causes an issue for society or victims.
whenwillitstop 10y ago
My experiences exactly, pinged by companies obsessively for my big data skills, all trying to pay me less than I am currently making.
stale2002 10y ago
Although your statement is technically true, it is basically meaningless. Yes, you can always always always find somebody to do a job is your a willing to pay 10 million dollars. That means that "shortages" are impossible. It means that you can never have a shortage in any situation, because you can always pay 10 million dollars for a single visit to the doctor. But this line of logic isn't very useful when talking about "shortages". If you had to pay a million dollars for a loaf of bread, is there a shortage of bread? IE, billions of people will starve to death by next week, because they can't afford to buy food. Most people would say "Yes, there is a shortage of bread". When people talk about shortages, they are obviously talking about a shortage at a certain price point. There is no other definition of the word shortage that makes sense. A good definition that I use for the term shortage is "If the government could snap its fingers and instantly produce large amounts of X overnight, would the world be a better place"? If the answer is "Yes, the world would be in a much much better place", then that means there is a shortage of X. If the answer is "No, the world would only be a little better". Then that means that there is NOT a shortage of X.
- deleted 10y ago
  [deleted]
- sbov 10y ago
  Haven't you just done a 180? I mean, I'm pretty sure the world would be in a much much better place if the government could snap its fingers and instantly produce large amounts of almost anything. Therefore there is a shortage of almost everything.
  - sksnxjis 10y ago
    Given the millions of unemployed Americans, it seems this is not true for at least some occupations. Wal-Mart greeters can be wonderful people and I'm not saying they aren't valuable as humans. But in labor market terms, there is clearly not a shortage of them.
- jnordwick 10y ago
  If they were offering $300k and still couldn't find attract top talent from other industries, then we could have a discussion about a shortage, but the low six figures doesn't show a shortage situation. It shows a market with plenty of headroom for salaries to grow.
SmellTheGlove 10y ago
I think it's definitely true. Functionally, I'm a director of data engineering (with a big company, so my real title is way more generic). Usually in the initial screen, we'll talk general dollars, and my number is always out of range. For my level and the fact that I'm reasonably happy where I live now, the number is 200k + relocation (more for Bay Area, but lets not go there), and I don't think that's unreasonable for a director level who is presumably going to also develop your more junior DEs. I don't fancy it up too much, either. I build teams that make the data move and land it clean so that your PhDs can do the smaaaht stuff with it. I can stack BI and Analytics on top, but a lot of people can do that starting from clean data - and clean data is what I do. But I do get the impression that we're viewed as janitors and plumbers - who you'd be thrilled to see at 3am when your shit(ter) broke, right?
- sloxy 10y ago
  "director of data engineering" This is already generic.
  - SmellTheGlove 10y ago
    Oh boy. My HR title would drive you nuts then. It's just director information management. We also have info analysts and info managers. Our department color is gray.
dizzystar 10y ago
Are they looking for someone who must have every box ticked or are they looking for someone with enough qualifications yet needing work so much they are willing to undercut themselves? Are they justifying their salary offer because you tick 90% of the boxes and not 100%? I've been looking for work in data engineering and databases for 9 months, and while I'm certainly not as qualified and experienced as you are, I consider myself capable. I've definitely passed the take home and whiteboard tests I've been given, etc. When I read about a "shortage," I wonder if this is more indicative of unicorn searching than anything else.
- SmellTheGlove 10y ago
  That to me is a classic recruiting problem in technical positions, data engineering included. Unless you have a manager handling it themselves, the person doing the initial screen really is ticking boxes because they may not know any better. Once a resume gets to me, and I'm only speaking for myself here, I'm looking for the challenges you've faced and the problems you've solved. I actually care very little about what tech you used because odds are we'll have something different, but we'll need to solve problems. If someone is solid in some related technical skillset, can think critically, and communicate the details of what they've tackled in the past, learning our specific tech stack is going to be the easy part. Let me put it another way - when I look for interns or entry level hires, the number of those that can do more than spell SAS or Teradata approaches zero very quickly. But if they've solved challenges of the magnitude that they'd be expected to solve with us initially, the tech is secondary to process and problem solving. As we look more experienced, I'd still be limiting myself to candidates from a set of "legacy" industries that prefer these sorts of tools if I insisted on checking those boxes at the outset. I'd prefer to teach a really smart person to use the things that they don't know yet if I have it my way.
  - walshemj 10y ago
    Quite I am sure my experience doing MR back in the early 80's for British Telecom would be usefull today - but I suspect that I might struggle to get past the hr screen. That was when 17 top of the line supermini's Pr!me 750's was a huge cluster (we where the largest non back user in the UK) - probably about the same as a 10-20k core Hadoop setup would be today.
deleted 10y ago
[deleted]
harichinnan 10y ago
We all like $400K the investment bankers make. But Finance Industry had developed a business where they could pay their workers $400K and still make a huge profit for their investors. Except for Googles and Facebooks, the average tech startup is not making Finance industry level profits. Also Finance requires proper education and training. Not so much for App development. So for everyone who complains about getting $150K offers, there are a 100 thousand people right here in US applying for $60K technical analyst jobs.
- __derek__ 10y ago
  > Except for Googles and Facebooks, the average tech startup is not making Finance industry level profits. And they don't have finance/Google/Facebook level needs for data engineers. They can't reasonably claim to need top-level skills and then beggar out on the cost.
biztos 10y ago
Maybe you should start a Data Science & Engineering consultancy. The same people who would offer $150K to an employee often have bosses who would love to spend $500K for a person-year of (contract) work if it comes with a high probability of success.
- SmellTheGlove 10y ago
  I have thought about that. Many times. I have a couple of barriers, most of which are temporary: 1. I have student debt from my law degree, and I have a lower risk tolerance until that's paid off. 2. My daughter is 4, it's nice to be around for the early years, and the corporate gig is quite comfortable in terms of hours. 3. I'm in Maine. Most clients would require me to travel, which impacts #2. I do have a former colleague here that started a data consultancy. I should grab a beer with him and see if we have common ground in the short term. It's not quite starting your own thing, but it might be fun.
  - edoceo 10y ago
    +1 for making family important!
  - xapata 10y ago
    Running an individual or small consulting company isn't that hard. Scaling a consulting company, on the other hand, is quite difficult.
x0x0 10y ago
In particular, employers whining about lack of X need to ponder raising wages to where employees can afford homes in a city where prices are now within spitting distance of $1k/ft2. When your basic pitch is, "We desperately need [data engineers | machine learning engineers | computer vision engineers | what have you] so desperate to live in CA they'll accept never being able to afford a home unless our lottery tickets pay out", it should be unsurprising they have a hard time finding the talent they claim to need. Or, they could accept remote workers! Even remote workers near sfbay, who just don't want to burn 2.5 hours/day commuting in and out of sf...

tobyjsullivan 10y ago

"we're in the middle of a talent shortage [and don't believe in upskilling]."

ThomPete 10y ago
Upskilling is one of the most ineffective costly ways to try and "re-program" workers and it mostly doesn't work because it's not about skills it's about talent.
- linkregister 10y ago
  Talent that occurs through the genetic/epigenetic process of having attained a Data Science masters degree after earning a Computer Science degree? I am a believer in inherent talent but Data Engineering is a skill set.
  - ThomPete 10y ago
    Engineering is a talent skill there is a world of difference between teaching someone starting from scratch and then starting someone first having to unlearn what they learned to then learn perhaps a completely new way of thinking. Most of the reskill programs I have heard of failed miserably exactly because the skill isn't enough.
- duaneb 10y ago
  How would you distinguish talent from experience?
  - ThomPete 10y ago
    I would say that talent needs experience to be valuable.
- crispyambulance 10y ago
  I sort of agree in that orgs can't simply create massive education programs to re-purpose skill-sets/talent. That might have been possible "back in the day" before project managers were breathing down people's necks, but not today. But the brightside is that talented people will find a way to "upskill" themselves in whatever environment they find themselves in. It is then up to the candidates to sell themselves and for the potential employers to be flexible about considering different backgrounds and nurturing the development of cross-functional skills that are needed for so-called data-engineers. The skills listed in the article are all fairly common but its hard to find enough of these skills within individuals. For example, its not hard to find folks who can do the care and feeding of sql-server databases, or skilled programmers, or analysts who understand the business domain intimately. The problem is getting all of these together in one individual in a "know-enough-to-be-dangerous" level.
  - ThomPete 10y ago
    Yeah if up-skilling means "I used to do front end development now I do back end development" the up-skilling is fairly easy to do. But if you used to work as a plumber and want to up-skill to data analyst (or vice versa) it's not that simple.
- etjossem 10y ago
  That's not always the case. Talent doesn't exist in a vacuum. Someone with a natural talent for picking up new development skills will still learn data engineering far faster when provided with proper resources and strong internal mentorship. I can see how you might make this observation after observing a poorly conducted training program.
  - ThomPete 10y ago
    The problem is that there aren't that many people with natural talents. They exist but it's very hard to sync them up with where demand is. Also this is not just one poorly conducted training program. Denmark spent billions up-skilling parts of their work force. The results where simply no there. Something like 6 out of every 1000 person or something like that.

snowwrestler 10y ago

This argument comes up all the time on HN, but I don't think it means anything. It seems to me that the ability to fill an opening by offering more salary can't disprove a talent shortage, because it is always possible to do so. Thought experiment: If 100 companies had openings for a skill set that only one person could deliver, all 100 companies could eventually fill their openings by sequentially outbidding each other for the services of that one person. So how would we know if a talent shortage really exists for a certain job? I can think of a couple potential hints: if starting salaries are going up much faster than the national average, or if the unemployment rate for that job is much lower than the national unemployment rate. Either would seem to indicate that, relative to the job market as a whole, there was a greater demand than supply for that particular job.

trendia 10y ago
In this case, though, there are way more than 6,600 people in the US that would be able to get do that data engineering job, including: 1. physicists 2. Wall St. quants 3. game programmers 4. PhD statisticians So, the problem is not that there aren't 6,600 people in the US that can do it, it's that the companies can't pay or don't want to pay the $200,000 + that would be required to hire them.
- blahi 10y ago
  phd statistician can write ETLs and data infrastructure?
  - ronald_raygun 10y ago
    There can be a lot of trickiness in this. I worked on A/B testing framework at one of the big software cos and me and all my team had a masters or phd in math or stats. While 95+% of our job was data infrastructure and ETLs there is another dimension to making it work and be correct from a statistical point of view.
- Declanomous 10y ago
  This comment will sound a bit self-serving, but it supports your point. I have most of the skills necessary to be a data engineer. My degree in biology, but I nearly got a double major in computer science with a minor in math. (I wanted to work in bioinformatics, but it's nearly impossible to make more than a pittance without a PhD) I didn't pursue the double major because I felt taking classes outside of those three fields was more useful to my development. Instead of working as a data engineer, I'm working at a non-profit doing pretty much everything involving data for them, as well as running their appeals, and doing almost all of the analysis. I'll lead off by saying the biggest downside of working for this particular non-profit is the salary. However, there are a lot of things I like about this job: 1) Location: I want to be located in Chicago. I have 0 interest in moving out to the West Coast. I'm up in the air about working remotely, because I feel like there is a lot of value in working with people in person. 2) The role is very broad. I get to do a lot of exciting things with data, but it is also a marketing and communication role as well. I am included in nearly every strategic discussion, not just those pertaining to data or technology. 3) Work life balance is very good. I am never expected to work more than 40 hours a week. My boss makes sure that everyone is focused on their lives, to the point where he basically kicked me out of the office for a week because I was waffling about taking a vacation. He makes sure that people know they aren't expected to check their email or do work on the VPN during off-hours. 4) The work I do makes a difference. Not in a "I make something people use" difference, but in a "my work has rescued people from being homeless and fed starving kids" difference. My first couple of jobs out of college were totally lacking this aspect, and I didn't realize how much it meant to me until I started working at a place like this. I've been here a few years now, and so it's approaching the time where I should start looking for a new job if I want to continue to grow, but I'm having trouble visualizing what that would be. From my perspective, the problem with hiring is that job listings really focus on titles rather than roles, even in smaller organizations. I think my best bet of finding an organization matches the first two points, if not all four, is through my network rather than through job postings. So, to your point, the only way I see myself in a narrow-title role like a "data engineer" is if I really need money.

achompas 10y ago

Yeah, this is a selling problem. It feels like you're far more likely to gain traction starting a data team than taking an IC-track DE role. It's easier for companies to justify $200k+ for your skillset in that case, even if it takes you away from pure engineering. Alternatively, you can just join a large tech org. Netflix etc. have no problem paying good DEs north of $200k in total comp.

sdoowpilihp 10y ago

This has been my experience with any "senior" engineering / BI / DS role. There is a particularly high level of price sensitivity to anything above 200k.

tuna-piano 10y ago

Of course this is more or less always true - there are only shortages or excesses of things when prices don't or can't adjust freely. If there was 1 gallon of water left on earth, Bill gates would buy that gallon for $50 billion, and everyone else would die of dehydration. There has always been a shortage of maids willing to do all my house work for $10. And there is a shortage of data engineers at $x, but there wouldn't be a shortage at $1M/year (because less companies would want one, and more people would be willing to do the work).

beachstartup 10y ago
> If there was 1 gallon of water left on earth, Bill gates would buy that gallon for $50 billion, and everyone else would die of dehydration. really? who would sell the last gallon of water on earth?
- spoonie 10y ago
  Someone who just drank the last 2nd last gallon. ;)
  - qaq 10y ago
    and what would he use the money for :)?
    - atombath 10y ago
      bullets =)
      - beachstartup 10y ago
        all the bullets would have been fired by then.
- wickedlogic 10y ago
  If they needed water to prime the last pump on earth?

bobosha 10y ago

>Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." That's true for just about anything. "there is no epipen crisis, only a crisis at what you are willing to pay" "There is no poverty , only poverty at a given income level" *"there is no crime problem, only crime problem at a given crime level" what you are saying is self-contradictory. If you (or others) are able to turn down 150K offers...you know what you are.

wavefunction 10y ago
You must admit that the price of epipens is an artificially inflated one only possible due to government imposed monopoly, not one driving by true market forces. Poverty is simply a description wealth and is always comparative. We can define poverty as any level we so desire. One might argue that any crime is a problem, as long as it causes an issue for society or victims.

whenwillitstop 10y ago

My experiences exactly, pinged by companies obsessively for my big data skills, all trying to pay me less than I am currently making.

stale2002 10y ago

Although your statement is technically true, it is basically meaningless. Yes, you can always always always find somebody to do a job is your a willing to pay 10 million dollars. That means that "shortages" are impossible. It means that you can never have a shortage in any situation, because you can always pay 10 million dollars for a single visit to the doctor. But this line of logic isn't very useful when talking about "shortages". If you had to pay a million dollars for a loaf of bread, is there a shortage of bread? IE, billions of people will starve to death by next week, because they can't afford to buy food. Most people would say "Yes, there is a shortage of bread". When people talk about shortages, they are obviously talking about a shortage at a certain price point. There is no other definition of the word shortage that makes sense. A good definition that I use for the term shortage is "If the government could snap its fingers and instantly produce large amounts of X overnight, would the world be a better place"? If the answer is "Yes, the world would be in a much much better place", then that means there is a shortage of X. If the answer is "No, the world would only be a little better". Then that means that there is NOT a shortage of X.

deleted 10y ago
[deleted]
sbov 10y ago
Haven't you just done a 180? I mean, I'm pretty sure the world would be in a much much better place if the government could snap its fingers and instantly produce large amounts of almost anything. Therefore there is a shortage of almost everything.
- sksnxjis 10y ago
  Given the millions of unemployed Americans, it seems this is not true for at least some occupations. Wal-Mart greeters can be wonderful people and I'm not saying they aren't valuable as humans. But in labor market terms, there is clearly not a shortage of them.
jnordwick 10y ago
If they were offering $300k and still couldn't find attract top talent from other industries, then we could have a discussion about a shortage, but the low six figures doesn't show a shortage situation. It shows a market with plenty of headroom for salaries to grow.

SmellTheGlove 10y ago

I think it's definitely true. Functionally, I'm a director of data engineering (with a big company, so my real title is way more generic). Usually in the initial screen, we'll talk general dollars, and my number is always out of range. For my level and the fact that I'm reasonably happy where I live now, the number is 200k + relocation (more for Bay Area, but lets not go there), and I don't think that's unreasonable for a director level who is presumably going to also develop your more junior DEs. I don't fancy it up too much, either. I build teams that make the data move and land it clean so that your PhDs can do the smaaaht stuff with it. I can stack BI and Analytics on top, but a lot of people can do that starting from clean data - and clean data is what I do. But I do get the impression that we're viewed as janitors and plumbers - who you'd be thrilled to see at 3am when your shit(ter) broke, right?

sloxy 10y ago
"director of data engineering" This is already generic.
- SmellTheGlove 10y ago
  Oh boy. My HR title would drive you nuts then. It's just director information management. We also have info analysts and info managers. Our department color is gray.

dizzystar 10y ago

Are they looking for someone who must have every box ticked or are they looking for someone with enough qualifications yet needing work so much they are willing to undercut themselves? Are they justifying their salary offer because you tick 90% of the boxes and not 100%? I've been looking for work in data engineering and databases for 9 months, and while I'm certainly not as qualified and experienced as you are, I consider myself capable. I've definitely passed the take home and whiteboard tests I've been given, etc. When I read about a "shortage," I wonder if this is more indicative of unicorn searching than anything else.

SmellTheGlove 10y ago
That to me is a classic recruiting problem in technical positions, data engineering included. Unless you have a manager handling it themselves, the person doing the initial screen really is ticking boxes because they may not know any better. Once a resume gets to me, and I'm only speaking for myself here, I'm looking for the challenges you've faced and the problems you've solved. I actually care very little about what tech you used because odds are we'll have something different, but we'll need to solve problems. If someone is solid in some related technical skillset, can think critically, and communicate the details of what they've tackled in the past, learning our specific tech stack is going to be the easy part. Let me put it another way - when I look for interns or entry level hires, the number of those that can do more than spell SAS or Teradata approaches zero very quickly. But if they've solved challenges of the magnitude that they'd be expected to solve with us initially, the tech is secondary to process and problem solving. As we look more experienced, I'd still be limiting myself to candidates from a set of "legacy" industries that prefer these sorts of tools if I insisted on checking those boxes at the outset. I'd prefer to teach a really smart person to use the things that they don't know yet if I have it my way.
- walshemj 10y ago
  Quite I am sure my experience doing MR back in the early 80's for British Telecom would be usefull today - but I suspect that I might struggle to get past the hr screen. That was when 17 top of the line supermini's Pr!me 750's was a huge cluster (we where the largest non back user in the UK) - probably about the same as a 10-20k core Hadoop setup would be today.

deleted 10y ago

[deleted]

harichinnan 10y ago

We all like $400K the investment bankers make. But Finance Industry had developed a business where they could pay their workers $400K and still make a huge profit for their investors. Except for Googles and Facebooks, the average tech startup is not making Finance industry level profits. Also Finance requires proper education and training. Not so much for App development. So for everyone who complains about getting $150K offers, there are a 100 thousand people right here in US applying for $60K technical analyst jobs.

__derek__ 10y ago
> Except for Googles and Facebooks, the average tech startup is not making Finance industry level profits. And they don't have finance/Google/Facebook level needs for data engineers. They can't reasonably claim to need top-level skills and then beggar out on the cost.

biztos 10y ago

Maybe you should start a Data Science & Engineering consultancy. The same people who would offer $150K to an employee often have bosses who would love to spend $500K for a person-year of (contract) work if it comes with a high probability of success.

SmellTheGlove 10y ago
I have thought about that. Many times. I have a couple of barriers, most of which are temporary: 1. I have student debt from my law degree, and I have a lower risk tolerance until that's paid off. 2. My daughter is 4, it's nice to be around for the early years, and the corporate gig is quite comfortable in terms of hours. 3. I'm in Maine. Most clients would require me to travel, which impacts #2. I do have a former colleague here that started a data consultancy. I should grab a beer with him and see if we have common ground in the short term. It's not quite starting your own thing, but it might be fun.
- edoceo 10y ago
  +1 for making family important!
- xapata 10y ago
  Running an individual or small consulting company isn't that hard. Scaling a consulting company, on the other hand, is quite difficult.

x0x0 10y ago

In particular, employers whining about lack of X need to ponder raising wages to where employees can afford homes in a city where prices are now within spitting distance of $1k/ft2. When your basic pitch is, "We desperately need [data engineers | machine learning engineers | computer vision engineers | what have you] so desperate to live in CA they'll accept never being able to afford a home unless our lottery tickets pay out", it should be unsurprising they have a hard time finding the talent they claim to need. Or, they could accept remote workers! Even remote workers near sfbay, who just don't want to burn 2.5 hours/day commuting in and out of sf...