9 ms·

We’re in the Middle of a Data Engineering Talent Shortage

143 points by hankmh 10y ago

mattnewton 10y ago
I'm trying to switch careers into "Data Engineering" now, as a full stack developer who is more interested in ML, and I've found almost no traction internally at my company or externally. It looks like I may just accept a full stack position at a good company that does a lot of data science for now, but though I would ask - Where are all these jobs?
- PaulHoule 10y ago
  "Data Engineering" is most of the work that needs to be done, but I think companies haven't identified it as a category. From my P.O.V., "Full Stack Engineer" is a place you don't want to be because it means putting out fires with whatever junk javascript is in the front end. It seems like everybody who's built a serious javascript application has invented their own Virtual DOM because none of the popular Virtual DOM libraries are good for much other than wasting time and CPU cycles. "Data Scientist" is a bad title in it's own way, in the sense that "Computer Science" is bad, but worse. To a lot of people there is a Brahmin kind of attitude associated with "Scientist" -- i.e. an aversion to getting your hands dirty. Real world data is pretty dirty and you aren't going to get far in getting value out of it unless you spend 80-90% of your time dealing with the dirt.
  - pjmlp 10y ago
    There are "Full Stack Engineer" doing pure native applications, which is what I have been doing the last three years after escaping the web back into native land.
  - mattnewton 10y ago
    You are correct. I thought full stack meant before building the app start to finish, but the reality is often closer to putting out other people's fires in every layer. It does pay well though and you learn a lot of what can go wrong.
    - PaulHoule 10y ago
      The fact that it pays well makes it a job you're likely to get laid off from. Most managers would rather hire two junior developers so they can screw it up faster or better yet hire some people in another country who are really fast and cheap at screwing it up.
      - mattnewton 10y ago
        That may be true but I'm not worried about that, I worry more about getting comfortable doing useless work. If I got fired it would be so much easier to go back to school, as the dream of lots of money while learning on the side would evaporate.
- willis77 10y ago
  We (Kaggle) run a data science jobs board (https://www.kaggle.com/jobs https://www.kaggle.com/jobs) that gets a few data engineer listings from time to time. Not all of these are active, but you may find a few interested companies via - https://www.google.com/#q=site:https://www.kaggle.com/jobs+%22data+engineer%22 https://www.google.com/#q=site:https://www.kaggle.com/jobs+%...
  - mattnewton 10y ago
    Thank you guys! Doing Kaggle competitions is what got me interested in seriously pursuing ML in the first place. You are all seriously awesome. I'll look again at the board but, I didn't see anything there before that wanted software engineering skills (which I have with industry experience), and not a graduate degree (which I don't), and happened to be commutable from my place just south of the bay. But I will keep looking!
- minimaxir 10y ago
  ML falls more under a Data Science role than Data Engineering, although ML is much more difficult without proper Data Engineering.
- ironchef 10y ago
  I see tons of them. If you're interested in ML, you're probably more looking towards data science. Data engineering (in general) is more about getting the data in a state where it can be used (extracted, cleaned, moved, transformed, etc.) at least from what i've commonly seen in the industry. A decent breakdown is here: https://blog.insightdatascience.com/data-science-vs-data-engineering-62da7678adaa https://blog.insightdatascience.com/data-science-vs-data-eng...
- alexbeloi 10y ago
  You might want to look at "Machine Learning Engineer" positions if you want to do ML in practice, it's starting to be a title I see somewhat often now. As others have pointed out Data Engineering is more about building data pipelines, making architecture decisions for your ML stack, things like that. Less about model building, prototyping and training, which is what I think of when somebody says they 'do' ML.
  - mattnewton 10y ago
    Right, I'm not picky about the title. I'm looking at those positions too. The main thing is, I want to be able to contribute using my existing software engineering skills from day 1, while picking up the ML stuff. It's been really hard to basically work an unrelated job during the day and go home and do kaggles for practice, so I am hoping to get more of an intersection as a launching place. Anything touching the data or the models will do :)
- achompas 10y ago
  My official title is "Data Scientist" although I'm closer to the "ML Engineer" someone else mentions in a child comment. Frankly speaking, if your company doesn't need a data engineer, it won't hire one or move you into that role. They likely don't, either, if you're experiencing this pushback -- data engineers often develop ETL pipelines or data warehouses, both of which are very useful if your company has a data team and very useless if it does not. That said, you may want to move closer to my role. There's actually a shortage of data-savvy people who can also write production software, and you would nicely complement a more research-inclined data scientist or analyst -- someone with far more experience with research/analysis than development.
  - mattnewton 10y ago
    I think the company does need data engineers but wants someone with a graduate degree from Stanford or CMU in that position, even though the actual work is in building up infrastructure for those people. And I understand. I've only really got software engineering skills to contribute at this point and I'm picking up the ML from kaggles on the side; I am looking for a position that can increase my overlap between those, because learning at home while working on unrelated stuff is making me move slowly and painfully. Your experience sounds exactly like what I'm looking for - data-savvy writing production code, complementing a research-heavy team I can learn from. How did you get started in that?
    - achompas 10y ago
      I honestly fell into it by luck. I moved to NYC, studied machine learning in grad school, networked my ass off, and landed an internship. From there I went full time as something of an ML engineer at a company with a strong tech culture, and learned as much as I could in both tech and ML/statistics. The rest is history (although I'm by no means a rockstar or whatever). My path is hard to reproduce -- it starts with being in NYC or SF at a specific point in time, before the labor market became saturated with data science bootcamps and PhDs furiously learning Python while working on their dissertations. Your best bet at this point is to produce a few data-related projects (maybe work on open source like scikit-learn and pandas?) and network like crazy. Someone somewhere will have a need for someone like you.
- bcbrown 10y ago
  You should put your email in your profile. If you're in Seattle, send me an email.
skynetv2 10y ago
anything and everything is marketed as "data science" and "data engineering" these days becasue this is the buzzword of the day. I've been dealing with large data even before "big data" was a word but i dont call myself "data scientist" or "data engineer". I am still a software engineer working on what benefits my organization. "Serial Entrepreneur" is the same these days, claimed by anyone who had a lemonade stand as a kid.
- Swizec 10y ago
  > I am still a software engineer working on what benefits my organization But if you saw a nearby local maximum that's higher than your current local maximum, wouldn't you change what you call yourself, if it means being paid more but doing the same work? This is similar to how the average "software engineer" makes about $30k/year more than the average "programmer".
untilHellbanned 10y ago
Ahh the ol' write a post about a not well understood distinction and then proceed to not explain the distinction. Looks like we need more English engineers too.
realworldview 10y ago
We surely need data mechanics.
jnordwick 10y ago
Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where we start to part ways. If you are really serious about a shortage, you should be really serious about making offers that can be competitive, but I keep seeing the same $150k offers. That isn't a "shortage" kind of offer.
- tobyjsullivan 10y ago
  "we're in the middle of a talent shortage [and don't believe in upskilling]."
  - ThomPete 10y ago
    Upskilling is one of the most ineffective costly ways to try and "re-program" workers and it mostly doesn't work because it's not about skills it's about talent.
    - linkregister 10y ago
      Talent that occurs through the genetic/epigenetic process of having attained a Data Science masters degree after earning a Computer Science degree? I am a believer in inherent talent but Data Engineering is a skill set.
      - ThomPete 10y ago
        Engineering is a talent skill there is a world of difference between teaching someone starting from scratch and then starting someone first having to unlearn what they learned to then learn perhaps a completely new way of thinking. Most of the reskill programs I have heard of failed miserably exactly because the skill isn't enough.
    - 10y ago
rch 10y ago
I've heard more than one CTO/Sr. Engineer refer to people in these roles as 'data grunts' or something similarly dismissive. Then they're mystified as to why solid engineers are so quick to move up or out, year after year.
GeneralMayhem 10y ago
There are 6600 jobs listed and 6500 individuals on LinkedIn with that particular title, and therefore there's a shortage? Seriously? * How many aren't on LinkedIn? * Since the whole article is about how the job title is poorly defined and growing in prevalence, why would you assume that people who don't already have such a job would use the term? * The "growth" charts on the full study are just as bad - how much of that is just from renaming existing generic developer positions, since "data engineer" is clearly a relatively new term?
- sportanova 10y ago
  6500 data engineers on all of Linkedin, but 6600 job openings in the bay area. so there are more job openings in one area than all data engineers on linkedin
dmatthewson 10y ago
From the article: "Data engineers are the janitors who keep your data clean and flowing." Hm, I wonder why he's having problems hiring janitors.
- kafkaesq 10y ago
  Data engineers are the janitors who keep your data clean and flowing. In a boldface font, no less. The cockiness behind that language is really quite astounding.
  - praccu 10y ago
    It's really true, though. It's brutal, ugly work with no hope of an end. Edit: Favorite paper on the topic: http://research.google.com/pubs/pub43146.html http://research.google.com/pubs/pub43146.html
    - kafkaesq 10y ago
      So is the work that doctors, lawyers, and other highly-skilled people do, by and large. Everyone knows that day-to-day aspects of these jobs are hardly glamorous (or even cerebral), the vast majority of the time. Yet somehow we manage to accord these people with their due degree of respect, and wouldn't think of referring to them as "janitors".
      - wavefunction 10y ago
        I don't see what's so bad about janitors, though. They do very thankless jobs for not much money, whereas doctors and lawyers and other high-skilled individuals are often well renumerated or offered certain social prestige that your post shows is quite lacking when a humble janitor is considered.
        kafkaesq 10y ago
        A perfectly valid point. But getting back to the original article -- it pretty much takes a SV alpha-nerd (or aspiring CEO seeking to cater to them) to come up with language like that.
otto_ortega 10y ago
Am I the only one who thinks there will be a ton of people changing their job title on LinkedIn to "Data Engineer" as a result of this article?
- collyw 10y ago
  I am thinking about it. Actually a friend recommended that I change my title to Data Engineer a few months back.
mrharrison 10y ago
We should rename this job position to Data Sanity Engineers. I have been thrown these projects at work before, where I'm the frontend engineer and I need to make some cool D3 visualization, but low behold the data is shit, and I have to help the backend team make the data useable. It's a mind-numbing job, that nobody wants, because it sounds like a one month task to get a good REST API up and working, but it usually takes three months, because you have to go back and forth making sure the data is right, and there is always 10 tricky edge cases that you have to work some magic on. Not only that but you need to have smart people cleaning the data, so that you don't make some big mistake down the line or your REST API is super slow, and you have to add another couple weeks or month to rework the data again. So that one month becomes three months, and most likely a year, because somebody will say that looks great but can we also add this, and it goes on and on. It's literally a mind-numbing job that most nobody wants. I have found that products like Tableau are the best for this, you still have to clean the data, but it helps speed up the process. Data cleaning is a super golden problem to solve.
- msie 10y ago
  Data Sanitation Engineers
- kafkaesq 10y ago
  Not only that but you need to have smart people cleaning the data, Which are difficult to find when you think of them as "janitors", and treat them accordingly.
- dizzystar 10y ago
  As a contradiction to this point, some people (me) really enjoy working with data, from cleaning, munging, creating, sorting, pipelining, etc, and find front-end visualization production excessively boring and mind-numbing. Give me emacs and a command line, and I have all the truth I need, which is far more honest, in my mind, than anything that can be created with D3 or Tableau. Beauty is in the eye of the beholder, and it doesn't really do anyone service to look down on the work others find enjoyable. If doing D3 makes you happy, that is awesome, and I can only congratulate you for your passion and your ability to look forward to work I don't "get," and I wish the feelings would be mutual.
ThePhysicist 10y ago
Data engineering sounds much better than "data plumbing", but in my experience the latter is a more accurate description of the work of a data engineer: Building -and often unclogging- pipes that transport data from A to B, and putting in filters to clean it and extract the useful bits. So why not change your LinkedIn job title to "data plumber", which is sure to get you some serious recruiter attention ;)
moandcompany 10y ago
I am a data engineer working on a machine learning team with models actively used as part of our product(s). From my experiences working in various contexts (applied machine learning, analytics, policy research, academics, etc...), there are several of factors that contribute to this shortage: (1) "data engineering" often requires a lot of breadth and knowledge, (2) "data engineering" is often (derisively and naively) referred to as the "janitorial work" of data science, (3) the spectrum of roles and requirements within the "data engineering" domain, in terms of job descriptions, can range from database systems administration, to ETL, to data warehousing, curation of data services / APIs, business intelligence, to the design/deployment/operation of pipelines and distributed data processing and storage systems (these aren't mutually exclusive, but often job descriptions fall into one of these stovepipes). Some of my quick thoughts and anecdata: Companies have made large investments in creating 'data science' teams, and many of those companies have trouble realizing value from those investments. A part of this stems from investments and teams with no tangible vision of how that team will generate value. And there are several other contributing factors… "Dirty work." People haven't learned how to, and more often don't want to do it. There's a vast number of tutorials and boot camps out there that teach newcomers how to "learn data science" with clean datasets -- this is ideal for learning those basics, but the real world usually does not have clean or ideal datasets -- the dataset may not even exist -- and there are a number of non-ideal constraints. There are people that wish to call themselves “data scientists” that “don’t want to write code” and would “prefer to do the analysis and storytelling” Engineering as the application of science with real world constraints: there are a number of factors that we take into account, often acquired through painful experience, that aren’t part of these tutorials, bootcamps, or academic environments. Many “data scientists” I’ve met have a hard time adapting to and working with these constraints (e.g. we believe that the application of data science would solve/address __ problem, but: how do we know and show that it works and is useful? what are the dependencies, and costs of developing and applying that solution? is it a one-time solution, or is it going to be a recurring application? does the solution require people? who will use it? what are the assumptions or expectations of those operators and users? is it suitable? is it maintainable? is it sustainable? how long will it take? what are the risks involved and how do we manage them? is it re-usable, and can we amortize its costs over time? is it worth doing? This is part of a methodology that comes from experience, versus what is taught in data science) Larger teams with more people/financial/political resources can specialize and take advantage of these divisions of labor, which helps recognize the process aspects of applying data science and address some of the above Short story: if you view data engineering as "janitorial work" you're missing the big picture Anyone else notice that the attributes of a 'unicorn' data scientist include the traits of a 'data engineer?'
cutler 10y ago
I'm puzzled at the omission of Scala and Spark in this report.
tom_b 10y ago
Ignoring the breathless nature of the article, this is a buzzword label for a commodity skill set that pays a commodity salary in tech. It is also the commodity skill set that my employers have all paid me for. There has been for a long time hype around new technology and labels for business intelligence, data warehousing, big data, and now data engineering/science. I'm not saying there are not some roles in this space that return huge value to organizations, but that these opportunities are much rarer than the buzz indicates. I wonder if the perceived shortage is mainly hype as the shift to new cloud technologies makes many of the older ideas a little less useful - if you are plowing data into BigQuery, you probably aren't so worried about your star schema data model for reporting. I would strongly advise people that look at these types of articles to look at the roles in question and ask "Is this role on the critical path to customers paying us?" My experience has been that the answer is often "No." This is bad. I have also seen situations where businesses that do rely on smart data integration can show that they are selling dollar bills for ten cents that still have trouble getting customers on board with spending that ten cents. Business is weird.
makmanalp 10y ago
Quick sidenote, anyone know where the databases / distributed systems engineering jobs are at? E.g. if one wanted to not use these tools but also go help build these tools? I can think of Facebook, Google, Microsoft, IBM (which locations and groups within these companies / where?). I can also think of Confluent, CitusDB, Databricks, etc.
- rhizome 10y ago
  Market Research is a $40B industry that depends almost completely on these concepts. I'm not sure how prevalent distributed systems are with MR companies, but that's an implementation detail anyway.
  - serge2k 10y ago
    > that's an implementation detail anyway. Which is what the poster was asking for.
protomyth 10y ago
I worked for about 10 years doing exactly what they want, but I ended up having to write a lot of the tools which means I'm not able to check the boxes on some tool you require which gets me punted by HR. I'm starting to think that the message is if HR is going to do checklists then developers should really make sure they work mostly with contracts that use popular checklist items.
- mulmen 10y ago
  As a data person I would really like to put some numbers on how much the typical HR hiring process costs a business. I don't know anybody that says they are happy with how hiring works in he tech industry but I've also never seen an HR person try and improve the process.
  - pyb 10y ago
    That's because the system is already optimised for the needs of HR people.
cheriot 10y ago
It was only 20 years ago that companies hired a "web master" or a generalist to do everything. But pieces of those jobs became specialized. Now we need UX, UI programmer, general engineers, dev ops, data engineers, a data scientist, etc. And how many companies are still interviewing with fizzbuzz?
collyw 10y ago
So I know SQL, Python, Django, Java (though its been a while), Javascrit, Linux, some cloud computing and a bit of devops. Am I a data engineer? Software engineer, with a lot of database background? What makes a data engineer different from a software engineer?
- njd 10y ago
  - The challenge for an organization is to recognize that there is a significant difference between the 'data engineer' working on a vertical project and the 'data engineer' responsible for integrating data across the enterprise. - The project 'data engineer', in today's world, most likely will be a software developer responsible for ETL, etc. The data design will be more or less up to the software developer. - An enterprise 'data engineer' is more concerned with data that affects the enterprise. This typically involves some sort of data integration. For example, how to integrate relevant data from N projects (e.g. A,B,C .. Z) where each project has its own idea of how to represent similar concepts (e.g. person, user, customer), with different provenance, truth assertions, access rules, data retention periods, granularity of metadata (e.g. at the attribute level vs entity level), etc. The enterprise is interested in questions like 'What did we know and when did we know it?", etc. The enterprise 'data engineer' will probably levy requirements on the project 'data engineer' to meet the enterprise's needs.
jboggan 10y ago
It's digital Charlie Work [0], that's why. I really enjoy that kind of work but it is difficult to articulate your business value in that environment. The best thing is working closely with a data scientist/front-end dev who can deliver products to the analysts and executives that need the data and make sure that you get the credit for enabling new streams of data. But most of the time you are putting out someone else's dumpster fire. One advantage of data engineering: unlike front-end work, there are few non-technical people who will have an opinion on how you are doing things and burden you with bikeshedding. [0] - http://www.avclub.com/tvclub/its-always-sunny-philadelphia-charlie-work-214628 http://www.avclub.com/tvclub/its-always-sunny-philadelphia-c...
binalpatel 10y ago
The fact that the original, unmodified article referred to data engineers as "janitors" pretty much says it all. It's very analogous to front-office and back-office work in Investment Banking. "Data Scientist" are the front-office, with all the prestige, and "Data Engineers" are the back-office, doing a lot of the heavy lifting without nearly as much recognition. In my opinion there shouldn't be a delineation. You shouldn't be a data scientist if you can't gather, process, and clean up your own data.
- biztos 10y ago
  Ideally you'd have a symbiosis, and each side would recognize the importance of the other. Even if you require your data scientists to be able to do engineering work, it's probably way more efficient to have some good generalist Software Engineers doing all the "pre-math" work and freeing your statisticians up for what they're (hopefully) good at. Plus as a side effect, your software will probably be better.
ef5a0b0628 10y ago
Every time something comes up on HN about a talent shortage in a field related to software engineering, it hurts. I have been unsuccessfully looking for a full time position since my last start up (I was not a founder) folded six months ago. I have been on over 25 in person interviews and gone through untold degrading whiteboard interviews, code tests, trick questions, and take home projects; all have ended in rejection. This industry has a need to torture candidates because we are all considered to be liars by default. Much is said about combating impostor syndrome in ourselves but we are too eager to engender it in others. It seems people in this industry refuse to understand that some people are not perfect. I never graduated college because I hated it with the very fiber of my being, so I am not particularly great at white boarding answers to algorithm questions off the top of my head in a high pressure environment. If I need them during my job, I look up answers and learn from people who are much smarter than I am. My personal identity has been shattered, as I thought my ~5-10 year history of success in the industry indicated I was in demand and talented. I saw posts like this and thought that if the worst happened I'd still be able to find a job. The idea that there is a talent shortage is a lie, or candidates like me wouldn't be treated as I have been. I'm not asking for a free job, or a handout. I have had a successful career so far and am capable of doing good work. But I'm not a specialist in Big Data Machine Learning Neural Networks. I have struggled with bipolar disorder and suicidal ideation most of my life. I've dealt with the death of my beloved grandmother and my father who was instrumental in my choosing to be an engineer with only minor lapses in control. Nothing has caused me to consider taking my own life as much as the past 6 months. It seems there is no future for me in the only career I have any skill in and which is a huge part of my identity. And to constantly be told that there is such a shortage of engineers only salts the wound.
- ultramagas 10y ago
  Hey, I'm going through something similar. I had to quit an amazing job because my wife and I pursued a dream and moved to Europe (no remote). I had always had an easy time getting a job before but this time it was different. Granted I knew it'd be tougher since for remote jobs, the world is the competition. But it was a summer of endless shitty timed hackerrank-style tests (virtual whiteboard hazing). I would tell my co-workers about them and they'd laugh in bewilderment at the questions that were asked in what should be a technical screener, and these are extremely smart and productive software guys that have started companies, written books, give conference talks. One funny question I got for a frontend React job: write a function that takes a sequence of bits that represent a negative-binary number (not a base-2 number that is negative, but a base-(-2) number) and return its negated value in base-2. For a frontend job. It was one of 4 questions to be answered in 90 minutes. gtfo. A few companies would reply, most strung me along while -- I realize now -- they were keeping me as a backup(-backup) incase their "A-player" turned them down. Countless interviews, hours on takehome projects, it was tough. I learned to cut bait if the company was slow to move forward, had weeklong periods of no communication, etc. I (just very recently) found it's easier to land small contract gigs because the barrier to entry seems to be lower, demonstrate value, and keep getting work from those guys after the initial project was done. It is different but so far I actually like the freedom that comes with contracting. I haven't been at it long enough to experience the downsides. There's definitely not a shortage of talent. It's that every company thinks they need "A-players", when the vast, vast majority are doing a damn basic CRUD app. Just wanted to say I hear you brother and share my story in some solidarity. You will find something, just keep plugging away. Each "failed" attempt makes you better no matter how many attempts it takes. Cliche of course but it is true. I am very lucky in that I don't face the mental demons you do, even then this job search hit me pretty hard. Please be proactive and take care of yourself, body and mind (body goes a long way toward mind also).
slantedview 10y ago
These "shortage" stories always make me roll my eyes, because they're usually about money more than anything. And money is usually about cost of living more than anything. If you choose to locate your company in one of the highest cost of living regions in the world, then you are complicit in the "shortage". Supply and demand - pay up. Or don't.
edoceo 10y ago
We hire only the best! We only hire the top 1% of candidates. But only 1 out of 100 are qualified :(
lifeisstillgood 10y ago
Weirdly the problem is most hires have it backwards. Before going out to the market and discovering what talent exists and consequently what salary it will take to get them to join (ie negotiate) most organisations decide on a salary range, usually reflecting the current internal structure not the current external market. The longer an organisation has existed the more out of whack with the market its internal set up is. As such companies decide on their price point first, then go looking. Which is of course backwards.
wpiel 10y ago
What I've learned from the comments: If something is valuable, there is a shortage of it. I'm not even sure if I'm being sarcastic.
LawrenceHecht 10y ago
Just checked, the # of data engineers rose to 9,246 (42%) in the last six months. So, the shortage is at least being addressed by people changing their job titles on LinkedIn.

143 points by hankmh 10y ago

I'm trying to switch careers into "Data Engineering" now, as a full stack developer who is more interested in ML, and I've found almost no traction internally at my company or externally. It looks like I may just accept a full stack position at a good company that does a lot of data science for now, but though I would ask - Where are all these jobs?

PaulHoule 10y ago
"Data Engineering" is most of the work that needs to be done, but I think companies haven't identified it as a category. From my P.O.V., "Full Stack Engineer" is a place you don't want to be because it means putting out fires with whatever junk javascript is in the front end. It seems like everybody who's built a serious javascript application has invented their own Virtual DOM because none of the popular Virtual DOM libraries are good for much other than wasting time and CPU cycles. "Data Scientist" is a bad title in it's own way, in the sense that "Computer Science" is bad, but worse. To a lot of people there is a Brahmin kind of attitude associated with "Scientist" -- i.e. an aversion to getting your hands dirty. Real world data is pretty dirty and you aren't going to get far in getting value out of it unless you spend 80-90% of your time dealing with the dirt.
- pjmlp 10y ago
  There are "Full Stack Engineer" doing pure native applications, which is what I have been doing the last three years after escaping the web back into native land.
- mattnewton 10y ago
  You are correct. I thought full stack meant before building the app start to finish, but the reality is often closer to putting out other people's fires in every layer. It does pay well though and you learn a lot of what can go wrong.
  - PaulHoule 10y ago
    The fact that it pays well makes it a job you're likely to get laid off from. Most managers would rather hire two junior developers so they can screw it up faster or better yet hire some people in another country who are really fast and cheap at screwing it up.
    - mattnewton 10y ago
      That may be true but I'm not worried about that, I worry more about getting comfortable doing useless work. If I got fired it would be so much easier to go back to school, as the dream of lots of money while learning on the side would evaporate.
willis77 10y ago
We (Kaggle) run a data science jobs board (https://www.kaggle.com/jobs https://www.kaggle.com/jobs) that gets a few data engineer listings from time to time. Not all of these are active, but you may find a few interested companies via - https://www.google.com/#q=site:https://www.kaggle.com/jobs+%22data+engineer%22 https://www.google.com/#q=site:https://www.kaggle.com/jobs+%...
- mattnewton 10y ago
  Thank you guys! Doing Kaggle competitions is what got me interested in seriously pursuing ML in the first place. You are all seriously awesome. I'll look again at the board but, I didn't see anything there before that wanted software engineering skills (which I have with industry experience), and not a graduate degree (which I don't), and happened to be commutable from my place just south of the bay. But I will keep looking!
minimaxir 10y ago
ML falls more under a Data Science role than Data Engineering, although ML is much more difficult without proper Data Engineering.
ironchef 10y ago
I see tons of them. If you're interested in ML, you're probably more looking towards data science. Data engineering (in general) is more about getting the data in a state where it can be used (extracted, cleaned, moved, transformed, etc.) at least from what i've commonly seen in the industry. A decent breakdown is here: https://blog.insightdatascience.com/data-science-vs-data-engineering-62da7678adaa https://blog.insightdatascience.com/data-science-vs-data-eng...
alexbeloi 10y ago
You might want to look at "Machine Learning Engineer" positions if you want to do ML in practice, it's starting to be a title I see somewhat often now. As others have pointed out Data Engineering is more about building data pipelines, making architecture decisions for your ML stack, things like that. Less about model building, prototyping and training, which is what I think of when somebody says they 'do' ML.
- mattnewton 10y ago
  Right, I'm not picky about the title. I'm looking at those positions too. The main thing is, I want to be able to contribute using my existing software engineering skills from day 1, while picking up the ML stuff. It's been really hard to basically work an unrelated job during the day and go home and do kaggles for practice, so I am hoping to get more of an intersection as a launching place. Anything touching the data or the models will do :)
achompas 10y ago
My official title is "Data Scientist" although I'm closer to the "ML Engineer" someone else mentions in a child comment. Frankly speaking, if your company doesn't need a data engineer, it won't hire one or move you into that role. They likely don't, either, if you're experiencing this pushback -- data engineers often develop ETL pipelines or data warehouses, both of which are very useful if your company has a data team and very useless if it does not. That said, you may want to move closer to my role. There's actually a shortage of data-savvy people who can also write production software, and you would nicely complement a more research-inclined data scientist or analyst -- someone with far more experience with research/analysis than development.
- mattnewton 10y ago
  I think the company does need data engineers but wants someone with a graduate degree from Stanford or CMU in that position, even though the actual work is in building up infrastructure for those people. And I understand. I've only really got software engineering skills to contribute at this point and I'm picking up the ML from kaggles on the side; I am looking for a position that can increase my overlap between those, because learning at home while working on unrelated stuff is making me move slowly and painfully. Your experience sounds exactly like what I'm looking for - data-savvy writing production code, complementing a research-heavy team I can learn from. How did you get started in that?
  - achompas 10y ago
    I honestly fell into it by luck. I moved to NYC, studied machine learning in grad school, networked my ass off, and landed an internship. From there I went full time as something of an ML engineer at a company with a strong tech culture, and learned as much as I could in both tech and ML/statistics. The rest is history (although I'm by no means a rockstar or whatever). My path is hard to reproduce -- it starts with being in NYC or SF at a specific point in time, before the labor market became saturated with data science bootcamps and PhDs furiously learning Python while working on their dissertations. Your best bet at this point is to produce a few data-related projects (maybe work on open source like scikit-learn and pandas?) and network like crazy. Someone somewhere will have a need for someone like you.
bcbrown 10y ago
You should put your email in your profile. If you're in Seattle, send me an email.

skynetv2 10y ago

anything and everything is marketed as "data science" and "data engineering" these days becasue this is the buzzword of the day. I've been dealing with large data even before "big data" was a word but i dont call myself "data scientist" or "data engineer". I am still a software engineer working on what benefits my organization. "Serial Entrepreneur" is the same these days, claimed by anyone who had a lemonade stand as a kid.

Swizec 10y ago
> I am still a software engineer working on what benefits my organization But if you saw a nearby local maximum that's higher than your current local maximum, wouldn't you change what you call yourself, if it means being paid more but doing the same work? This is similar to how the average "software engineer" makes about $30k/year more than the average "programmer".

untilHellbanned 10y ago

Ahh the ol' write a post about a not well understood distinction and then proceed to not explain the distinction. Looks like we need more English engineers too.

realworldview 10y ago

We surely need data mechanics.

jnordwick 10y ago

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay." I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where we start to part ways. If you are really serious about a shortage, you should be really serious about making offers that can be competitive, but I keep seeing the same $150k offers. That isn't a "shortage" kind of offer.

tobyjsullivan 10y ago
"we're in the middle of a talent shortage [and don't believe in upskilling]."
- ThomPete 10y ago
  Upskilling is one of the most ineffective costly ways to try and "re-program" workers and it mostly doesn't work because it's not about skills it's about talent.
  - linkregister 10y ago
    Talent that occurs through the genetic/epigenetic process of having attained a Data Science masters degree after earning a Computer Science degree? I am a believer in inherent talent but Data Engineering is a skill set.
    - ThomPete 10y ago
      Engineering is a talent skill there is a world of difference between teaching someone starting from scratch and then starting someone first having to unlearn what they learned to then learn perhaps a completely new way of thinking. Most of the reskill programs I have heard of failed miserably exactly because the skill isn't enough.
  - 10y ago

rch 10y ago

I've heard more than one CTO/Sr. Engineer refer to people in these roles as 'data grunts' or something similarly dismissive. Then they're mystified as to why solid engineers are so quick to move up or out, year after year.

GeneralMayhem 10y ago

There are 6600 jobs listed and 6500 individuals on LinkedIn with that particular title, and therefore there's a shortage? Seriously? * How many aren't on LinkedIn? * Since the whole article is about how the job title is poorly defined and growing in prevalence, why would you assume that people who don't already have such a job would use the term? * The "growth" charts on the full study are just as bad - how much of that is just from renaming existing generic developer positions, since "data engineer" is clearly a relatively new term?

sportanova 10y ago
6500 data engineers on all of Linkedin, but 6600 job openings in the bay area. so there are more job openings in one area than all data engineers on linkedin

dmatthewson 10y ago

From the article: "Data engineers are the janitors who keep your data clean and flowing." Hm, I wonder why he's having problems hiring janitors.

kafkaesq 10y ago
Data engineers are the janitors who keep your data clean and flowing. In a boldface font, no less. The cockiness behind that language is really quite astounding.
- praccu 10y ago
  It's really true, though. It's brutal, ugly work with no hope of an end. Edit: Favorite paper on the topic: http://research.google.com/pubs/pub43146.html http://research.google.com/pubs/pub43146.html
  - kafkaesq 10y ago
    So is the work that doctors, lawyers, and other highly-skilled people do, by and large. Everyone knows that day-to-day aspects of these jobs are hardly glamorous (or even cerebral), the vast majority of the time. Yet somehow we manage to accord these people with their due degree of respect, and wouldn't think of referring to them as "janitors".
    - wavefunction 10y ago
      I don't see what's so bad about janitors, though. They do very thankless jobs for not much money, whereas doctors and lawyers and other high-skilled individuals are often well renumerated or offered certain social prestige that your post shows is quite lacking when a humble janitor is considered.
      - kafkaesq 10y ago
        A perfectly valid point. But getting back to the original article -- it pretty much takes a SV alpha-nerd (or aspiring CEO seeking to cater to them) to come up with language like that.

otto_ortega 10y ago

Am I the only one who thinks there will be a ton of people changing their job title on LinkedIn to "Data Engineer" as a result of this article?

collyw 10y ago
I am thinking about it. Actually a friend recommended that I change my title to Data Engineer a few months back.

mrharrison 10y ago

We should rename this job position to Data Sanity Engineers. I have been thrown these projects at work before, where I'm the frontend engineer and I need to make some cool D3 visualization, but low behold the data is shit, and I have to help the backend team make the data useable. It's a mind-numbing job, that nobody wants, because it sounds like a one month task to get a good REST API up and working, but it usually takes three months, because you have to go back and forth making sure the data is right, and there is always 10 tricky edge cases that you have to work some magic on. Not only that but you need to have smart people cleaning the data, so that you don't make some big mistake down the line or your REST API is super slow, and you have to add another couple weeks or month to rework the data again. So that one month becomes three months, and most likely a year, because somebody will say that looks great but can we also add this, and it goes on and on. It's literally a mind-numbing job that most nobody wants. I have found that products like Tableau are the best for this, you still have to clean the data, but it helps speed up the process. Data cleaning is a super golden problem to solve.

msie 10y ago
Data Sanitation Engineers
kafkaesq 10y ago
Not only that but you need to have smart people cleaning the data, Which are difficult to find when you think of them as "janitors", and treat them accordingly.
dizzystar 10y ago
As a contradiction to this point, some people (me) really enjoy working with data, from cleaning, munging, creating, sorting, pipelining, etc, and find front-end visualization production excessively boring and mind-numbing. Give me emacs and a command line, and I have all the truth I need, which is far more honest, in my mind, than anything that can be created with D3 or Tableau. Beauty is in the eye of the beholder, and it doesn't really do anyone service to look down on the work others find enjoyable. If doing D3 makes you happy, that is awesome, and I can only congratulate you for your passion and your ability to look forward to work I don't "get," and I wish the feelings would be mutual.

ThePhysicist 10y ago

Data engineering sounds much better than "data plumbing", but in my experience the latter is a more accurate description of the work of a data engineer: Building -and often unclogging- pipes that transport data from A to B, and putting in filters to clean it and extract the useful bits. So why not change your LinkedIn job title to "data plumber", which is sure to get you some serious recruiter attention ;)

moandcompany 10y ago

I am a data engineer working on a machine learning team with models actively used as part of our product(s). From my experiences working in various contexts (applied machine learning, analytics, policy research, academics, etc...), there are several of factors that contribute to this shortage: (1) "data engineering" often requires a lot of breadth and knowledge, (2) "data engineering" is often (derisively and naively) referred to as the "janitorial work" of data science, (3) the spectrum of roles and requirements within the "data engineering" domain, in terms of job descriptions, can range from database systems administration, to ETL, to data warehousing, curation of data services / APIs, business intelligence, to the design/deployment/operation of pipelines and distributed data processing and storage systems (these aren't mutually exclusive, but often job descriptions fall into one of these stovepipes). Some of my quick thoughts and anecdata: Companies have made large investments in creating 'data science' teams, and many of those companies have trouble realizing value from those investments. A part of this stems from investments and teams with no tangible vision of how that team will generate value. And there are several other contributing factors… "Dirty work." People haven't learned how to, and more often don't want to do it. There's a vast number of tutorials and boot camps out there that teach newcomers how to "learn data science" with clean datasets -- this is ideal for learning those basics, but the real world usually does not have clean or ideal datasets -- the dataset may not even exist -- and there are a number of non-ideal constraints. There are people that wish to call themselves “data scientists” that “don’t want to write code” and would “prefer to do the analysis and storytelling” Engineering as the application of science with real world constraints: there are a number of factors that we take into account, often acquired through painful experience, that aren’t part of these tutorials, bootcamps, or academic environments. Many “data scientists” I’ve met have a hard time adapting to and working with these constraints (e.g. we believe that the application of data science would solve/address __ problem, but: how do we know and show that it works and is useful? what are the dependencies, and costs of developing and applying that solution? is it a one-time solution, or is it going to be a recurring application? does the solution require people? who will use it? what are the assumptions or expectations of those operators and users? is it suitable? is it maintainable? is it sustainable? how long will it take? what are the risks involved and how do we manage them? is it re-usable, and can we amortize its costs over time? is it worth doing? This is part of a methodology that comes from experience, versus what is taught in data science) Larger teams with more people/financial/political resources can specialize and take advantage of these divisions of labor, which helps recognize the process aspects of applying data science and address some of the above Short story: if you view data engineering as "janitorial work" you're missing the big picture Anyone else notice that the attributes of a 'unicorn' data scientist include the traits of a 'data engineer?'

cutler 10y ago

I'm puzzled at the omission of Scala and Spark in this report.

tom_b 10y ago

Ignoring the breathless nature of the article, this is a buzzword label for a commodity skill set that pays a commodity salary in tech. It is also the commodity skill set that my employers have all paid me for. There has been for a long time hype around new technology and labels for business intelligence, data warehousing, big data, and now data engineering/science. I'm not saying there are not some roles in this space that return huge value to organizations, but that these opportunities are much rarer than the buzz indicates. I wonder if the perceived shortage is mainly hype as the shift to new cloud technologies makes many of the older ideas a little less useful - if you are plowing data into BigQuery, you probably aren't so worried about your star schema data model for reporting. I would strongly advise people that look at these types of articles to look at the roles in question and ask "Is this role on the critical path to customers paying us?" My experience has been that the answer is often "No." This is bad. I have also seen situations where businesses that do rely on smart data integration can show that they are selling dollar bills for ten cents that still have trouble getting customers on board with spending that ten cents. Business is weird.

makmanalp 10y ago

Quick sidenote, anyone know where the databases / distributed systems engineering jobs are at? E.g. if one wanted to not use these tools but also go help build these tools? I can think of Facebook, Google, Microsoft, IBM (which locations and groups within these companies / where?). I can also think of Confluent, CitusDB, Databricks, etc.

rhizome 10y ago
Market Research is a $40B industry that depends almost completely on these concepts. I'm not sure how prevalent distributed systems are with MR companies, but that's an implementation detail anyway.
- serge2k 10y ago
  > that's an implementation detail anyway. Which is what the poster was asking for.

protomyth 10y ago

I worked for about 10 years doing exactly what they want, but I ended up having to write a lot of the tools which means I'm not able to check the boxes on some tool you require which gets me punted by HR. I'm starting to think that the message is if HR is going to do checklists then developers should really make sure they work mostly with contracts that use popular checklist items.

mulmen 10y ago
As a data person I would really like to put some numbers on how much the typical HR hiring process costs a business. I don't know anybody that says they are happy with how hiring works in he tech industry but I've also never seen an HR person try and improve the process.
- pyb 10y ago
  That's because the system is already optimised for the needs of HR people.

cheriot 10y ago

It was only 20 years ago that companies hired a "web master" or a generalist to do everything. But pieces of those jobs became specialized. Now we need UX, UI programmer, general engineers, dev ops, data engineers, a data scientist, etc. And how many companies are still interviewing with fizzbuzz?

collyw 10y ago

So I know SQL, Python, Django, Java (though its been a while), Javascrit, Linux, some cloud computing and a bit of devops. Am I a data engineer? Software engineer, with a lot of database background? What makes a data engineer different from a software engineer?

njd 10y ago
- The challenge for an organization is to recognize that there is a significant difference between the 'data engineer' working on a vertical project and the 'data engineer' responsible for integrating data across the enterprise. - The project 'data engineer', in today's world, most likely will be a software developer responsible for ETL, etc. The data design will be more or less up to the software developer. - An enterprise 'data engineer' is more concerned with data that affects the enterprise. This typically involves some sort of data integration. For example, how to integrate relevant data from N projects (e.g. A,B,C .. Z) where each project has its own idea of how to represent similar concepts (e.g. person, user, customer), with different provenance, truth assertions, access rules, data retention periods, granularity of metadata (e.g. at the attribute level vs entity level), etc. The enterprise is interested in questions like 'What did we know and when did we know it?", etc. The enterprise 'data engineer' will probably levy requirements on the project 'data engineer' to meet the enterprise's needs.

jboggan 10y ago

It's digital Charlie Work [0], that's why. I really enjoy that kind of work but it is difficult to articulate your business value in that environment. The best thing is working closely with a data scientist/front-end dev who can deliver products to the analysts and executives that need the data and make sure that you get the credit for enabling new streams of data. But most of the time you are putting out someone else's dumpster fire. One advantage of data engineering: unlike front-end work, there are few non-technical people who will have an opinion on how you are doing things and burden you with bikeshedding. [0] - http://www.avclub.com/tvclub/its-always-sunny-philadelphia-charlie-work-214628 http://www.avclub.com/tvclub/its-always-sunny-philadelphia-c...

binalpatel 10y ago

The fact that the original, unmodified article referred to data engineers as "janitors" pretty much says it all. It's very analogous to front-office and back-office work in Investment Banking. "Data Scientist" are the front-office, with all the prestige, and "Data Engineers" are the back-office, doing a lot of the heavy lifting without nearly as much recognition. In my opinion there shouldn't be a delineation. You shouldn't be a data scientist if you can't gather, process, and clean up your own data.

biztos 10y ago
Ideally you'd have a symbiosis, and each side would recognize the importance of the other. Even if you require your data scientists to be able to do engineering work, it's probably way more efficient to have some good generalist Software Engineers doing all the "pre-math" work and freeing your statisticians up for what they're (hopefully) good at. Plus as a side effect, your software will probably be better.

ef5a0b0628 10y ago

Every time something comes up on HN about a talent shortage in a field related to software engineering, it hurts. I have been unsuccessfully looking for a full time position since my last start up (I was not a founder) folded six months ago. I have been on over 25 in person interviews and gone through untold degrading whiteboard interviews, code tests, trick questions, and take home projects; all have ended in rejection. This industry has a need to torture candidates because we are all considered to be liars by default. Much is said about combating impostor syndrome in ourselves but we are too eager to engender it in others. It seems people in this industry refuse to understand that some people are not perfect. I never graduated college because I hated it with the very fiber of my being, so I am not particularly great at white boarding answers to algorithm questions off the top of my head in a high pressure environment. If I need them during my job, I look up answers and learn from people who are much smarter than I am. My personal identity has been shattered, as I thought my ~5-10 year history of success in the industry indicated I was in demand and talented. I saw posts like this and thought that if the worst happened I'd still be able to find a job. The idea that there is a talent shortage is a lie, or candidates like me wouldn't be treated as I have been. I'm not asking for a free job, or a handout. I have had a successful career so far and am capable of doing good work. But I'm not a specialist in Big Data Machine Learning Neural Networks. I have struggled with bipolar disorder and suicidal ideation most of my life. I've dealt with the death of my beloved grandmother and my father who was instrumental in my choosing to be an engineer with only minor lapses in control. Nothing has caused me to consider taking my own life as much as the past 6 months. It seems there is no future for me in the only career I have any skill in and which is a huge part of my identity. And to constantly be told that there is such a shortage of engineers only salts the wound.

ultramagas 10y ago
Hey, I'm going through something similar. I had to quit an amazing job because my wife and I pursued a dream and moved to Europe (no remote). I had always had an easy time getting a job before but this time it was different. Granted I knew it'd be tougher since for remote jobs, the world is the competition. But it was a summer of endless shitty timed hackerrank-style tests (virtual whiteboard hazing). I would tell my co-workers about them and they'd laugh in bewilderment at the questions that were asked in what should be a technical screener, and these are extremely smart and productive software guys that have started companies, written books, give conference talks. One funny question I got for a frontend React job: write a function that takes a sequence of bits that represent a negative-binary number (not a base-2 number that is negative, but a base-(-2) number) and return its negated value in base-2. For a frontend job. It was one of 4 questions to be answered in 90 minutes. gtfo. A few companies would reply, most strung me along while -- I realize now -- they were keeping me as a backup(-backup) incase their "A-player" turned them down. Countless interviews, hours on takehome projects, it was tough. I learned to cut bait if the company was slow to move forward, had weeklong periods of no communication, etc. I (just very recently) found it's easier to land small contract gigs because the barrier to entry seems to be lower, demonstrate value, and keep getting work from those guys after the initial project was done. It is different but so far I actually like the freedom that comes with contracting. I haven't been at it long enough to experience the downsides. There's definitely not a shortage of talent. It's that every company thinks they need "A-players", when the vast, vast majority are doing a damn basic CRUD app. Just wanted to say I hear you brother and share my story in some solidarity. You will find something, just keep plugging away. Each "failed" attempt makes you better no matter how many attempts it takes. Cliche of course but it is true. I am very lucky in that I don't face the mental demons you do, even then this job search hit me pretty hard. Please be proactive and take care of yourself, body and mind (body goes a long way toward mind also).

slantedview 10y ago

These "shortage" stories always make me roll my eyes, because they're usually about money more than anything. And money is usually about cost of living more than anything. If you choose to locate your company in one of the highest cost of living regions in the world, then you are complicit in the "shortage". Supply and demand - pay up. Or don't.

edoceo 10y ago

We hire only the best! We only hire the top 1% of candidates. But only 1 out of 100 are qualified :(

lifeisstillgood 10y ago

Weirdly the problem is most hires have it backwards. Before going out to the market and discovering what talent exists and consequently what salary it will take to get them to join (ie negotiate) most organisations decide on a salary range, usually reflecting the current internal structure not the current external market. The longer an organisation has existed the more out of whack with the market its internal set up is. As such companies decide on their price point first, then go looking. Which is of course backwards.

wpiel 10y ago

What I've learned from the comments: If something is valuable, there is a shortage of it. I'm not even sure if I'm being sarcastic.

LawrenceHecht 10y ago

Just checked, the # of data engineers rose to 9,246 (42%) in the last six months. So, the shortage is at least being addressed by people changing their job titles on LinkedIn.