14 ms·

Desperate for Data Scientists

157 points by charlysl 8y ago

techstrategist 8y ago
Can anyone with industry hiring knowledge comment on this gap? Maybe I’ve misunderstood but from HN / other online discussions it seems that employers in Data Science have been increasing education requirements to MSc, PhD, and so forth for many jobs that more appropriately require a strong statistical understanding combined with rigorous data processing skills and some imagination. Is this a real “skill gap” or an attempt to create artificial scarcity of hot jobs?
- isoprophlex 8y ago
  IMO, as companies' data science capabilities mature, people with actual skills are needed. 5 years ago DS was hot and new. As a middle manager you would call up a consulting meat farm and ask for data science consultants: presto, you have a data science team. These people would, likely, be random BSc's with a SAS base certificate. In 2018 data science, and data driven decision making, is still hot, but hiring managers know they need people with a formal education. So, education reqs go up and scarcity increases.
  - compcoffee 8y ago
    >but hiring managers know they need people with a formal education It's the exact opposite: as better tools are developed, there is less "formal education" required. In the same way that Web Devs are in high demand, and nobody needs a Computer Science degree to do that work. The reality is, "data driven decision making" is something businesses large and small are discovering they need. And often times that sort of analysis requires a regression, or maybe something slightly more sophisticated. What these businesses do not need is some cutting edge algorithm or complex AI pipeline. There is a ton of gate-keeping in this space, as the the PhDs learn that domain experience and business acumen are more important than a deep knowledge of the algorithms. A significant amount of the work is cleaning and processing the data. This is the way of the world. At the end of the day, most business problems boil down to inference and prediction, and most times you don't need to be incredibly precise; the easy 80% will do.
  - conjectures 8y ago
    +1 because mooc style learning alone just isn't enough in most cases.
    - albemuth 8y ago
      What would you say is missing from a mooc-based training?
      - conjectures 8y ago
        Having looked through the contents of a couple of data science moocs they often try to give intuition without math. When trying to put it into practice, if we go the slightest distance off piste we are lost without the math. So it's a fragile form of knowledge. If we take someone who's already strong at math from some other training and teach them more stats/ML concepts and intuition, then great. But the value add of the mooc learning is quite low in that scenario. The best alternatives (barring a formal course) imo, are text books that have exercises like Bayesian Data Analysis by Gelman.
- AmericanChopper 8y ago
  In my experience, data science is turning massive amounts of potentially unstructured data into some sort of insights for business. I guess depending on the volume and structure of the data, and the level of insight required, you may need to be more or less qualified to do the job. A bachelors degree would be fine for most of the roles I’ve seen in places I’ve worked, but I’ve never worked anywhere that had google level demands. I think getting insights out of massive amounts of unstructured data is a relatively new problem for most organisations, so that might explain the skill gap. Personally I also think the ubiquity of ORMs might also play a role, as they remove a lot of the day-to-day data handling that many engineers would otherwise have to do. But I might be completely wrong about that.
- hellogoodbyeeee 8y ago
  I think we are still in the wild west of the Data Science field, at least in the Midwestern U.S. I have seen job postings for Data scientists that are simply rebranded data analysts to postings looking for advanced math and programming ability. I am suspicious that bootcamps and online courses will set individuals on to long term data science careers. I think there will be a large demand for individuals who understand basic data science principles such as the difference between supervised and unsupervised learning problems. I think these people will be given subscriptions to services that automate a lot of the actual data science work flow. They will use programs like Data Robot. However, I also think these individuals will be paid closer to what a data analyst is than what a software engineer gets. I think that certain companies will always be looking for people who can actually do traditional data science work from scratch because the automated tools will not apply to their specific use cases. I think these individuals will probably generally have a master's or PhD. I think right now the field is too new for managers to be able to tell the difference between what kind of Data Scientists they need. My experience has been a lot of companies need the former, but they think they need (or they want) the latter. I suspect in the coming decade these two roles will find different titles.
  - compcoffee 8y ago
    >I have seen job postings for Data scientists that are simply rebranded data analysts to postings looking for advanced math and programming ability. Being as how "Data Science" is such a nebulous term, what is wrong with such a posting? I'd say that a data analyst with some advanced math and programming ability is a pretty rare skillset. Is it any surprise such a person is in high demand?
    - telchar 8y ago
      When looking for data science jobs it's nice to know from the prospective hire side that your skills will be used well. Coming into a job only to find out you're way overqualified can be a downer, particularly in a field where you need to keep your teeth sharp. I think the same goes for programming - imagine finding out on your first day you'll just be doing user testing and not writing any code.
- DrNuke 8y ago
  There is no gap, actually. It is plain, old supply vs demand. Second and third class firms wanting the few Einsteins out there for peanuts, being flooded with CVs from kaggle noobs and mooc kids. Point is the latter are good enough for the jobs on offer but the former feel somewhat entitled to be choosy & stingy, even if they have no real business case to pursue.
- tixocloud 8y ago
  There is some value in the additional rigorous data processing skill set. The intent is to get away from just merely slicing and dicing data to come up with some insight that explains causality. Whether we are afforded the time to do a proper and thorough analysis is a different matter.
- citation_please 8y ago
  I'm on a hiring committee for a job with the title of "Machine Learning Engineer". The most qualified candidates we have come through the process are physics PhDs who did a significant amount of coding during their PhD and have been able to segue into machine learning because they were interested. Our other tier of hireable candidates have been individuals with 4+ years of industry experience, usually with CS PhDs with a machine learning specialty. The rest of our team came from internal transfers, people who were less qualifed but proved that they would have a positive impact on the output of the team.
  - drawnwren 8y ago
    I'm curious why you are seeing such strong candidates from the physics department. Care to elaborate?
    - analog31 8y ago
      I'm a physicist, and got my degree in the early 90s, but maybe I can shed some light on this. First, physics has been computation driven since the 1940s. When I was in grad school, programming was vital to my project. I wrote mountains of code, and also designed my own electronics. Second, I noticed a weird difference between physics and engineering. Engineering students were given problems that were expected to be solved within a particular domain of engineering. A physics student might be given a problem with no idea of how to solve it, much less even how to define the problem itself clearly. That was my project. So a physics student could find themselves having to learn practically any technical skill. Third, a matter of motivation. We knew that we would need to make ourselves employable. My project would occasionally fail in some spectacular way that would require me to learn more programming, or more electronics. Imagine that. ;-) Fourth, possibly also as a matter of employability, math and physics people have always figured out how to worm our way into ill-defined, nascent areas of technology. These are areas that have not yet created a mainstream training pipeline, so we can plausibly claim to be as well trained as anybody. "Embedded systems" was such a field when I was finishing school.
      - citation_please 8y ago
- bertil 8y ago
  As the hiring manager for that kind of profile for more than ten years, the problem is rather simple: they want data scientist, but they don’t have good data. The position are still open but no one well-informed would apply. Managers want to understand what’s happening in their company; they’ve been told data scientist can make that happen, or better, fix the problems automatically. They expect the solution to come without significant investment in data collection. One, even over-paid, data scientist is a far smaller budget than having your dozens of teams make an effort in fixing data collection, and setting up a reporting team. That kind of problem is easy to detect during the interview, if you’ve been there, and any data scientist has. What’s left are fresh graduates without diploma, who were taught statistics, but not how to get a project running -- so they don’t know what to say during an interview. In more details: Too much focus on stats. The actual skillset required should not be that demanding on statistical side: interviews generally focus on testing that exclusively. It takes the form of Whomever has any statistical background will grill on what they know best, their own MSc thesis. The thing is: no one can know the minutia of every model, certainly not a student’s interpretation of them, so those end up being a lot of “I would have to go back to the documentation.” Unless your interviewer is mature enough to either know that’s reasonable or stop doing that, it’s a lost cause. At best, you end up hiring a pile of people with the same focus, and every problem will look like a nail to a group of people carrying hammers. One alternative is to ask about something they did recently, but that’s rarely very instructive, either because the interviewer has no idea if that makes sense, or because the implementation has a lot of properties that make the whole process tedious. The focus should be the ability to learn new techniques at speed. Because the vast majority of the time, the right model is: a. a basic implementation of a better one, just to get things started (fix the data collection, the engineering of the input to the serving model, think about scaling the serving model, improve the actions taken from the decision, improve the decision format and the actual model objective) b. a model that you haven’t heard about, and you need to learn about it within a calendar that is conditioned by a. not being sufficient. Be more project-focused Description like “codes and does statistics better than either” fail to mention that the bulk of the time, a data scientist is actually more of a project manager, trying to identify blockers, get the integration of their tool prioritised, compare the impact and risk of different approaches; there’s also a big chunk (if not all of it) of being an analyst, generally to clarify what problem is being asked, or checking that the data is consistent. That’s if all is reasonably good: far too often, the time is spent trying to get management to grasp that Machine learning can only predict something, and that what to do with that, what to expose to the users, how, when, is more the remit of the product team i.e. you don’t have a real direction. Many large companies actively select against people willing to say that the kind is naked. Some company test for “business sensitivity” but that too often means checking that aspiring data scientist can walk through a P&L. Reality is closer to: Can you listen to a project manager, frustratingly oblivious to his own contradiction and general ass-holery, rant for 45 minutes without strangling him? Because you need the support of his team to get through. Fight for good data I have yet to see a company that had data you could use for data science on day 1. The vast majority of the time, it was inconsistent. Not a little bit, or hard to find: big, glaring, obvious contradictions like 20% of the revenue unaccounted for. If there are analysts, they generally have found a way around it, rather than address the issue. In more mature companies, there were some problem that would make basic model stumble, and more nuanced one go to very stupid conclusions. The classic case of a trip-up on good, but not well-structured: What drives retention? Completely failing to delivery your service: people will re-order within the hour, a larger ratio than any other case. Executives pay for the servers so they think that data quality if measured in peta-bytes. They get fed a continuous streamed of over-analysed data, filtered by hand from any inconsistency. They have no idea how inconsistent it really is. Any improvement is seen as a cost centre, rather than the ability for the company to be better informed. Don’t get me started on technical debt, short-term focus, insensitivity. What data scientists need, before they even join: clear objectives, clear breakdowns to understand what effects are correlated, clear plans to address some of the concern, pilot tests to make sure that those solutions would work, and collect some data, audited data pipeline with clear documentation to avoid misinterpretation, dedicated engineering to help with data logging, hosting computation, serving model. I am willing to bet that hardly any of those dozen of thousands of jobs looking for candidates don’t have most of that. There is a small piece of good news: Agrawal & al. summarised their theory on how ML will make predictions cheaper in a all-access book, Prediction Machines. The predictable success of that book will help. Basically, it tells about what ML does, without ever doing any ML at all: it just looks at how it’s a lynchpin technology, like search engine, digital photo, etc.
acbart 8y ago
I've had a few friends express interest in pivoting their domain knowledge + small programming knowledge into a Data Science job. Are there many success stories or guides out there for those who make such a transition? I went the formal route of getting the CS degree, so I never really thought about how you turn a non-CS degree into a CSy type job.
- ThePhysicist 8y ago
  Most data scientists I know don't have CS degrees but instead a background in physics, biology, math, economics or statistics. For a data science role focused around exploratory analysis, model building and idea generation (as opposed to a more engineering-focused data scientist) I'd say that a basic understanding of programming concepts and familiarity with the most essential tools (version control, Linux, Python/R) is probably enough to be accepted into a data science role. Data science degree programs are still quite novel so most companies don't expect to recruit people with such a degree, hence there are good opportunities for people with a STEM education I'd say.
  - ekianjo 8y ago
    > Data science degree programs are still quite novel Indeed, plus you have zero business experience straight out of such a program so you are at a disadvantage if you are starting your career as such.
  - bobmarley1 8y ago
    I've actually seen many many job postings looking for people with a PhD in Data Science which is truly mindblowing because there are only 2-3 such programs in the Country and most of them just started so none of their students are even on the job market yet.
    - PAClearner 8y ago
      ummm you are probably wrong. those job postings probably want people with PhDs in Computer Science/Statistics/Applied Math that did hard things with data.
TadaScientist 8y ago
You don't need to have a CS degree to be in "Data Scientist". In fact, I would argue that you need a statistics degree. There is a difference between a data scientist and an engineer.
- disgruntledphd2 8y ago
  The problem with stats degrees is twofold: 1) They often lack experience with messy (i.e. real) data 2) They sometimes lack experience with programming. Data science, as currently practiced requires some CS, some stats (with a focus on experimental design/observational analysis) and communication skills. These skills are rarely taught together in undergrad, but lots of PhD's acquire them by accident, hence the propensity of PhD's to end up in these roles.
pvaldes 8y ago
Maybe oral communication skills are overrated. Who knows? People currently talk a lot also with their hands.
tw1010 8y ago
I feel like I hear just as many people getting degrees in "AI" nowdays as plain CS degrees. A much higher percentage than five years ago. It feels like everyone and their grandma is minoring or majoring in AI or machine learning. And when I talk with companies about it, they're lukewarm at best. I really wouldn't bet my personal career at this point on going into this area. It'll be filled to the brim sooner than you expect. It's a much better, and just as intellectually satisfying, bet to just go into plain traditional CS in my opinion. (A counter argument is that I felt about the same sentiment around three years ago, but my intuitions doesn't appear to have manifested, so maybe I have poor intuitions, or I'm missing some piece of the puzzle.)
jakecrouch 8y ago
I view modern AI as a combination of things that are too early and too late - data science hasn't changed much since the 1990s, but people have a misconception that there is still a lot more to do in data science because of the undeserved excitement around deep learning.
- sgt101 8y ago
  Errm - I am pretty sure that we didn't have MCMC or causality analysis in the 90's. Or R-studio.
  - johnmyleswhite 8y ago
    Perhaps I've misunderstood your desire to be sarcastic, but I think your timeline is off. To give two concrete examples, Geman and Geman's "Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images" paper is from 1984 and Holland's "Statistics and Causal Inference" paper is from 1986.
singingfish 8y ago
I'm available for the right data science gig. Somewhat unconventional CV available on request. I'll try to bring my competent but intellectually foggy business analyst colleague along for the ride if you like.
DoofusOfDeath 8y ago
It would be nice if the story touched on compensation levels. If all of those companies offered $400k/year, with guaranteed employment for 7 years, they would suddenly find no shortage of data scientists. Although other industries might suddenly wonder what happened to all their statisticians / physicists / etc. And two years later there would be a glut of data scientists who had just completed their master's degree.
- wenc 8y ago
  Not every labor shortage problem is solved by adjusting compensation levels upwards. The compensation has to match the value created.
  - bobmarley1 8y ago
    Well not according to the people who constantly talk about Supply and Demand. If there's a massive supply shortage as implied there ought to be a corresponding increase in Compensation no?
  - jlelonm 8y ago
    Why is that, economically speaking? If there's a labor shortage, doesn't that mean it's undervalued in the first place? Forgive me if that sounds naive, I'm no economist.
    - wenc 8y ago
      Because it makes the assumption that the system has no constraints, which is often untrue. First of all, you have to ask yourself why there is a labor shortage. If the underlying reason is insufficient pay, then sure, increasing pay will fix it. This is often the case for fungible talent -- incentivize something enough, and people will shift resources to it. However in many situations, certain types of work are not fungible. There are natural barriers to entry and qualification issues. Surgeons, for instance. You can incentivize and compensate all you want, but the fact is, not everyone is cut out to be a surgeon, so you have a funneling effect. It's not even a matter of pay. Then there's desirability issues. Deep sea welding is a highly specialized (and dangerous) trade that pays handsomely, but not everyone wants to do it. Some physically demanding jobs also have natural attrition issues in spite of compensation. When the inflow of talent is smaller than outflow over a long period of time, a shortage results. There's also a training and timing issues. Let's say it's 2011 you want someone who can build the kind of infrastructure that powers Netflix, in 6-12 months. The talent and experience pipeline would still have been brewing at the time, so there's going to be a temporary shortage of talent and experience until folks gain experience and mature in the field. In that instance, you can still hire and grow personnel over time, but there would still be a shortage of existing talent. Talent pipelines take time to build, are highly dependent on talent pools available and the ecosystems around it. Just throwing money at the problem doesn't always work. Compensation is just one end of it (the opportunity end); there are also significant long-term investments needed on the other end (the cradle end, which includes education, development, etc.). Texas Instruments did this -- they funded what eventually became UT Dallas... but it took many many years and the outcome was uncertain. There's geographic issues. To use an extremely unlikely example, let's say you wanted someone at the level of Jeff Dean or Sanjay Ghemawat, but you would need them to relocate to Podunk, Iowa. Very few people at that level would want to relocate for any amount of pay, hence a shortage. I realize this is a pretty extreme example, but I worked at a company in an undesirable part of the country and it was difficult to get truly talented folks to move out there even with significant premiums on compensation. Now you can keep increasing compensation until you get someone who's willing to move, but they're usually not the kind of talent you were looking for in the first place. Also there's a break-even point at which the compensation doesn't make sense for the value the position is likely to generate, so companies will just not hire. My point is there are all kinds of real and complex reasons why increasing compensation alone will not solve all shortage problems.
PeterStuer 8y ago
Two remarks from an old AI guy here. (1) If you're going to hop on the 'hot' AI/ML/DS bandwagon, make sure that you actually have 'work' for the highly coveted person you are trying to hire. 'Everyone else is doing it', 'all our clients have data, so selling them data-sciency projects shoudn't be a problem', 'Hire a first DS person, to have them start a brand new DS division' are things that probably are doomed to fail. If you weren't selling classic Data Analyses/Intelligence services before, chances are you are lacking the foundation in your sales and support staff, and maybe even your customer type, to be able to embed this into your company and into a new line of business. The risk your prize new employee will walk to greener horizons to work with peers on exiting real projects is high. (2) Know where you are heading. If you just want to embellish your 'normal' projects with some gimmicks from the online 'Cognitive Services' api's offered by the various vendors, then yes, sending some of your regular staff on a quick 'boot-camp' style course will probably work. You don't need a 'data-scientist' for that. And if you hire a real one for that type of work he'll probably walk within 2 weeks.
- tixocloud 8y ago
  Sage advice. There is a significant misconception on what a data scientist does and many roles don’t seem to truly bring out their potential. Most get grouped into doing everything around “data”.
- maxxxxx 8y ago
  Agreed. From what I can tell the data scientists at my company pretty much are running a few SQL queries with simple stats. They also seem to have a high turnover. It looks like they got hired mainly because the company wanted "data science" without knowing what it means.
- amrrs 8y ago
  That's a very good advice. That expectation and reality mismatch causes a lot of problems and that's also well captured in this article. https://towardsdatascience.com/why-so-many-data-scientists-are-leaving-their-jobs-a1f0329d7ea4 https://towardsdatascience.com/why-so-many-data-scientists-a...
- apohn 8y ago
olivermarks 8y ago
I smell buzzword HR bingo
- thanatropism 8y ago
  Yeah. I find "data science" a cringey expression and I'm not looking for a job currently, so my LinkedIn emphasizes my strengths according to my ego/self-appraisal, not a "desperate" reaching out for fashionable job titles.
adiusmus 8y ago
Ah yes. A shortage. Someone didn’t do their planning. Bubble bubble tool and trouble. Or is this about more visas? And paying people less. Seriously though, what exactly is a data scientist? What is the requisite skill set? Is this masters level entry or would my feline require a Pointy Headed Degree upgrade to play this game?
- adiusmus 8y ago
  Data scientist buzzword compliance: Spark, Hadoop, Hive, Pig, SQL, Neo4J, MySQL, Python, R, Scala, Tensorflow, A/B Testing, NLP, anything machine learning
- adiusmus 8y ago
  Masters and PhD are required for particular employers. Others not so much but there are caveats. High probability of being the “data” guy who is then made to be the database admin. Better summary at: https://blog.insightdatascience.com/from-phd-to-data-scientist-17209c66a9d9 https://blog.insightdatascience.com/from-phd-to-data-scienti...
NatW 8y ago
I've heard it said, "A data scientist is a better statistician than a typical programmer and a better programmer than a typical statistician." I 'graduated' from the Data Analyst Nanodegree @ Udacity in my spare time, and have experience with both. I lack a background in math beyond university Calculus, though, so that seems the major barrier for me to get a foothold in the industry.
- tixocloud 8y ago
  It depends on how companies staff their data science teams. I’m comfortable with a lead data scientist that oversees “less math” data analysts who build the models. If you get the high level concepts of stats and linear algebra, it should be enough that someone else with a PhD can help fill the knowledge gap.
- friday99 8y ago
  The other way to phrase that is "A data scientist is a worse programmer than a typical programmer and a worse statistician than a typical statistician." If you need a programmer, hire a programmer. If you need a statistician, hire a statistician. I wouldn't ever hire someone who took a bootcamp that is "a better plumber than a typical doctor and a better doctor than a typical plumber."
  - deleted 8y ago
    [deleted]
duchenne 8y ago
I think that the challenge to hire a good data scientist is that it requires skills in both: - hard science: strong statistics or ML to analyze data, some CS to organize the data. - soft skills: strong business sense to understand what is important, some communication skills to explain it. A data scientist needs very good skills and experience in both, but they are somehow anti-correlated in my experience.
- tixocloud 8y ago
  That’s why we usually have a team that complements each others’ skill set. The data scientist while great to have everything does not necessarily need to have everything. I tend to focus on their core strengths and depending on what their growth/career aspirations are, help to develop the skills. If it’s soft skills, happy to help them develop that edge.
citation_please 8y ago
As someone who has been producing value in a data science/machine learning role for multiple years, it's disheartening to see comments that I may be blacklisted from positions due to "only" having a bachelor's degree. Somewhat non-humbly, I was valedictorian at my high school, I triple-majored at a respectable Big 10 school, I actively use all 3 majors on a daily basis, in a foreign country, and sometimes in a language that is not my mother tongue (as an American). I can't justify spending time and money on a master's degree (millennial wealth problems) where many courses would just be putting a formal, academic spin on ideas that I'm familiar with from a practical business-value-producing point of view. Any advice on how I can effectively jump off the black-lists?
- SOLAR_FIELDS 8y ago
  The UG online masters of computer science is a good option. Many people in this site speak highly of the curriculum[1]. Last time I checked the total for the entire curricula if you pass every course on the first try is around $7000. As a bachelor holder myself looking to break into some of these higher-salary and in-depth roles I’m certainly considering it. It’s even better if you have a company that will pay for it, and because it’s so cheap even the most meager offerings from companies will cover a good portion of it. 1: https://news.ycombinator.com/item?id=15018002 https://news.ycombinator.com/item?id=15018002
  - treyfitty 8y ago
    UG and GaTech are two different schools
    - madcaptenor 8y ago
      I'm not even sure what school "UG" refers to. (People generally call the University of Georgia "UGA".) In any case, the program here is from Georgia Tech.
  - citation_please 8y ago
    Yeah I'm definitely familiar with the Georgia Tech offering. But my impression is that it will be a $7000 + $(my_hourly_rate) * (hours_spent) certificate that will only get me past the employers who have a "Select your highest degree level" drop-down on their application form. Is it that much more respected than a collection of MOOCs? As someone who's also involved in hiring, if I see no industry experience + GA tech online degree it's still on an entirely separate tier than 2 years of industry experience. But that's my bias I suppose, and part of the reason that I'm not the only one on the hiring committee.
itg 8y ago
In my experience, the title 'Data Scientist' has become to mean data analyst at most companies, meaning working with Excel, Tableau, and SQL. Maybe R if you're lucky. Companies doing ML/AI will usually have a small team of Research Scientists who mostly hold PhDs, and a team of supporting ML Engineers.
- tixocloud 8y ago
  I’d be more inclined to think that at a startup, because of the lack of infrastructure, a data scientist will do a lot more than research.
  - mlthoughts2018 8y ago
    It’s even worse at established companies because data scientists still only do the data plumbing and simplistic analytics tasks, but not due to anything reasonable, like the “many hats” needs of a startup, but instead because of IT dysfunctiom and bureaucracy.
    - tixocloud 8y ago
      Think it depends but it does happen. We’ve been able to carve ourselves out of IT with our own infrastructure and our data scientists specifically focus on research and analysis. The only issue is when we try to get anything to prod and we hit IT.
      - mlthoughts2018 8y ago
        > “The only issue is when we try to get anything to prod and we hit IT.” Which inevitably means the one person on the data science team who is good with linux and docker suddenly becomes the IT wizard, and their time gets sucked up by having to find ways to go around utterly stupid barriers and tactics used by IT to avoid doing work to help you. Source: I am currently this person for my team.
        deleted 8y ago
antt 8y ago
As someone who has been employed in this space recently: No there is not. There is however a lot of middle managers who have heard the term and want to pay crud vba developer wages for people to make pretty graphs in excel. The credential inflation is also getting ridiculous. I have an MSc and I don't see how me being able to do quantum chromodynamics at one point in my mid 20s adds value to a business ten years later.
- collyw 8y ago
  That's what I am thinking. Any half decent software engineer could probably produce the results they need from a few SQL queries. But "data science" is hot at the moment.
  - telchar 8y ago
    That really depends. I'm working sometimes on problems that physician researchers have been cranking away on for decades. We want to do better than their state of the art. That's not something you can throw a software engineer at. There is a pretty wide spread of problem difficulty out there.
  - dswalter 8y ago
    A half-decent software engineer can easily and quickly produce a lot of data products. If you divide the responsibilities of a data scientist into two broad categories: * Can I get the answer I need/build the product I need? * What is the question I should be answering? What do I need to build? Software engineers tend to do quite well at the former, but are not necessarily guaranteed to be successful at the latter.
    - collyw 8y ago
      In my experience its the software engineers bosses that are poor at the later.
  - i-am-charmander 8y ago
    Comments like this reinforce the idea that data science is too broad a term. Everything data-related is thrown into the "data science" bucket. You describe what I'd call business intelligence or even advanced analytics. SQL jockeys (not a pejorative) really slicing into the data and presenting it in meaningful ways. There's usually exists some science behind these exercises. Then there's machine learning, which also gets clubbed into the data science camp. Try capturing a convolutional neural network in SQL - you might go bald from pulling your hair out. It's totally not the tool for the job. While perhaps any half-decent software engineer could maneuver his/her way around the great ML libraries out there to produce a CNN (or something of the like), I would hesitate to give them a project that involves true science (bring out your pitchforks). Experimental design, hypothesis testing, etc. While being a SQL expert could help turnaround time and enable you to test more interesting hypotheses, I wouldn't leave most DBAs to develop and carry out an experiment. So I guess my point is "data science" is too broad and overused a term to be meaningful.
tmalsburg2 8y ago
I'm teaching stats and scientific computing at university. I thought that data science might be a good exit plan in case I get feed up with academia. However, even though data scientists seem to be highly sought after in Germany (judging from what I see on linkedin), the salaries are fairly disappointing.
JustSomeNobody 8y ago
Any time I see businesses complaining about shortage of skilled labor, I read it to mean they aren't willing to pay enough to attract the skilled labor they want. The formula is simple. Businesses are in business to return as much money to investors as possible. IE: they're all about the money. Employees, more and more, are saying, "So am I. You want yours, so I want mine." Pay more and you'll attract more talent. Simple.
another-cuppa 8y ago
I was hired as a data scientist for a large company that thought they needed data scientists. They didn't and I added no value to the company. I left after about a year. If there are many people like me then that might explain the shortage.
Aeolun 8y ago
Maybe they’re all desperate for good data scientists? The truly good data scientists that I’ve found have actually been scientists. Literally all “data scientists” I’ve found in a corporate environment have been so useless it is depressing.
- ccdsacc 8y ago
  My experience is that most valuable data scientists are those who are good engineers.
madenine 8y ago
The market for serious data scientists is growing, but is not as dire as these articles make it seem. The private sector has wrapped 'senior data analyst','BI developer','data base administrator', and 'data engineer' into 'data scientist', along with the ML/AI/stats roles you would expect. So sure, when you put all the roles that touch data into one basket, and we're in a world that demands increasing familiarity with data systems and data analysis techniques... a gap is going to appear. Color me shocked Typical data scientist role I see posted is looking for: - SQL - AWS/Azure - Tableau/Power BI/Other BI tools - Python/R - Increasingly Hadoop/Spark With Java/Scala/C#/C++ listed as 'nice to haves' along with 'machine learning' and 'big data'. These are not research roles, and while they often want a masters degree or more they really don't need them. If you fit that bill you can probably get a role paying $95-120k just about anywhere in the US. You'll wrangle data, make pipelines, build dashboards and occasionally deploy a model or two when the planets align. If you want to/are capable of doing more ground breaking/cutting edge work, these roles will crush your soul in 8-12 months.
- nabla9 8y ago
  > Typical data scientist role I see posted is looking for: - SQL - AWS/Azure - Tableau/Power BI/Other BI tools - Python/R - Increasingly Hadoop/Spark They are looking for programmers. Unless... C programming makes you a hardware engineer, Fortran programming makes you a physicist, Ada programming makes you a rocket scientist, and Lisp programming makes you a god.
- antt 8y ago
  >If you fit that bill you can probably get a role paying $95-120k just about anywhere in the US. You'll wrangle data, make pipelines, build dashboards and occasionally deploy a model or two when the planets align. If you expect to pay that little to anyone who knows the above to any degree you will have a bad time. A number of projects I have joined absolutely look like someone treated them as a way to learn the technology before moving onto better paying jobs, leaving the business with a terrible mess that's largely unfixable without a rewrite. You're better off hiring one developer that knows what they are doing at above market rates than three who don't for the same price.
dostres 8y ago
I’d have a hard time finding a job in ML for $300K but I’d have an easy time finding a job in PHP contracting for $350K. This is despite having 10 years experience as a ML practitioner. I quit my last job after bringing in over $100M in profit and only getting a $30K bonus. Now it would take ~$500K and an ‘eat what you kill model’ for me to consider going back to work. These days I make my money from licensing ML products which gives me a lot of time to study - which is what I enjoy doing.
- gcb0 8y ago
  where do you find php contracting for that which lasts long enough to make 350 a year?
  - dostres 8y ago
    Initially one of those elite consulting marketplace things. And then my own pool of referrals. I hate PHP so I set my rate higher on those projects and people still pay it.
    - j88439h84 8y ago
      What is an elite consulting marketplace? Where do I find one?
  - deleted 8y ago
    [deleted]
- tw1010 8y ago
  Care to share a link to your licensed ML product, for those of us who're curious? Got any tips for people who are playing with the idea of doing something similar?
  - dostres 8y ago
    It’s face recognition software, I don’t have a website, and for obvious reasons I’m pretty quiet about it. The field has changed so much that I don’t know if I could still give advice.
Max_Mustermann 8y ago
How do you see the chances of someone without a degree getting an entry-level, low-paid, remote job in this space? What would be things that could help out? Certifications? Nano-degrees like Udacitys? A portfolio of public work?
- clu1590 8y ago
  It's highly unlikely. There is a lot of competition for entry-level jobs and not many remote positions available. The barriers to entry are exactly academic qualifications over certifications and portfolios.
- ellisv 8y ago
  Remote job? Probably not without experience and/or connections. Portfolio is going to be the most helpful, followed by certifications and nano degrees.
En_gr_Student 8y ago
Who are you kidding? They aren't hungry. There is lots of propaganda flying around, but they aren't really even looking hard. HR games the system.

157 points by charlysl 8y ago

Can anyone with industry hiring knowledge comment on this gap? Maybe I’ve misunderstood but from HN / other online discussions it seems that employers in Data Science have been increasing education requirements to MSc, PhD, and so forth for many jobs that more appropriately require a strong statistical understanding combined with rigorous data processing skills and some imagination. Is this a real “skill gap” or an attempt to create artificial scarcity of hot jobs?

isoprophlex 8y ago
IMO, as companies' data science capabilities mature, people with actual skills are needed. 5 years ago DS was hot and new. As a middle manager you would call up a consulting meat farm and ask for data science consultants: presto, you have a data science team. These people would, likely, be random BSc's with a SAS base certificate. In 2018 data science, and data driven decision making, is still hot, but hiring managers know they need people with a formal education. So, education reqs go up and scarcity increases.
- compcoffee 8y ago
  >but hiring managers know they need people with a formal education It's the exact opposite: as better tools are developed, there is less "formal education" required. In the same way that Web Devs are in high demand, and nobody needs a Computer Science degree to do that work. The reality is, "data driven decision making" is something businesses large and small are discovering they need. And often times that sort of analysis requires a regression, or maybe something slightly more sophisticated. What these businesses do not need is some cutting edge algorithm or complex AI pipeline. There is a ton of gate-keeping in this space, as the the PhDs learn that domain experience and business acumen are more important than a deep knowledge of the algorithms. A significant amount of the work is cleaning and processing the data. This is the way of the world. At the end of the day, most business problems boil down to inference and prediction, and most times you don't need to be incredibly precise; the easy 80% will do.
- conjectures 8y ago
  +1 because mooc style learning alone just isn't enough in most cases.
  - albemuth 8y ago
    What would you say is missing from a mooc-based training?
    - conjectures 8y ago
      Having looked through the contents of a couple of data science moocs they often try to give intuition without math. When trying to put it into practice, if we go the slightest distance off piste we are lost without the math. So it's a fragile form of knowledge. If we take someone who's already strong at math from some other training and teach them more stats/ML concepts and intuition, then great. But the value add of the mooc learning is quite low in that scenario. The best alternatives (barring a formal course) imo, are text books that have exercises like Bayesian Data Analysis by Gelman.
AmericanChopper 8y ago
In my experience, data science is turning massive amounts of potentially unstructured data into some sort of insights for business. I guess depending on the volume and structure of the data, and the level of insight required, you may need to be more or less qualified to do the job. A bachelors degree would be fine for most of the roles I’ve seen in places I’ve worked, but I’ve never worked anywhere that had google level demands. I think getting insights out of massive amounts of unstructured data is a relatively new problem for most organisations, so that might explain the skill gap. Personally I also think the ubiquity of ORMs might also play a role, as they remove a lot of the day-to-day data handling that many engineers would otherwise have to do. But I might be completely wrong about that.
hellogoodbyeeee 8y ago
I think we are still in the wild west of the Data Science field, at least in the Midwestern U.S. I have seen job postings for Data scientists that are simply rebranded data analysts to postings looking for advanced math and programming ability. I am suspicious that bootcamps and online courses will set individuals on to long term data science careers. I think there will be a large demand for individuals who understand basic data science principles such as the difference between supervised and unsupervised learning problems. I think these people will be given subscriptions to services that automate a lot of the actual data science work flow. They will use programs like Data Robot. However, I also think these individuals will be paid closer to what a data analyst is than what a software engineer gets. I think that certain companies will always be looking for people who can actually do traditional data science work from scratch because the automated tools will not apply to their specific use cases. I think these individuals will probably generally have a master's or PhD. I think right now the field is too new for managers to be able to tell the difference between what kind of Data Scientists they need. My experience has been a lot of companies need the former, but they think they need (or they want) the latter. I suspect in the coming decade these two roles will find different titles.
- compcoffee 8y ago
  >I have seen job postings for Data scientists that are simply rebranded data analysts to postings looking for advanced math and programming ability. Being as how "Data Science" is such a nebulous term, what is wrong with such a posting? I'd say that a data analyst with some advanced math and programming ability is a pretty rare skillset. Is it any surprise such a person is in high demand?
  - telchar 8y ago
    When looking for data science jobs it's nice to know from the prospective hire side that your skills will be used well. Coming into a job only to find out you're way overqualified can be a downer, particularly in a field where you need to keep your teeth sharp. I think the same goes for programming - imagine finding out on your first day you'll just be doing user testing and not writing any code.
DrNuke 8y ago
There is no gap, actually. It is plain, old supply vs demand. Second and third class firms wanting the few Einsteins out there for peanuts, being flooded with CVs from kaggle noobs and mooc kids. Point is the latter are good enough for the jobs on offer but the former feel somewhat entitled to be choosy & stingy, even if they have no real business case to pursue.
tixocloud 8y ago
There is some value in the additional rigorous data processing skill set. The intent is to get away from just merely slicing and dicing data to come up with some insight that explains causality. Whether we are afforded the time to do a proper and thorough analysis is a different matter.
citation_please 8y ago
I'm on a hiring committee for a job with the title of "Machine Learning Engineer". The most qualified candidates we have come through the process are physics PhDs who did a significant amount of coding during their PhD and have been able to segue into machine learning because they were interested. Our other tier of hireable candidates have been individuals with 4+ years of industry experience, usually with CS PhDs with a machine learning specialty. The rest of our team came from internal transfers, people who were less qualifed but proved that they would have a positive impact on the output of the team.
- drawnwren 8y ago
  I'm curious why you are seeing such strong candidates from the physics department. Care to elaborate?
  - analog31 8y ago
    I'm a physicist, and got my degree in the early 90s, but maybe I can shed some light on this. First, physics has been computation driven since the 1940s. When I was in grad school, programming was vital to my project. I wrote mountains of code, and also designed my own electronics. Second, I noticed a weird difference between physics and engineering. Engineering students were given problems that were expected to be solved within a particular domain of engineering. A physics student might be given a problem with no idea of how to solve it, much less even how to define the problem itself clearly. That was my project. So a physics student could find themselves having to learn practically any technical skill. Third, a matter of motivation. We knew that we would need to make ourselves employable. My project would occasionally fail in some spectacular way that would require me to learn more programming, or more electronics. Imagine that. ;-) Fourth, possibly also as a matter of employability, math and physics people have always figured out how to worm our way into ill-defined, nascent areas of technology. These are areas that have not yet created a mainstream training pipeline, so we can plausibly claim to be as well trained as anybody. "Embedded systems" was such a field when I was finishing school.
    - citation_please 8y ago
bertil 8y ago
As the hiring manager for that kind of profile for more than ten years, the problem is rather simple: they want data scientist, but they don’t have good data. The position are still open but no one well-informed would apply. Managers want to understand what’s happening in their company; they’ve been told data scientist can make that happen, or better, fix the problems automatically. They expect the solution to come without significant investment in data collection. One, even over-paid, data scientist is a far smaller budget than having your dozens of teams make an effort in fixing data collection, and setting up a reporting team. That kind of problem is easy to detect during the interview, if you’ve been there, and any data scientist has. What’s left are fresh graduates without diploma, who were taught statistics, but not how to get a project running -- so they don’t know what to say during an interview. In more details: Too much focus on stats. The actual skillset required should not be that demanding on statistical side: interviews generally focus on testing that exclusively. It takes the form of Whomever has any statistical background will grill on what they know best, their own MSc thesis. The thing is: no one can know the minutia of every model, certainly not a student’s interpretation of them, so those end up being a lot of “I would have to go back to the documentation.” Unless your interviewer is mature enough to either know that’s reasonable or stop doing that, it’s a lost cause. At best, you end up hiring a pile of people with the same focus, and every problem will look like a nail to a group of people carrying hammers. One alternative is to ask about something they did recently, but that’s rarely very instructive, either because the interviewer has no idea if that makes sense, or because the implementation has a lot of properties that make the whole process tedious. The focus should be the ability to learn new techniques at speed. Because the vast majority of the time, the right model is: a. a basic implementation of a better one, just to get things started (fix the data collection, the engineering of the input to the serving model, think about scaling the serving model, improve the actions taken from the decision, improve the decision format and the actual model objective) b. a model that you haven’t heard about, and you need to learn about it within a calendar that is conditioned by a. not being sufficient. Be more project-focused Description like “codes and does statistics better than either” fail to mention that the bulk of the time, a data scientist is actually more of a project manager, trying to identify blockers, get the integration of their tool prioritised, compare the impact and risk of different approaches; there’s also a big chunk (if not all of it) of being an analyst, generally to clarify what problem is being asked, or checking that the data is consistent. That’s if all is reasonably good: far too often, the time is spent trying to get management to grasp that Machine learning can only predict something, and that what to do with that, what to expose to the users, how, when, is more the remit of the product team i.e. you don’t have a real direction. Many large companies actively select against people willing to say that the kind is naked. Some company test for “business sensitivity” but that too often means checking that aspiring data scientist can walk through a P&L. Reality is closer to: Can you listen to a project manager, frustratingly oblivious to his own contradiction and general ass-holery, rant for 45 minutes without strangling him? Because you need the support of his team to get through. Fight for good data I have yet to see a company that had data you could use for data science on day 1. The vast majority of the time, it was inconsistent. Not a little bit, or hard to find: big, glaring, obvious contradictions like 20% of the revenue unaccounted for. If there are analysts, they generally have found a way around it, rather than address the issue. In more mature companies, there were some problem that would make basic model stumble, and more nuanced one go to very stupid conclusions. The classic case of a trip-up on good, but not well-structured: What drives retention? Completely failing to delivery your service: people will re-order within the hour, a larger ratio than any other case. Executives pay for the servers so they think that data quality if measured in peta-bytes. They get fed a continuous streamed of over-analysed data, filtered by hand from any inconsistency. They have no idea how inconsistent it really is. Any improvement is seen as a cost centre, rather than the ability for the company to be better informed. Don’t get me started on technical debt, short-term focus, insensitivity. What data scientists need, before they even join: clear objectives, clear breakdowns to understand what effects are correlated, clear plans to address some of the concern, pilot tests to make sure that those solutions would work, and collect some data, audited data pipeline with clear documentation to avoid misinterpretation, dedicated engineering to help with data logging, hosting computation, serving model. I am willing to bet that hardly any of those dozen of thousands of jobs looking for candidates don’t have most of that. There is a small piece of good news: Agrawal & al. summarised their theory on how ML will make predictions cheaper in a all-access book, Prediction Machines. The predictable success of that book will help. Basically, it tells about what ML does, without ever doing any ML at all: it just looks at how it’s a lynchpin technology, like search engine, digital photo, etc.

acbart 8y ago

I've had a few friends express interest in pivoting their domain knowledge + small programming knowledge into a Data Science job. Are there many success stories or guides out there for those who make such a transition? I went the formal route of getting the CS degree, so I never really thought about how you turn a non-CS degree into a CSy type job.

ThePhysicist 8y ago
Most data scientists I know don't have CS degrees but instead a background in physics, biology, math, economics or statistics. For a data science role focused around exploratory analysis, model building and idea generation (as opposed to a more engineering-focused data scientist) I'd say that a basic understanding of programming concepts and familiarity with the most essential tools (version control, Linux, Python/R) is probably enough to be accepted into a data science role. Data science degree programs are still quite novel so most companies don't expect to recruit people with such a degree, hence there are good opportunities for people with a STEM education I'd say.
- ekianjo 8y ago
  > Data science degree programs are still quite novel Indeed, plus you have zero business experience straight out of such a program so you are at a disadvantage if you are starting your career as such.
- bobmarley1 8y ago
  I've actually seen many many job postings looking for people with a PhD in Data Science which is truly mindblowing because there are only 2-3 such programs in the Country and most of them just started so none of their students are even on the job market yet.
  - PAClearner 8y ago
    ummm you are probably wrong. those job postings probably want people with PhDs in Computer Science/Statistics/Applied Math that did hard things with data.

TadaScientist 8y ago

You don't need to have a CS degree to be in "Data Scientist". In fact, I would argue that you need a statistics degree. There is a difference between a data scientist and an engineer.

disgruntledphd2 8y ago
The problem with stats degrees is twofold: 1) They often lack experience with messy (i.e. real) data 2) They sometimes lack experience with programming. Data science, as currently practiced requires some CS, some stats (with a focus on experimental design/observational analysis) and communication skills. These skills are rarely taught together in undergrad, but lots of PhD's acquire them by accident, hence the propensity of PhD's to end up in these roles.

pvaldes 8y ago

Maybe oral communication skills are overrated. Who knows? People currently talk a lot also with their hands.

tw1010 8y ago

I feel like I hear just as many people getting degrees in "AI" nowdays as plain CS degrees. A much higher percentage than five years ago. It feels like everyone and their grandma is minoring or majoring in AI or machine learning. And when I talk with companies about it, they're lukewarm at best. I really wouldn't bet my personal career at this point on going into this area. It'll be filled to the brim sooner than you expect. It's a much better, and just as intellectually satisfying, bet to just go into plain traditional CS in my opinion. (A counter argument is that I felt about the same sentiment around three years ago, but my intuitions doesn't appear to have manifested, so maybe I have poor intuitions, or I'm missing some piece of the puzzle.)

jakecrouch 8y ago

I view modern AI as a combination of things that are too early and too late - data science hasn't changed much since the 1990s, but people have a misconception that there is still a lot more to do in data science because of the undeserved excitement around deep learning.

sgt101 8y ago
Errm - I am pretty sure that we didn't have MCMC or causality analysis in the 90's. Or R-studio.
- johnmyleswhite 8y ago
  Perhaps I've misunderstood your desire to be sarcastic, but I think your timeline is off. To give two concrete examples, Geman and Geman's "Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images" paper is from 1984 and Holland's "Statistics and Causal Inference" paper is from 1986.

singingfish 8y ago

I'm available for the right data science gig. Somewhat unconventional CV available on request. I'll try to bring my competent but intellectually foggy business analyst colleague along for the ride if you like.

DoofusOfDeath 8y ago

It would be nice if the story touched on compensation levels. If all of those companies offered $400k/year, with guaranteed employment for 7 years, they would suddenly find no shortage of data scientists. Although other industries might suddenly wonder what happened to all their statisticians / physicists / etc. And two years later there would be a glut of data scientists who had just completed their master's degree.

wenc 8y ago
Not every labor shortage problem is solved by adjusting compensation levels upwards. The compensation has to match the value created.
- bobmarley1 8y ago
  Well not according to the people who constantly talk about Supply and Demand. If there's a massive supply shortage as implied there ought to be a corresponding increase in Compensation no?
- jlelonm 8y ago
  Why is that, economically speaking? If there's a labor shortage, doesn't that mean it's undervalued in the first place? Forgive me if that sounds naive, I'm no economist.
  - wenc 8y ago
    Because it makes the assumption that the system has no constraints, which is often untrue. First of all, you have to ask yourself why there is a labor shortage. If the underlying reason is insufficient pay, then sure, increasing pay will fix it. This is often the case for fungible talent -- incentivize something enough, and people will shift resources to it. However in many situations, certain types of work are not fungible. There are natural barriers to entry and qualification issues. Surgeons, for instance. You can incentivize and compensate all you want, but the fact is, not everyone is cut out to be a surgeon, so you have a funneling effect. It's not even a matter of pay. Then there's desirability issues. Deep sea welding is a highly specialized (and dangerous) trade that pays handsomely, but not everyone wants to do it. Some physically demanding jobs also have natural attrition issues in spite of compensation. When the inflow of talent is smaller than outflow over a long period of time, a shortage results. There's also a training and timing issues. Let's say it's 2011 you want someone who can build the kind of infrastructure that powers Netflix, in 6-12 months. The talent and experience pipeline would still have been brewing at the time, so there's going to be a temporary shortage of talent and experience until folks gain experience and mature in the field. In that instance, you can still hire and grow personnel over time, but there would still be a shortage of existing talent. Talent pipelines take time to build, are highly dependent on talent pools available and the ecosystems around it. Just throwing money at the problem doesn't always work. Compensation is just one end of it (the opportunity end); there are also significant long-term investments needed on the other end (the cradle end, which includes education, development, etc.). Texas Instruments did this -- they funded what eventually became UT Dallas... but it took many many years and the outcome was uncertain. There's geographic issues. To use an extremely unlikely example, let's say you wanted someone at the level of Jeff Dean or Sanjay Ghemawat, but you would need them to relocate to Podunk, Iowa. Very few people at that level would want to relocate for any amount of pay, hence a shortage. I realize this is a pretty extreme example, but I worked at a company in an undesirable part of the country and it was difficult to get truly talented folks to move out there even with significant premiums on compensation. Now you can keep increasing compensation until you get someone who's willing to move, but they're usually not the kind of talent you were looking for in the first place. Also there's a break-even point at which the compensation doesn't make sense for the value the position is likely to generate, so companies will just not hire. My point is there are all kinds of real and complex reasons why increasing compensation alone will not solve all shortage problems.

PeterStuer 8y ago

Two remarks from an old AI guy here. (1) If you're going to hop on the 'hot' AI/ML/DS bandwagon, make sure that you actually have 'work' for the highly coveted person you are trying to hire. 'Everyone else is doing it', 'all our clients have data, so selling them data-sciency projects shoudn't be a problem', 'Hire a first DS person, to have them start a brand new DS division' are things that probably are doomed to fail. If you weren't selling classic Data Analyses/Intelligence services before, chances are you are lacking the foundation in your sales and support staff, and maybe even your customer type, to be able to embed this into your company and into a new line of business. The risk your prize new employee will walk to greener horizons to work with peers on exiting real projects is high. (2) Know where you are heading. If you just want to embellish your 'normal' projects with some gimmicks from the online 'Cognitive Services' api's offered by the various vendors, then yes, sending some of your regular staff on a quick 'boot-camp' style course will probably work. You don't need a 'data-scientist' for that. And if you hire a real one for that type of work he'll probably walk within 2 weeks.

tixocloud 8y ago
Sage advice. There is a significant misconception on what a data scientist does and many roles don’t seem to truly bring out their potential. Most get grouped into doing everything around “data”.
maxxxxx 8y ago
Agreed. From what I can tell the data scientists at my company pretty much are running a few SQL queries with simple stats. They also seem to have a high turnover. It looks like they got hired mainly because the company wanted "data science" without knowing what it means.
amrrs 8y ago
That's a very good advice. That expectation and reality mismatch causes a lot of problems and that's also well captured in this article. https://towardsdatascience.com/why-so-many-data-scientists-are-leaving-their-jobs-a1f0329d7ea4 https://towardsdatascience.com/why-so-many-data-scientists-a...
apohn 8y ago

olivermarks 8y ago

I smell buzzword HR bingo

thanatropism 8y ago
Yeah. I find "data science" a cringey expression and I'm not looking for a job currently, so my LinkedIn emphasizes my strengths according to my ego/self-appraisal, not a "desperate" reaching out for fashionable job titles.

adiusmus 8y ago

Ah yes. A shortage. Someone didn’t do their planning. Bubble bubble tool and trouble. Or is this about more visas? And paying people less. Seriously though, what exactly is a data scientist? What is the requisite skill set? Is this masters level entry or would my feline require a Pointy Headed Degree upgrade to play this game?

adiusmus 8y ago
Data scientist buzzword compliance: Spark, Hadoop, Hive, Pig, SQL, Neo4J, MySQL, Python, R, Scala, Tensorflow, A/B Testing, NLP, anything machine learning
adiusmus 8y ago
Masters and PhD are required for particular employers. Others not so much but there are caveats. High probability of being the “data” guy who is then made to be the database admin. Better summary at: https://blog.insightdatascience.com/from-phd-to-data-scientist-17209c66a9d9 https://blog.insightdatascience.com/from-phd-to-data-scienti...

NatW 8y ago

I've heard it said, "A data scientist is a better statistician than a typical programmer and a better programmer than a typical statistician." I 'graduated' from the Data Analyst Nanodegree @ Udacity in my spare time, and have experience with both. I lack a background in math beyond university Calculus, though, so that seems the major barrier for me to get a foothold in the industry.

tixocloud 8y ago
It depends on how companies staff their data science teams. I’m comfortable with a lead data scientist that oversees “less math” data analysts who build the models. If you get the high level concepts of stats and linear algebra, it should be enough that someone else with a PhD can help fill the knowledge gap.
friday99 8y ago
The other way to phrase that is "A data scientist is a worse programmer than a typical programmer and a worse statistician than a typical statistician." If you need a programmer, hire a programmer. If you need a statistician, hire a statistician. I wouldn't ever hire someone who took a bootcamp that is "a better plumber than a typical doctor and a better doctor than a typical plumber."
- deleted 8y ago
  [deleted]

duchenne 8y ago

I think that the challenge to hire a good data scientist is that it requires skills in both: - hard science: strong statistics or ML to analyze data, some CS to organize the data. - soft skills: strong business sense to understand what is important, some communication skills to explain it. A data scientist needs very good skills and experience in both, but they are somehow anti-correlated in my experience.

tixocloud 8y ago
That’s why we usually have a team that complements each others’ skill set. The data scientist while great to have everything does not necessarily need to have everything. I tend to focus on their core strengths and depending on what their growth/career aspirations are, help to develop the skills. If it’s soft skills, happy to help them develop that edge.

citation_please 8y ago

As someone who has been producing value in a data science/machine learning role for multiple years, it's disheartening to see comments that I may be blacklisted from positions due to "only" having a bachelor's degree. Somewhat non-humbly, I was valedictorian at my high school, I triple-majored at a respectable Big 10 school, I actively use all 3 majors on a daily basis, in a foreign country, and sometimes in a language that is not my mother tongue (as an American). I can't justify spending time and money on a master's degree (millennial wealth problems) where many courses would just be putting a formal, academic spin on ideas that I'm familiar with from a practical business-value-producing point of view. Any advice on how I can effectively jump off the black-lists?

SOLAR_FIELDS 8y ago
The UG online masters of computer science is a good option. Many people in this site speak highly of the curriculum[1]. Last time I checked the total for the entire curricula if you pass every course on the first try is around $7000. As a bachelor holder myself looking to break into some of these higher-salary and in-depth roles I’m certainly considering it. It’s even better if you have a company that will pay for it, and because it’s so cheap even the most meager offerings from companies will cover a good portion of it. 1: https://news.ycombinator.com/item?id=15018002 https://news.ycombinator.com/item?id=15018002
- treyfitty 8y ago
  UG and GaTech are two different schools
  - madcaptenor 8y ago
    I'm not even sure what school "UG" refers to. (People generally call the University of Georgia "UGA".) In any case, the program here is from Georgia Tech.
- citation_please 8y ago
  Yeah I'm definitely familiar with the Georgia Tech offering. But my impression is that it will be a $7000 + $(my_hourly_rate) * (hours_spent) certificate that will only get me past the employers who have a "Select your highest degree level" drop-down on their application form. Is it that much more respected than a collection of MOOCs? As someone who's also involved in hiring, if I see no industry experience + GA tech online degree it's still on an entirely separate tier than 2 years of industry experience. But that's my bias I suppose, and part of the reason that I'm not the only one on the hiring committee.

itg 8y ago

In my experience, the title 'Data Scientist' has become to mean data analyst at most companies, meaning working with Excel, Tableau, and SQL. Maybe R if you're lucky. Companies doing ML/AI will usually have a small team of Research Scientists who mostly hold PhDs, and a team of supporting ML Engineers.

tixocloud 8y ago
I’d be more inclined to think that at a startup, because of the lack of infrastructure, a data scientist will do a lot more than research.
- mlthoughts2018 8y ago
  It’s even worse at established companies because data scientists still only do the data plumbing and simplistic analytics tasks, but not due to anything reasonable, like the “many hats” needs of a startup, but instead because of IT dysfunctiom and bureaucracy.
  - tixocloud 8y ago
    Think it depends but it does happen. We’ve been able to carve ourselves out of IT with our own infrastructure and our data scientists specifically focus on research and analysis. The only issue is when we try to get anything to prod and we hit IT.
    - mlthoughts2018 8y ago
      > “The only issue is when we try to get anything to prod and we hit IT.” Which inevitably means the one person on the data science team who is good with linux and docker suddenly becomes the IT wizard, and their time gets sucked up by having to find ways to go around utterly stupid barriers and tactics used by IT to avoid doing work to help you. Source: I am currently this person for my team.
      - deleted 8y ago

antt 8y ago

As someone who has been employed in this space recently: No there is not. There is however a lot of middle managers who have heard the term and want to pay crud vba developer wages for people to make pretty graphs in excel. The credential inflation is also getting ridiculous. I have an MSc and I don't see how me being able to do quantum chromodynamics at one point in my mid 20s adds value to a business ten years later.

collyw 8y ago
That's what I am thinking. Any half decent software engineer could probably produce the results they need from a few SQL queries. But "data science" is hot at the moment.
- telchar 8y ago
  That really depends. I'm working sometimes on problems that physician researchers have been cranking away on for decades. We want to do better than their state of the art. That's not something you can throw a software engineer at. There is a pretty wide spread of problem difficulty out there.
- dswalter 8y ago
  A half-decent software engineer can easily and quickly produce a lot of data products. If you divide the responsibilities of a data scientist into two broad categories: * Can I get the answer I need/build the product I need? * What is the question I should be answering? What do I need to build? Software engineers tend to do quite well at the former, but are not necessarily guaranteed to be successful at the latter.
  - collyw 8y ago
    In my experience its the software engineers bosses that are poor at the later.
- i-am-charmander 8y ago
  Comments like this reinforce the idea that data science is too broad a term. Everything data-related is thrown into the "data science" bucket. You describe what I'd call business intelligence or even advanced analytics. SQL jockeys (not a pejorative) really slicing into the data and presenting it in meaningful ways. There's usually exists some science behind these exercises. Then there's machine learning, which also gets clubbed into the data science camp. Try capturing a convolutional neural network in SQL - you might go bald from pulling your hair out. It's totally not the tool for the job. While perhaps any half-decent software engineer could maneuver his/her way around the great ML libraries out there to produce a CNN (or something of the like), I would hesitate to give them a project that involves true science (bring out your pitchforks). Experimental design, hypothesis testing, etc. While being a SQL expert could help turnaround time and enable you to test more interesting hypotheses, I wouldn't leave most DBAs to develop and carry out an experiment. So I guess my point is "data science" is too broad and overused a term to be meaningful.

tmalsburg2 8y ago

I'm teaching stats and scientific computing at university. I thought that data science might be a good exit plan in case I get feed up with academia. However, even though data scientists seem to be highly sought after in Germany (judging from what I see on linkedin), the salaries are fairly disappointing.

JustSomeNobody 8y ago

Any time I see businesses complaining about shortage of skilled labor, I read it to mean they aren't willing to pay enough to attract the skilled labor they want. The formula is simple. Businesses are in business to return as much money to investors as possible. IE: they're all about the money. Employees, more and more, are saying, "So am I. You want yours, so I want mine." Pay more and you'll attract more talent. Simple.

another-cuppa 8y ago

I was hired as a data scientist for a large company that thought they needed data scientists. They didn't and I added no value to the company. I left after about a year. If there are many people like me then that might explain the shortage.

Aeolun 8y ago

Maybe they’re all desperate for good data scientists? The truly good data scientists that I’ve found have actually been scientists. Literally all “data scientists” I’ve found in a corporate environment have been so useless it is depressing.

ccdsacc 8y ago
My experience is that most valuable data scientists are those who are good engineers.

madenine 8y ago

The market for serious data scientists is growing, but is not as dire as these articles make it seem. The private sector has wrapped 'senior data analyst','BI developer','data base administrator', and 'data engineer' into 'data scientist', along with the ML/AI/stats roles you would expect. So sure, when you put all the roles that touch data into one basket, and we're in a world that demands increasing familiarity with data systems and data analysis techniques... a gap is going to appear. Color me shocked Typical data scientist role I see posted is looking for: - SQL - AWS/Azure - Tableau/Power BI/Other BI tools - Python/R - Increasingly Hadoop/Spark With Java/Scala/C#/C++ listed as 'nice to haves' along with 'machine learning' and 'big data'. These are not research roles, and while they often want a masters degree or more they really don't need them. If you fit that bill you can probably get a role paying $95-120k just about anywhere in the US. You'll wrangle data, make pipelines, build dashboards and occasionally deploy a model or two when the planets align. If you want to/are capable of doing more ground breaking/cutting edge work, these roles will crush your soul in 8-12 months.

nabla9 8y ago
> Typical data scientist role I see posted is looking for: - SQL - AWS/Azure - Tableau/Power BI/Other BI tools - Python/R - Increasingly Hadoop/Spark They are looking for programmers. Unless... C programming makes you a hardware engineer, Fortran programming makes you a physicist, Ada programming makes you a rocket scientist, and Lisp programming makes you a god.
antt 8y ago
>If you fit that bill you can probably get a role paying $95-120k just about anywhere in the US. You'll wrangle data, make pipelines, build dashboards and occasionally deploy a model or two when the planets align. If you expect to pay that little to anyone who knows the above to any degree you will have a bad time. A number of projects I have joined absolutely look like someone treated them as a way to learn the technology before moving onto better paying jobs, leaving the business with a terrible mess that's largely unfixable without a rewrite. You're better off hiring one developer that knows what they are doing at above market rates than three who don't for the same price.

dostres 8y ago

I’d have a hard time finding a job in ML for $300K but I’d have an easy time finding a job in PHP contracting for $350K. This is despite having 10 years experience as a ML practitioner. I quit my last job after bringing in over $100M in profit and only getting a $30K bonus. Now it would take ~$500K and an ‘eat what you kill model’ for me to consider going back to work. These days I make my money from licensing ML products which gives me a lot of time to study - which is what I enjoy doing.

gcb0 8y ago
where do you find php contracting for that which lasts long enough to make 350 a year?
- dostres 8y ago
  Initially one of those elite consulting marketplace things. And then my own pool of referrals. I hate PHP so I set my rate higher on those projects and people still pay it.
  - j88439h84 8y ago
    What is an elite consulting marketplace? Where do I find one?
- deleted 8y ago
  [deleted]
tw1010 8y ago
Care to share a link to your licensed ML product, for those of us who're curious? Got any tips for people who are playing with the idea of doing something similar?
- dostres 8y ago
  It’s face recognition software, I don’t have a website, and for obvious reasons I’m pretty quiet about it. The field has changed so much that I don’t know if I could still give advice.

Max_Mustermann 8y ago

How do you see the chances of someone without a degree getting an entry-level, low-paid, remote job in this space? What would be things that could help out? Certifications? Nano-degrees like Udacitys? A portfolio of public work?

clu1590 8y ago
It's highly unlikely. There is a lot of competition for entry-level jobs and not many remote positions available. The barriers to entry are exactly academic qualifications over certifications and portfolios.
ellisv 8y ago
Remote job? Probably not without experience and/or connections. Portfolio is going to be the most helpful, followed by certifications and nano degrees.

En_gr_Student 8y ago

Who are you kidding? They aren't hungry. There is lots of propaganda flying around, but they aren't really even looking hard. HR games the system.