8 ms·

Big Data's Big Problem: Little Talent

162 points by Liu 14y ago

tomjen3 14y ago
Actually that is silly -- McKensey should now that there is and will never be a talent shortage. There will only be shortage of talent at a particular wage rate. If the companies paid newly graduated 'data-scientists' (what other kind of scientists are there? The tea-leaf reading kind?) 200k/year then they would have a lot more. It is pretty simple economics.
- throwaway1979 14y ago
  You're absolutely right. That said, I don't buy that big-data is as revolutionary as the Internet. While in theory, every single business can collect data and optimize based on what they see, this is way too complex for most businesses to deal with. While big data has certainly been critical for the business model of ad-based startups, I don't see it being used in other industries. People keep alluding to data-driven medicine and genetic analyses. These are some of the most complex information analyses domains, and yet, I don't see benefit commensurate with the big data hype. I'd love to hear counter examples though!
  - tgflynn 14y ago
    This technology will change the world more than the Internet or any other technology in human history. You're right that adoption is very slow. I'm convinced that businesses could save trillions of dollars by applying existing weak AI to their problems. Why aren't they doing it ? For one thing there's a huge gulf between the average business person's understanding of what is possible and what actually is. On the other hand the people who understand the technology don't have domain experience in various businesses. You can't develop solutions if you don't know what the problems are and it's very hard to guess at what economically relevant problems exist in fields you've never worked in. There are probably other barriers as well. Domain experts are unlikely to champion technologies that may, well, replace them. Bayesian networks were developed in academia 20 years ago that outperformed doctors at medical diagnosis. Why aren't they being applied ? There are probably many reasons but I suspect conscious or unconscious resistance on the part of the medical community plays a significant role. As for a talent shortage, I don't buy it. I'm exactly the sort of person this article talks about, with a strong mathematical background, excellent implementation skills and real world experience in developing and applying machine learning algorithms that have made millions of dollars for my former employers. I have had a website and a LinkedIn profile for over a year that make this fairly clear. How many consulting inquiries have I had ? Exactly zero.
    - zheng 14y ago
      Slightly off-topic, but your website breaks after visiting the RDMS page, as all the other links seem to be relative, so they attempt to go to pages such as /products/people.html
      - tgflynn 14y ago
        Sorry about that, thanks for letting me know. It should be fixed now.
    - csomar 14y ago
      I have had a website and a LinkedIn profile for over a year that make this fairly clear It doesn't work like that. You LinkedIn profile might easily land you any job in Software development, but not consulting. In my opinion, if you want to do consulting for big corp. you should figure out what it takes to it. An attractive website and presentation, few buzzwords, client testimonials, business cards, and the other blablabla. Yes, it's irrelevant (and shitty) to what you are actually doing, but that's actually the world of consulting.
      - tgflynn 14y ago
        That's definitely not my world and one reason I left big-corp in the first place. EDIT ADDED: It seems like a really broken market if buyer decisions are completely orthogonal to the product being purchased.
        tomjen3 14y ago
        I hope you are right. If you are, you have identified a potentially very, very lucrative option for a start-up. Broken markets can provide you a lot of money when you fix them.
        lgieron 14y ago
        This market is already (at least partially) covered by small consulting shops which provide sales front for competent freelancers who don't feel like doing the whole corporate networking&sales ritual.
- Tichy 14y ago
  Isn't that point of view also colored by ideology? Even supposing there are enough people capable of becoming that kind of "talent", what if those talents are also sought for in other kinds of jobs? Granted, if it were really urgent, perhaps companies would start looking in the most remote places for talents, so with a population of 6 billion perhaps there really would be enough who could be trained. How many of those 6 billions are "free" in a sense, as in not needed for maintenance of human life (farming, medicine, building shelter and so on)? But do economics really work that way? Could we extrapolate that logic to conclude that there is no problem in the world at all? All it takes is enough money to solve every problem - alas, the money doesn't seem to be there, or allocating it properly is apparently hard. (Hm, some of those talents might be able to help, for a true bootstrap solution).
  - NyxWulf 14y ago
    It's not that you can solve every problem, but supply and demand are very real. If the wage rates for Big Data get high enough more and more people will try out the field, which will generate a larger supply. The reasons companies don't just throw out huge salaries though has to do with the demand side. The salaries companies are willing to pay is related to the marginal advantage they can gain from hiring someone with that skillset. If for example a company will gain say 200k per year in total advantage, that would place a hard cap on how much they would be willing to pay in salary. So if the advantage is very high, companies will pay more. If the supply increases sufficiently wage rates will drop because there is over supply. If the supply doesn't increase enough, wages will increase more - however each company will drop out at it's own value point. This provides the natural limit to where most salaries cap out.
  - tomjen3 14y ago
    If those talents are also sought after for other jobs then the price will go up until one of the jobs will be done by some other method or some other person. I do have trouble imagining that anybody who is working on a farm would be a good data-scientist but then I no very, very little about farming. Economics is not tainted or colored by ideology, it is a science. It is the study of how best to allocate limited resources that have multiple conflicting uses. In this world there is nothing that is free, everything comes with some price. As long as there is a human want that is not fulfilled, there is no additional humans. That isn't necessarily bad though. You can charge societies progress to how few people are required to provide food to the rest. Once most Americans worked in argriculture, now only a few do. That is a good thing, because the rest of us can the do something else and satisfy some other human want. And the remaining farmers are better of too, since they don't have to work as hard and have things like tvs and computers.
- yummyfajitas 14y ago
  Companies already do pay close to $200k/year for entry level data scientists. (what other kind of scientists are there? The tea-leaf reading kind?) "Data scientist" refers to the guy who can set up a hadoop cluster, do statistics on TBs worth of data, derive useful conclusions and speed it up by tweaking the low level data formats or microoptimizing the calculation. The issue is rarely paying these guys an extra $20k, it's simply finding them. Setting up some lasers and a photonic crystal, imaging the output, making a graph in excel or matlab and drawing conclusions is a different skillset. Someone who can do the latter is a scientist who uses data, but he is not a data scientist.
  - Tichy 14y ago
    How hard can it be, though? Like taking a normal CS person and making them versatile with hadoop and so on? Could it be done for 20K$?
    - pnathan 14y ago
      > do statistics on TBs worth of data, derive useful conclusions That's gonna be the hard part. Most CS people I've met flee from math and, more generically, theory.
    - yummyfajitas 14y ago
      Making a CS person versatile with hadoop is not that hard. Making a CS person versatile with statistics is much harder. See Zed Shaw's seminal article "Programmers Need To Learn Statistics Or I Will Kill Them All". http://www.zedshaw.com/essays/programmer_stats.html http://www.zedshaw.com/essays/programmer_stats.html Making a math/science person versatile in CS is somewhat easier, but even that can be tricky. Many of them are bored by file formats, architecture, etc, and simply don't have the mindset of of engineering.
    - achompas 14y ago
      How hard can it be? Very hard. You run into all types of candidates who just aren't there yet: people working on research that's irrelevant to real world applications, people who have done data analysis/BI work that brand themselves as "data scientists," those who have the pedigree but cannot process and explore real-world data, those who have good analytical chops but not the distributed or advanced modeling experience, etc. I've witnessed it first-hand, and it's tough to find the right person.
- jandrewrogers 14y ago
  Bags of money are already being waved around, that is not the problem. Wages are already moving north of $200k for these positions because you can't find people with the basic skills for any amount of money. Being a "data scientist" as currently defined in practice requires someone to be a polymath with skills that are individually high value and not commonly found together. Roughly speaking, you need some aptitude and experience in the following areas: - mathematics, particularly statistics, computational geometry, machine learning, and probability theory - parallel algorithm design, something for which most software engineers have no skill - database ETL processes, formerly a highly specialized discipline only found in the database administration world You can learn the mathematics in school or with some study. Most software engineers never develop a knack for parallel algorithm design even when they try e.g. virtually all software engineers who claim to know parallel algorithms can't explain why hash joins do not parallelize well. Lastly, ETL is something that isn't normally found mixed with the other two but which usually requires some significant experience to do correctly. Even if you are a master of mathematics and parallel algorithms, ETL skills are something you usually learn by apprenticing with someone who is an ETL master for a couple years. Finding people that even have basic levels of skill at all three of these things is very difficult even if you loosen the criteria significantly. Unlike some other tech job fads, you can't mint a crop of data scientists in a year. When I look at the junior level data scientists we trained internally with great basic skills out of school, it has taken years to develop them. This level of effort and length of time is the real bottleneck.
  - ced 14y ago
    Computational geometry??? That's a new one for me. Do you mean only linear/convex programming? Incidentally, I would really like to hear about the kind of Real Work that data scientists end up doing with TBs of data, because I'm always fuzzy on the details. MCMC? Variational methods? SVMs? Or is it more oriented towards frequentist statistical methods, applied at "web-scale"?
    - jandrewrogers 14y ago
      I mean actual computational geometry. Reality is significantly non-Euclidean in complicated ways that have to be accounted for if precision matters. Spatio-temporal analytics or the processing of sensing data frequently requires this. For a simple example, the surface of the Earth is approximately an oblate spheroidal surface, not even a 2-sphere. You can use Euclidean approximations for many cartographic purposes but for analytics this can introduce large errors in the analysis. Understanding how to compute non-Euclidean geometry models is surprisingly useful.
_delirium 14y ago
I'm not sure that the kinds of employees that this article describes will ever be a large number. There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those. Companies might consider looking at better ways to build teams that combine talent that exists, instead of pining for more superstars. I'm reminded indirectly of an acquaintance of mine who works on repairing industrial machinery, where companies complain of a big skills shortage. They either fail to realize or are in denial about what that means in the 21st century, though. It might've been a one-person job in the 1950s, a skilled-labor type of repairman job. But today they want to find one person who can do the physical work (welding, etc.), EE type work, embedded-systems programming (and possibly reverse engineering), application-level programming to hook things up to their network, etc. Some of these people exist, but it's more common to find boutique consulting firms with 3-person teams of EE/CE/machinist or some such permutation. But companies balk at paying consulting fees equivalent to three professional salaries for something they think "should" be doable by one person with a magical combination of skills, who will work for maybe $80k. So they complain that there is a shortage of people who can repair truck scales (for example).
- disgruntledphd2 14y ago
  I completely take most of your points, but I think that pretty much all quantitative PhD's are going to be close to "data scientists". Given that stats and explaining your research are requirements, all that's left is to train them to program, which a lot of people are already doing. As a matter of fact, since I heard about this big data stuff I've been honing my skills in this area, in case the hype actually manifests.
  - tgflynn 14y ago
    Anyone who can earn a PhD can learn to program but being good at engineering the very complex processes that are needed for effective machine learning applications is a skill that is not so easily acquired. I've worked with quite a number of quantitative PhD level people in my career and most often the quality of their code leaves much to be desired.
mjw 14y ago
As an engineer who's investing in developing "deep expertise in statistics and machine learning" I can only stand to benefit from it, but something about the current wave of Big Data hype makes me instinctively a bit wary. Does this skills shortage really exist to the extent claimed? are there really enough people out there who would know what to do with a 'data scientist' if they were able to hire one? I see more talk than action, I see vendors circling around looking to flog freshly-buzzword-compliant BI tools, prognosticators trying to push nervous businesses into engaging in an arms race over data. Of course there's real value there too, for some at least. I hope my concerns prove unfounded, but worth retaining a healthy skepticism I feel :-)
- rch 14y ago
  At a conference I attended last month, one of the keynotes estimated that there might be 250 people in the country with the skills need to build non-trivial, ontology-based data systems. Even if that is an wild exaggeration, it is at least evidence of a perceived shortage. Also note that an ability to transfer domain experts' knowledge into working models is at least as important as the Stats+ML bits.
  - PaulHoule 14y ago
    I'd say the current academic research in ML is not oriented towards producing people who can use ML in real applications. I've hovered around the periphery of a world-leading ML research group, and the first takeaway I have is that 7 years ago I thought the stuff they were working on was going to take the world by storm, but looking back, I can say it hasn't. This group does a number of research projects on narrowly defined topics. 4 out of 5 of these projects try out some refinement of the method that doesn't really work. Maybe 1 out of 5, if that, point to a real improvement. The big thing that's lacking are serious attempts to push the state of the art by attacking a problem holistically and "taking no prisoners" -- yet this is exactly the kind of thinking necessary to commercialize ML. The leader of the group got tenure so he thinks everything is going OK. He won't even offer an analysis of why this technology hasn't been widely commercialized. PhD students from this group usually interview at Google, Microsoft and Facebook but these three employers are the only ones they consider as an alternative to academic employment.
harscoat 14y ago
Surprised Ben Rooney did not mention IBM acquisition of Vivisimo the day before (Apri. 25) this article (Apr.26) http://www-03.ibm.com/press/us/en/pressrelease/37491.wss http://www-03.ibm.com/press/us/en/pressrelease/37491.wss "IBM Advances Big Data Analytics with Acquisition of Vivisimo"
sandee 14y ago
Basically a solution looking for a problem. They are right, the complexity that big data caters for requires expertise at both technical and business level that would be costly (though may not be at infrastructure level). In the current economy, it looks even more difficult where businesses want to squeeze the maximum out of dollar investment. IMO, its too early stage for big data solution adoption. However stage could be set for startups who can come up innovative solution that brings the cost level down together with simple and useful easy to grasp solutions.
maeon3 14y ago
Why don't the people who hire doctors, dentists, and lawyers suffer from the same talent shortage that the people who hire 'big data' computer scientists feel? Because it's better across the board to start your own startup than work your ass off for a 4% raise at a place which recognizes you as top talent. I'm on the verge of starting a startup myself, removing myself from the people in this list. There is a shortage of talent in computer science, but never in the other disciplines, it may take another 30 years for the suits to have the ability to understand why.
- enfilade 14y ago
  I agree with your overall line of thinking. In your last sentence you wrote: "There is a shortage of talent in computer science, but never in the other disciplines, it may take another 30 years for the suits to have the ability to understand why." Could you elaborate on this point -- do you feel that there is not a shortage in disciplines such as dentistry and law because many people are willing to work very hard for only 4% raises? Thank you!
  - tomjen3 14y ago
    You can actually make good money by going into the law or medicine. You have to work and and be skilled of course, but lets be honest that is also required for a start-up. I can't help but think that ability is because you can't run a law firm without being a lawyer so the boss has some idea of what it means to be a good lawyer and how to treat them.
    - anothermachine 14y ago
      Most lawyers at the top-income end hate their bosses and jobs. Law firm partnership track at large firms is a dog-eat-dog 80-hour week hell. People are only happy when they are the few who claw to the top, or the many who drop out. The ones in the middle are suffering as bad as any stereotypical bank programmer.
chintan 14y ago
How media sees Big Data: BIG database => BIG machine learning algo => BIG MODELS => PREDICTIONs, Insights => $$$ How it is actually done: awk -F"\|" '{print $1}' SCRAPED_file_pipe.txt | sort | uniq -c | head -n 10 => $$$
- gaius 14y ago
  Ah, you've used Ab Initio then.
giardini 14y ago
Looks like the prelude to yet another H1-B buildup.
- alecco 14y ago
  Only this time US is less attractive. Even mexicans started to move out.
- tosseraccount 14y ago
  The only way we can really measure "shortage" is via compensation. If that's so, we need more hedge fund managers and surgeons, not grunt data crunchers. There are many problems with guest workers. The richest people in the world get special access to indentured labor. It targets specific industries thus amounting to a subsidy. It helps big business crush small business. The H1-B in particular is a tool to increase outsourcing and keep wages down (wages which are typically earned in the highest cost of living areas in the country at 60 hours a week). H1-B ? No, thank you! On the job training and good wages? Yes, please!
ambiate 14y ago
I just listened to a lecture on this at BU. Emerging Internet Technologies at IBM or something of that nature. He was basically trying to sell us his product that crawled the internet (mainly a firehose at Twitter) and gathered statistics for advertisers and presented it in pretty graphics. The main issue they had was developing language recognition. Deciding if a user 'liked', 'loved', 'hated' or was 'neutral' about a product. Another issue that stood out to me had to do with their reliance on the internet. Just because 200 users tweeted that 'this movie is going to suck' does not really represent the overall opinion. To reiterate, the whole buzz of the lecture was the biggest turn off. He wasn't explaining about how to expand on his product or where to go from here. Just that they had developed a product and we could use it instead of attempting to develop one ourselves.
- ohashi 14y ago
  Maybe 200 is too small a sample size, but you can glean and predict stuff with that sort of data. My master's thesis was about predicting box office sales based off twitter data. (coincidentally these guys published a couple months before me: http://www.fastcompany.com/1604125/twitter-predicts-box-office-sales-better-than-anything-else http://www.fastcompany.com/1604125/twitter-predicts-box-offi...) but we had incredibly similar results. It is pretty interesting to see what you can do with that data. If it makes you feel any better, you can build it yourself, I certainly did. I didn't even use any libraries like NLTK to build my sentiment analysis. Read some research papers and built it from scratch (code wise at least, the ideas used were fairly common). It's a fun challenge. I still work with that code every day and use it in my startup now :)
deleted 14y ago
[deleted]
chrisrhoden 14y ago
Can anyone explain what talent refers to in this context? Is it someone who has learned this stuff, someone who is capable of learning it, or someone who was born with an innate understanding?
gambler 14y ago
Managers frequently wail about skill shortages, but very often it's pure hypocrisy. The real problem is the reluctance to do any training (and I don't mean formal training) combined with the desire to get proven experts in whatever field. Proven experts must have years of experience in applying their expertise. If no one lets people with less experience to work in that field, where the hell would those experts appear from? Another dimension? Can I be a 80% developer and 20% "data scientist" in your company to try the new role out? The bigger your company is, the less likely the answer to be a "yes". Since Big Data implies a big company, the resulting "shortage" is not surprising. It's self-made.
- ImprovedSilence 14y ago
  True statement. It seems the trend now days is "get a grad degree, foot the bill and time yourself". Many companies I see and work with tend towards that mentality, as opposed to building a base of highly skilled workers on from the inside. It's easier and cheaper for a company to ask if you have a piece of paper, than for them to train you and get you up to speed.
- paulsutter 14y ago
  Companies looking for "proven experts" are working on yesterday's problem. Find a better employer. Move somewhere with more choices if you need to.
paulsutter 14y ago
This article is nonsense. Talented developers are talented developers. At Quantcast we used Hadoop in production before it was even called Hadoop and now we process 10PB a day. We forbid our sourcers from using Hadoop as a resume search term because it meant absolutely nothing. Statisticians who can code are scarce, but companies that know how to use them are scarcer.
why-el 14y ago
So I was wondering if any fellow HNer is on a quest to be at least comfortable around these problems. Can you share your plans? Currently I am starting with some linear algebra and I have plans to move to statistics then pick up a book on machine learning. I would really use some advice.
- achompas 14y ago
  I'm on this path right now. Working simultaneously on a MSCS at NYU and a full-time developer gig at Knewton (a company which truly understands the value and risks of data R&D). I'd really start with this awesome curriculum[0] by Joseph Misti. He nails the mix of modeling, algorithms, math/stats, development, and distributed/Unix skills one needs to become comfortable around these problems. More importantly, his advice agrees with my experience on the data team at Knewton--we really use a bit of all of the above skills to solve our problems. My email is in my profile if you (or anyone else) would like to chat about this a bit more. I'm also in NYC, and totally willing to grab tea and chat in person. [0] http://www.quora.com/What-skills-are-needed-for-machine-learning-jobs http://www.quora.com/What-skills-are-needed-for-machine-lear...
- seanharnett 14y ago
  Andrew Ng's machine learning class on coursera is a very nice, easy introduction to the subject.
  - why-el 14y ago
    Yes, and I started that before realizing that I need more background knowledge, hence starting with some math. :)
pmb 14y ago
"claims of severe talent shortage in Big Data http://online.wsj.com/article/SB10001424052702304723304577365700368073674.html http://online.wsj.com/article/SB1000142405270230472330457736... Ok... where are the high salaries (500k$ a year)? No? No real shortage." https://twitter.com/#!/lemire/status/196245665951649793 https://twitter.com/#!/lemire/status/196245665951649793 Business has a shortage of "big data" folks in much the same way I have a "huge sailboat" shortage. Neither of us want to pay for it. We want it, but not for the going rate. Only one of us has a media platform, though.
- jandrewrogers 14y ago
  The salaries are already moving north of $200k even outside of Silicon Valley and New York City and getting more expensive by the month. How high do they have to be before we have a "shortage"? The problem is not lack of money, it is that demand has greatly outstripped a finite supply. Very high wages do not automagically create new people with the requisite skills and this is the real bottleneck. It takes significant aptitude and years of training/experience to become useful as a "data scientist". It is not as easy as I think people are imagining. We train people with excellent raw skills where I work, usually strong applied mathematics backgrounds with natural programming skills. It is much easier than trying to find someone outside with these skills, though we do attempt outside recruitment. It still takes years to develop the people we train into a good, basic data scientist.
  - bearmf 14y ago
    Look, this job title is at most 2 years old. How can someone have years of experience in this? OTOH, there are plenty of people with strong applied math and good programming skills.
    - jandrewrogers 14y ago
      The set of skills existed before it had a trendy job title so you can have the experience even if it was called something else. This is true of most of the people currently working as data scientists. In a similar vein, I was designing big data systems years before "big data" became a term or trendy. For any particular odd skill mix you can come up with, there are people with that skill mix who are already doing a similar job. But usually people do not intentionally build that skill mix until it becomes an official job title and career path in the eyes of the public so it is a very small pool of people. In the case of modern data scientists, having strong applied mathematics and programming skills is about halfway to where you need to be and a good starting point. The demand has temporarily grown much faster than the convertible talent pool can develop the additional set of skills required.
kylemaxwell 14y ago
Not entirely sure this is true, to be honest. Most of "data science" lies in the work of collecting and cleaning the data to get it into a usable state. A recent story on the"fallacy of the data scientist shortage"[1] goes into more detail, but in reality what we want in this quantity are better data analysts. I love the idea of data science as, essentially, viewing statistical analysis from a computer science perspective, but the breathless predictions of a huge shortage seem a little overblown. [1]: http://smartdatacollective.com/nraden/48952/fallacy-data-scientist-shortage http://smartdatacollective.com/nraden/48952/fallacy-data-sci...
tlogan 14y ago
It seems the problem is that some companies are looking for person who is expert in setting up scalable systems (Hadoop cluster, storage, high availability, etc.) and that she/he also knows statistics and efficient ways of processing and understanding the data. Good luck with that. My observation is that requirements like this come from people who did mainly web programing (and actually that was making a lot of money so with money they become influential): assuming that this equivalent of writing both ruby code and javascript code. Building team is hard and in order to solve "big data" problem you need to build a balanced team.
thornad 14y ago
I've done this kind of thing most of my career, including doing it for NASA and Unilever Research. You can't really train an average graduate to do this. You need someone with a pretty highly developed integration between 1:intuitive/creative abilities, 2:mathematical/analytical skills, and 3:engineering/ability to make things happen. Add to that 4:work experience in the real world, and 5:ability to easily understand how things work in a field you delve into for the first time... And there's very few people in the world who can do this. At my previous work place we tried for a whole year to hire someone who would at have at least some of these skills and seems promising to develop the rest on the job. We couldn't find anyone although we interviewed about 30 different people (from about 500 resumes most of them with a PhD in ML from a good university). And this was in central London, UK.
- bearmf 14y ago
  But you never tried training them. Sure, no one can do it right off the bat, without prior experience.
mardack 14y ago
This is good news.
radikalus 14y ago
Until the pay is comparable to finance, good luck? I'd love to work on (arguably) cooler problems, but the combination of lower pay and the constant need to use the "hot new thing" to solve problems doesn't make transitioning look remotely attractive. Really, the second is the HUGE obstacle: - You don't know anything about aNNs? Sorry, no job. - Nobody uses aNNs anymore, SVMs are all that matters. Sorry, come back after you catch up. - SVMs? Man, we need someone who's got expertise in optimizing RFs and Bayesian Trees. We don't want "black box" machine learning. We need to "understand" the results. Sorry no job. - Decision trees? GTFO man. We're doing rNNs now. - I'm pretty impressed with your data mining knowledge, but we're looking for someone with a background in DLMs and GPs. Sorry, no job. - repeat until vomit/suicide I kind of wonder about the need for "badass" math skills; I'm not terribly convinced that math wizards are extraordinarily high value relative to people with other types of data analysis skills.

162 points by Liu 14y ago

Actually that is silly -- McKensey should now that there is and will never be a talent shortage. There will only be shortage of talent at a particular wage rate. If the companies paid newly graduated 'data-scientists' (what other kind of scientists are there? The tea-leaf reading kind?) 200k/year then they would have a lot more. It is pretty simple economics.

throwaway1979 14y ago
You're absolutely right. That said, I don't buy that big-data is as revolutionary as the Internet. While in theory, every single business can collect data and optimize based on what they see, this is way too complex for most businesses to deal with. While big data has certainly been critical for the business model of ad-based startups, I don't see it being used in other industries. People keep alluding to data-driven medicine and genetic analyses. These are some of the most complex information analyses domains, and yet, I don't see benefit commensurate with the big data hype. I'd love to hear counter examples though!
- tgflynn 14y ago
  This technology will change the world more than the Internet or any other technology in human history. You're right that adoption is very slow. I'm convinced that businesses could save trillions of dollars by applying existing weak AI to their problems. Why aren't they doing it ? For one thing there's a huge gulf between the average business person's understanding of what is possible and what actually is. On the other hand the people who understand the technology don't have domain experience in various businesses. You can't develop solutions if you don't know what the problems are and it's very hard to guess at what economically relevant problems exist in fields you've never worked in. There are probably other barriers as well. Domain experts are unlikely to champion technologies that may, well, replace them. Bayesian networks were developed in academia 20 years ago that outperformed doctors at medical diagnosis. Why aren't they being applied ? There are probably many reasons but I suspect conscious or unconscious resistance on the part of the medical community plays a significant role. As for a talent shortage, I don't buy it. I'm exactly the sort of person this article talks about, with a strong mathematical background, excellent implementation skills and real world experience in developing and applying machine learning algorithms that have made millions of dollars for my former employers. I have had a website and a LinkedIn profile for over a year that make this fairly clear. How many consulting inquiries have I had ? Exactly zero.
  - zheng 14y ago
    Slightly off-topic, but your website breaks after visiting the RDMS page, as all the other links seem to be relative, so they attempt to go to pages such as /products/people.html
    - tgflynn 14y ago
      Sorry about that, thanks for letting me know. It should be fixed now.
  - csomar 14y ago
    I have had a website and a LinkedIn profile for over a year that make this fairly clear It doesn't work like that. You LinkedIn profile might easily land you any job in Software development, but not consulting. In my opinion, if you want to do consulting for big corp. you should figure out what it takes to it. An attractive website and presentation, few buzzwords, client testimonials, business cards, and the other blablabla. Yes, it's irrelevant (and shitty) to what you are actually doing, but that's actually the world of consulting.
    - tgflynn 14y ago
      That's definitely not my world and one reason I left big-corp in the first place. EDIT ADDED: It seems like a really broken market if buyer decisions are completely orthogonal to the product being purchased.
      - tomjen3 14y ago
        I hope you are right. If you are, you have identified a potentially very, very lucrative option for a start-up. Broken markets can provide you a lot of money when you fix them.
        lgieron 14y ago
        This market is already (at least partially) covered by small consulting shops which provide sales front for competent freelancers who don't feel like doing the whole corporate networking&sales ritual.
Tichy 14y ago
Isn't that point of view also colored by ideology? Even supposing there are enough people capable of becoming that kind of "talent", what if those talents are also sought for in other kinds of jobs? Granted, if it were really urgent, perhaps companies would start looking in the most remote places for talents, so with a population of 6 billion perhaps there really would be enough who could be trained. How many of those 6 billions are "free" in a sense, as in not needed for maintenance of human life (farming, medicine, building shelter and so on)? But do economics really work that way? Could we extrapolate that logic to conclude that there is no problem in the world at all? All it takes is enough money to solve every problem - alas, the money doesn't seem to be there, or allocating it properly is apparently hard. (Hm, some of those talents might be able to help, for a true bootstrap solution).
- NyxWulf 14y ago
  It's not that you can solve every problem, but supply and demand are very real. If the wage rates for Big Data get high enough more and more people will try out the field, which will generate a larger supply. The reasons companies don't just throw out huge salaries though has to do with the demand side. The salaries companies are willing to pay is related to the marginal advantage they can gain from hiring someone with that skillset. If for example a company will gain say 200k per year in total advantage, that would place a hard cap on how much they would be willing to pay in salary. So if the advantage is very high, companies will pay more. If the supply increases sufficiently wage rates will drop because there is over supply. If the supply doesn't increase enough, wages will increase more - however each company will drop out at it's own value point. This provides the natural limit to where most salaries cap out.
- tomjen3 14y ago
  If those talents are also sought after for other jobs then the price will go up until one of the jobs will be done by some other method or some other person. I do have trouble imagining that anybody who is working on a farm would be a good data-scientist but then I no very, very little about farming. Economics is not tainted or colored by ideology, it is a science. It is the study of how best to allocate limited resources that have multiple conflicting uses. In this world there is nothing that is free, everything comes with some price. As long as there is a human want that is not fulfilled, there is no additional humans. That isn't necessarily bad though. You can charge societies progress to how few people are required to provide food to the rest. Once most Americans worked in argriculture, now only a few do. That is a good thing, because the rest of us can the do something else and satisfy some other human want. And the remaining farmers are better of too, since they don't have to work as hard and have things like tvs and computers.
yummyfajitas 14y ago
Companies already do pay close to $200k/year for entry level data scientists. (what other kind of scientists are there? The tea-leaf reading kind?) "Data scientist" refers to the guy who can set up a hadoop cluster, do statistics on TBs worth of data, derive useful conclusions and speed it up by tweaking the low level data formats or microoptimizing the calculation. The issue is rarely paying these guys an extra $20k, it's simply finding them. Setting up some lasers and a photonic crystal, imaging the output, making a graph in excel or matlab and drawing conclusions is a different skillset. Someone who can do the latter is a scientist who uses data, but he is not a data scientist.
- Tichy 14y ago
  How hard can it be, though? Like taking a normal CS person and making them versatile with hadoop and so on? Could it be done for 20K$?
  - pnathan 14y ago
    > do statistics on TBs worth of data, derive useful conclusions That's gonna be the hard part. Most CS people I've met flee from math and, more generically, theory.
  - yummyfajitas 14y ago
    Making a CS person versatile with hadoop is not that hard. Making a CS person versatile with statistics is much harder. See Zed Shaw's seminal article "Programmers Need To Learn Statistics Or I Will Kill Them All". http://www.zedshaw.com/essays/programmer_stats.html http://www.zedshaw.com/essays/programmer_stats.html Making a math/science person versatile in CS is somewhat easier, but even that can be tricky. Many of them are bored by file formats, architecture, etc, and simply don't have the mindset of of engineering.
  - achompas 14y ago
    How hard can it be? Very hard. You run into all types of candidates who just aren't there yet: people working on research that's irrelevant to real world applications, people who have done data analysis/BI work that brand themselves as "data scientists," those who have the pedigree but cannot process and explore real-world data, those who have good analytical chops but not the distributed or advanced modeling experience, etc. I've witnessed it first-hand, and it's tough to find the right person.
jandrewrogers 14y ago
Bags of money are already being waved around, that is not the problem. Wages are already moving north of $200k for these positions because you can't find people with the basic skills for any amount of money. Being a "data scientist" as currently defined in practice requires someone to be a polymath with skills that are individually high value and not commonly found together. Roughly speaking, you need some aptitude and experience in the following areas: - mathematics, particularly statistics, computational geometry, machine learning, and probability theory - parallel algorithm design, something for which most software engineers have no skill - database ETL processes, formerly a highly specialized discipline only found in the database administration world You can learn the mathematics in school or with some study. Most software engineers never develop a knack for parallel algorithm design even when they try e.g. virtually all software engineers who claim to know parallel algorithms can't explain why hash joins do not parallelize well. Lastly, ETL is something that isn't normally found mixed with the other two but which usually requires some significant experience to do correctly. Even if you are a master of mathematics and parallel algorithms, ETL skills are something you usually learn by apprenticing with someone who is an ETL master for a couple years. Finding people that even have basic levels of skill at all three of these things is very difficult even if you loosen the criteria significantly. Unlike some other tech job fads, you can't mint a crop of data scientists in a year. When I look at the junior level data scientists we trained internally with great basic skills out of school, it has taken years to develop them. This level of effort and length of time is the real bottleneck.
- ced 14y ago
  Computational geometry??? That's a new one for me. Do you mean only linear/convex programming? Incidentally, I would really like to hear about the kind of Real Work that data scientists end up doing with TBs of data, because I'm always fuzzy on the details. MCMC? Variational methods? SVMs? Or is it more oriented towards frequentist statistical methods, applied at "web-scale"?
  - jandrewrogers 14y ago
    I mean actual computational geometry. Reality is significantly non-Euclidean in complicated ways that have to be accounted for if precision matters. Spatio-temporal analytics or the processing of sensing data frequently requires this. For a simple example, the surface of the Earth is approximately an oblate spheroidal surface, not even a 2-sphere. You can use Euclidean approximations for many cartographic purposes but for analytics this can introduce large errors in the analysis. Understanding how to compute non-Euclidean geometry models is surprisingly useful.

_delirium 14y ago

I'm not sure that the kinds of employees that this article describes will ever be a large number. There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those. Companies might consider looking at better ways to build teams that combine talent that exists, instead of pining for more superstars. I'm reminded indirectly of an acquaintance of mine who works on repairing industrial machinery, where companies complain of a big skills shortage. They either fail to realize or are in denial about what that means in the 21st century, though. It might've been a one-person job in the 1950s, a skilled-labor type of repairman job. But today they want to find one person who can do the physical work (welding, etc.), EE type work, embedded-systems programming (and possibly reverse engineering), application-level programming to hook things up to their network, etc. Some of these people exist, but it's more common to find boutique consulting firms with 3-person teams of EE/CE/machinist or some such permutation. But companies balk at paying consulting fees equivalent to three professional salaries for something they think "should" be doable by one person with a magical combination of skills, who will work for maybe $80k. So they complain that there is a shortage of people who can repair truck scales (for example).

disgruntledphd2 14y ago
I completely take most of your points, but I think that pretty much all quantitative PhD's are going to be close to "data scientists". Given that stats and explaining your research are requirements, all that's left is to train them to program, which a lot of people are already doing. As a matter of fact, since I heard about this big data stuff I've been honing my skills in this area, in case the hype actually manifests.
- tgflynn 14y ago
  Anyone who can earn a PhD can learn to program but being good at engineering the very complex processes that are needed for effective machine learning applications is a skill that is not so easily acquired. I've worked with quite a number of quantitative PhD level people in my career and most often the quality of their code leaves much to be desired.

mjw 14y ago

As an engineer who's investing in developing "deep expertise in statistics and machine learning" I can only stand to benefit from it, but something about the current wave of Big Data hype makes me instinctively a bit wary. Does this skills shortage really exist to the extent claimed? are there really enough people out there who would know what to do with a 'data scientist' if they were able to hire one? I see more talk than action, I see vendors circling around looking to flog freshly-buzzword-compliant BI tools, prognosticators trying to push nervous businesses into engaging in an arms race over data. Of course there's real value there too, for some at least. I hope my concerns prove unfounded, but worth retaining a healthy skepticism I feel :-)

rch 14y ago
At a conference I attended last month, one of the keynotes estimated that there might be 250 people in the country with the skills need to build non-trivial, ontology-based data systems. Even if that is an wild exaggeration, it is at least evidence of a perceived shortage. Also note that an ability to transfer domain experts' knowledge into working models is at least as important as the Stats+ML bits.
- PaulHoule 14y ago
  I'd say the current academic research in ML is not oriented towards producing people who can use ML in real applications. I've hovered around the periphery of a world-leading ML research group, and the first takeaway I have is that 7 years ago I thought the stuff they were working on was going to take the world by storm, but looking back, I can say it hasn't. This group does a number of research projects on narrowly defined topics. 4 out of 5 of these projects try out some refinement of the method that doesn't really work. Maybe 1 out of 5, if that, point to a real improvement. The big thing that's lacking are serious attempts to push the state of the art by attacking a problem holistically and "taking no prisoners" -- yet this is exactly the kind of thinking necessary to commercialize ML. The leader of the group got tenure so he thinks everything is going OK. He won't even offer an analysis of why this technology hasn't been widely commercialized. PhD students from this group usually interview at Google, Microsoft and Facebook but these three employers are the only ones they consider as an alternative to academic employment.

harscoat 14y ago

Surprised Ben Rooney did not mention IBM acquisition of Vivisimo the day before (Apri. 25) this article (Apr.26) http://www-03.ibm.com/press/us/en/pressrelease/37491.wss http://www-03.ibm.com/press/us/en/pressrelease/37491.wss "IBM Advances Big Data Analytics with Acquisition of Vivisimo"

sandee 14y ago

Basically a solution looking for a problem. They are right, the complexity that big data caters for requires expertise at both technical and business level that would be costly (though may not be at infrastructure level). In the current economy, it looks even more difficult where businesses want to squeeze the maximum out of dollar investment. IMO, its too early stage for big data solution adoption. However stage could be set for startups who can come up innovative solution that brings the cost level down together with simple and useful easy to grasp solutions.

maeon3 14y ago

Why don't the people who hire doctors, dentists, and lawyers suffer from the same talent shortage that the people who hire 'big data' computer scientists feel? Because it's better across the board to start your own startup than work your ass off for a 4% raise at a place which recognizes you as top talent. I'm on the verge of starting a startup myself, removing myself from the people in this list. There is a shortage of talent in computer science, but never in the other disciplines, it may take another 30 years for the suits to have the ability to understand why.

enfilade 14y ago
I agree with your overall line of thinking. In your last sentence you wrote: "There is a shortage of talent in computer science, but never in the other disciplines, it may take another 30 years for the suits to have the ability to understand why." Could you elaborate on this point -- do you feel that there is not a shortage in disciplines such as dentistry and law because many people are willing to work very hard for only 4% raises? Thank you!
- tomjen3 14y ago
  You can actually make good money by going into the law or medicine. You have to work and and be skilled of course, but lets be honest that is also required for a start-up. I can't help but think that ability is because you can't run a law firm without being a lawyer so the boss has some idea of what it means to be a good lawyer and how to treat them.
  - anothermachine 14y ago
    Most lawyers at the top-income end hate their bosses and jobs. Law firm partnership track at large firms is a dog-eat-dog 80-hour week hell. People are only happy when they are the few who claw to the top, or the many who drop out. The ones in the middle are suffering as bad as any stereotypical bank programmer.

chintan 14y ago

How media sees Big Data: BIG database => BIG machine learning algo => BIG MODELS => PREDICTIONs, Insights => $$$ How it is actually done: awk -F"\|" '{print $1}' SCRAPED_file_pipe.txt | sort | uniq -c | head -n 10 => $$$

gaius 14y ago
Ah, you've used Ab Initio then.

giardini 14y ago

Looks like the prelude to yet another H1-B buildup.

alecco 14y ago
Only this time US is less attractive. Even mexicans started to move out.
tosseraccount 14y ago
The only way we can really measure "shortage" is via compensation. If that's so, we need more hedge fund managers and surgeons, not grunt data crunchers. There are many problems with guest workers. The richest people in the world get special access to indentured labor. It targets specific industries thus amounting to a subsidy. It helps big business crush small business. The H1-B in particular is a tool to increase outsourcing and keep wages down (wages which are typically earned in the highest cost of living areas in the country at 60 hours a week). H1-B ? No, thank you! On the job training and good wages? Yes, please!

ambiate 14y ago

I just listened to a lecture on this at BU. Emerging Internet Technologies at IBM or something of that nature. He was basically trying to sell us his product that crawled the internet (mainly a firehose at Twitter) and gathered statistics for advertisers and presented it in pretty graphics. The main issue they had was developing language recognition. Deciding if a user 'liked', 'loved', 'hated' or was 'neutral' about a product. Another issue that stood out to me had to do with their reliance on the internet. Just because 200 users tweeted that 'this movie is going to suck' does not really represent the overall opinion. To reiterate, the whole buzz of the lecture was the biggest turn off. He wasn't explaining about how to expand on his product or where to go from here. Just that they had developed a product and we could use it instead of attempting to develop one ourselves.

ohashi 14y ago
Maybe 200 is too small a sample size, but you can glean and predict stuff with that sort of data. My master's thesis was about predicting box office sales based off twitter data. (coincidentally these guys published a couple months before me: http://www.fastcompany.com/1604125/twitter-predicts-box-office-sales-better-than-anything-else http://www.fastcompany.com/1604125/twitter-predicts-box-offi...) but we had incredibly similar results. It is pretty interesting to see what you can do with that data. If it makes you feel any better, you can build it yourself, I certainly did. I didn't even use any libraries like NLTK to build my sentiment analysis. Read some research papers and built it from scratch (code wise at least, the ideas used were fairly common). It's a fun challenge. I still work with that code every day and use it in my startup now :)

deleted 14y ago

[deleted]

chrisrhoden 14y ago

Can anyone explain what talent refers to in this context? Is it someone who has learned this stuff, someone who is capable of learning it, or someone who was born with an innate understanding?

gambler 14y ago

Managers frequently wail about skill shortages, but very often it's pure hypocrisy. The real problem is the reluctance to do any training (and I don't mean formal training) combined with the desire to get proven experts in whatever field. Proven experts must have years of experience in applying their expertise. If no one lets people with less experience to work in that field, where the hell would those experts appear from? Another dimension? Can I be a 80% developer and 20% "data scientist" in your company to try the new role out? The bigger your company is, the less likely the answer to be a "yes". Since Big Data implies a big company, the resulting "shortage" is not surprising. It's self-made.

ImprovedSilence 14y ago
True statement. It seems the trend now days is "get a grad degree, foot the bill and time yourself". Many companies I see and work with tend towards that mentality, as opposed to building a base of highly skilled workers on from the inside. It's easier and cheaper for a company to ask if you have a piece of paper, than for them to train you and get you up to speed.
paulsutter 14y ago
Companies looking for "proven experts" are working on yesterday's problem. Find a better employer. Move somewhere with more choices if you need to.

paulsutter 14y ago

This article is nonsense. Talented developers are talented developers. At Quantcast we used Hadoop in production before it was even called Hadoop and now we process 10PB a day. We forbid our sourcers from using Hadoop as a resume search term because it meant absolutely nothing. Statisticians who can code are scarce, but companies that know how to use them are scarcer.

why-el 14y ago

So I was wondering if any fellow HNer is on a quest to be at least comfortable around these problems. Can you share your plans? Currently I am starting with some linear algebra and I have plans to move to statistics then pick up a book on machine learning. I would really use some advice.

achompas 14y ago
I'm on this path right now. Working simultaneously on a MSCS at NYU and a full-time developer gig at Knewton (a company which truly understands the value and risks of data R&D). I'd really start with this awesome curriculum[0] by Joseph Misti. He nails the mix of modeling, algorithms, math/stats, development, and distributed/Unix skills one needs to become comfortable around these problems. More importantly, his advice agrees with my experience on the data team at Knewton--we really use a bit of all of the above skills to solve our problems. My email is in my profile if you (or anyone else) would like to chat about this a bit more. I'm also in NYC, and totally willing to grab tea and chat in person. [0] http://www.quora.com/What-skills-are-needed-for-machine-learning-jobs http://www.quora.com/What-skills-are-needed-for-machine-lear...
seanharnett 14y ago
Andrew Ng's machine learning class on coursera is a very nice, easy introduction to the subject.
- why-el 14y ago
  Yes, and I started that before realizing that I need more background knowledge, hence starting with some math. :)

pmb 14y ago

"claims of severe talent shortage in Big Data http://online.wsj.com/article/SB10001424052702304723304577365700368073674.html http://online.wsj.com/article/SB1000142405270230472330457736... Ok... where are the high salaries (500k$ a year)? No? No real shortage." https://twitter.com/#!/lemire/status/196245665951649793 https://twitter.com/#!/lemire/status/196245665951649793 Business has a shortage of "big data" folks in much the same way I have a "huge sailboat" shortage. Neither of us want to pay for it. We want it, but not for the going rate. Only one of us has a media platform, though.

jandrewrogers 14y ago
The salaries are already moving north of $200k even outside of Silicon Valley and New York City and getting more expensive by the month. How high do they have to be before we have a "shortage"? The problem is not lack of money, it is that demand has greatly outstripped a finite supply. Very high wages do not automagically create new people with the requisite skills and this is the real bottleneck. It takes significant aptitude and years of training/experience to become useful as a "data scientist". It is not as easy as I think people are imagining. We train people with excellent raw skills where I work, usually strong applied mathematics backgrounds with natural programming skills. It is much easier than trying to find someone outside with these skills, though we do attempt outside recruitment. It still takes years to develop the people we train into a good, basic data scientist.
- bearmf 14y ago
  Look, this job title is at most 2 years old. How can someone have years of experience in this? OTOH, there are plenty of people with strong applied math and good programming skills.
  - jandrewrogers 14y ago
    The set of skills existed before it had a trendy job title so you can have the experience even if it was called something else. This is true of most of the people currently working as data scientists. In a similar vein, I was designing big data systems years before "big data" became a term or trendy. For any particular odd skill mix you can come up with, there are people with that skill mix who are already doing a similar job. But usually people do not intentionally build that skill mix until it becomes an official job title and career path in the eyes of the public so it is a very small pool of people. In the case of modern data scientists, having strong applied mathematics and programming skills is about halfway to where you need to be and a good starting point. The demand has temporarily grown much faster than the convertible talent pool can develop the additional set of skills required.

kylemaxwell 14y ago

Not entirely sure this is true, to be honest. Most of "data science" lies in the work of collecting and cleaning the data to get it into a usable state. A recent story on the"fallacy of the data scientist shortage"[1] goes into more detail, but in reality what we want in this quantity are better data analysts. I love the idea of data science as, essentially, viewing statistical analysis from a computer science perspective, but the breathless predictions of a huge shortage seem a little overblown. [1]: http://smartdatacollective.com/nraden/48952/fallacy-data-scientist-shortage http://smartdatacollective.com/nraden/48952/fallacy-data-sci...

tlogan 14y ago

It seems the problem is that some companies are looking for person who is expert in setting up scalable systems (Hadoop cluster, storage, high availability, etc.) and that she/he also knows statistics and efficient ways of processing and understanding the data. Good luck with that. My observation is that requirements like this come from people who did mainly web programing (and actually that was making a lot of money so with money they become influential): assuming that this equivalent of writing both ruby code and javascript code. Building team is hard and in order to solve "big data" problem you need to build a balanced team.

thornad 14y ago

I've done this kind of thing most of my career, including doing it for NASA and Unilever Research. You can't really train an average graduate to do this. You need someone with a pretty highly developed integration between 1:intuitive/creative abilities, 2:mathematical/analytical skills, and 3:engineering/ability to make things happen. Add to that 4:work experience in the real world, and 5:ability to easily understand how things work in a field you delve into for the first time... And there's very few people in the world who can do this. At my previous work place we tried for a whole year to hire someone who would at have at least some of these skills and seems promising to develop the rest on the job. We couldn't find anyone although we interviewed about 30 different people (from about 500 resumes most of them with a PhD in ML from a good university). And this was in central London, UK.

bearmf 14y ago
But you never tried training them. Sure, no one can do it right off the bat, without prior experience.

mardack 14y ago

This is good news.

radikalus 14y ago

Until the pay is comparable to finance, good luck? I'd love to work on (arguably) cooler problems, but the combination of lower pay and the constant need to use the "hot new thing" to solve problems doesn't make transitioning look remotely attractive. Really, the second is the HUGE obstacle: - You don't know anything about aNNs? Sorry, no job. - Nobody uses aNNs anymore, SVMs are all that matters. Sorry, come back after you catch up. - SVMs? Man, we need someone who's got expertise in optimizing RFs and Bayesian Trees. We don't want "black box" machine learning. We need to "understand" the results. Sorry no job. - Decision trees? GTFO man. We're doing rNNs now. - I'm pretty impressed with your data mining knowledge, but we're looking for someone with a background in DLMs and GPs. Sorry, no job. - repeat until vomit/suicide I kind of wonder about the need for "badass" math skills; I'm not terribly convinced that math wizards are extraordinarily high value relative to people with other types of data analysis skills.