7 ms·
I'm not sure that the kinds of employees that this article describes will ever be a large number. There could be more of them in the future, but someone who is
by _delirium 15y ago
I'm not sure that the kinds of employees that this article describes will ever be a large number. There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those. Companies might consider looking at better ways to build teams that combine talent that exists, instead of pining for more superstars.
I'm reminded indirectly of an acquaintance of mine who works on repairing industrial machinery, where companies complain of a big skills shortage. They either fail to realize or are in denial about what that means in the 21st century, though. It might've been a one-person job in the 1950s, a skilled-labor type of repairman job. But today they want to find one person who can do the physical work (welding, etc.), EE type work, embedded-systems programming (and possibly reverse engineering), application-level programming to hook things up to their network, etc. Some of these people exist, but it's more common to find boutique consulting firms with 3-person teams of EE/CE/machinist or some such permutation. But companies balk at paying consulting fees equivalent to three professional salaries for something they think "should" be doable by one person with a magical combination of skills, who will work for maybe $80k. So they complain that there is a shortage of people who can repair truck scales (for example).
- disgruntledphd2 15y agoI completely take most of your points, but I think that pretty much all quantitative PhD's are going to be close to "data scientists". Given that stats and explaining your research are requirements, all that's left is to train them to program, which a lot of people are already doing. As a matter of fact, since I heard about this big data stuff I've been honing my skills in this area, in case the hype actually manifests.
- tgflynn 15y agoAnyone who can earn a PhD can learn to program but being good at engineering the very complex processes that are needed for effective machine learning applications is a skill that is not so easily acquired. I've worked with quite a number of quantitative PhD level people in my career and most often the quality of their code leaves much to be desired.
- Dn_Ab 15y ago_delirium's post acknowledges your point but is looking for an even rarer person: "There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those". Someone that can program, understands statistics and can present the data in an appealing manner without losing significant fidelity. Many people underestimate the difficulty and skill required in presenting data in a way that makes sense and also actually says something. There is a significant gap between presenting data that is satisfactory to a research advisor and something that a business person with barely enough time to think can grasp without misconception.
- disgruntledphd2 15y agoAgain, I completely see the difference (and am actually in the process of moving full time to the private sector from academia, so will probably understand a lot more in six months) but visualising data well is not that hard. Step 1: learn R Step 2: Learn PCA Step 3: Learn ggplot2 Step 4: play with the different geoms until you understand them (seriously though, everyone's eyes are optimised to find patterns, and if you can apply significance testing to these then you should be good) Step 5: profit!? Note that I am being somewhat facetious here, but I suspect that the mathematical knowledge and ability to apply this to business problems will be the real limiting factors, as good practices in data analysis, programming and visualisation can be learned. Granted that will take a long time to learn, and there will be individual differences, but its doable. Whether or not it will be done at all though is another matter. Again, delirium's point is trivially true if one requires these people to know all of statistics, programming and data presentation as I don't think there's anyone who knows all of any one of these subjects. I suppose it somewhat depends on what the skill levels for each of these areas need to be, and that varies from person to person as well as from application to application.
- ryanlchan 15y agoAllow a short vignette from a former academic and now management consultant. We spent six months at a major pharmaceuticals client examining their reimbursement data. Poring over many millions of rows of transaction data and thousands of payment codes (which, of course, were unique across sales geographies), we determined the ten regions at highest risk of reimbursement collapse. R was used, maps were created, beers all around. But almost none of it was used for the executive presentation. In fact, the only part that was included was that we had ten regions that needed fixing, and our suggestions on how to fix it. You see, the CEO was dyslexic, the chairman of the board was colorblind, and the COO was a white-boarding kind of gal, so given this audience the nuts and bolts of our advanced statistical analysis were simply irrelevant. This is hardly surprising. If we are having so much trouble hiring people who are fluent in Big Data, how can we expect business leaders to be even conversant? With only slight exaggeration, the way you do your analysis and the visualizations that you create are not important. Companies are demanding Big Data scientists because they suddenly have lots of data and see the term Data Scientist in the news. But what they really want is not Data Scientists, it's business insights and implications from Big Data. The customer needs 1/4" holes, but we're all arguing over which brand of laser powered diamond drill they should buy.
- timr 15y ago"I think that pretty much all quantitative PhD's are going to be close to "data scientists"." Exactly. Fundamentally, "data science" is known far more widely by its other name: science. Yet we've reached the bizarro-world place where there are huge numbers of un- or under-employed scientists looking for work, while companies are freaking out about hiring the "rare" computer programmer who happens to know some statistics and self-identifies as a "data scientist". It's rather absurd.
- chimeracoder 15y ago> I think that pretty much all quantitative PhD's are going to be close to "data scientists" Having taken all but one of the core requirements for a masters' degree in statistics at a university with a well-respected statistics department, I can tell you that's very much not true. The true challenges in data science have almost nothing to do with what you spend 90% of your time as a graduate student studying (whether you're getting an MA or a PhD, this applies the same). You may happen to end up a qualified data scientist, but that's not by design of the program. The big problems in data science are almost a disjoint set from the big problems in statistics (at least the solved ones), and that's because the things that are tractable from a theoretical/mathematical perspective are very different from the ones that we hope to solve in the workforce. We're just starting to bridge this gap in recent years (particularly with the advent of computers), but that's a very, very nascent trend. This isn't unique to my university, either - most schools just simply aren't teaching the type of skills that a data scientist - not a statistician, but a data scientist - would need to be competitive in the work force. Those that do know these skills mostly do by chance - either because they branched into statistics from another discipline, because they were forced to learn it on the job, or because they took the time to learn it themselves. All three of those are pretty rare - I recently took a class in applied data mining and Bayesian statistics. Except for a few undergraduates majoring in comp sci, the class was mostly graduate students in statistics, and those who knew how to program were in the stark minority (and were very popular when we were picking project groups!) > all that's left is to train them to program And to turn everything that they've learned and studied for the past two, four, or more years on its head so that they can actually put it to use. Okay, not everything, but at least 80% of it. Seriously, studying statistics at a high level is incredibly valuable, but it's not sufficient - it's not even going to get you half of the way there.
- tel 15y agoI'm in the same boat and one of the funnier professors in math stat loves to talk about the students he hasn't "ruined" because they manage to learn programming and practical finite sample wisdom and go on to be successful in the industry. And then he talks about his other students, with great love, who just like proving theorems.
- 15y ago
- mgkimsal 15y agoI see this in software dev consulting, and it's probably in many other fields as well too. I see companies looking for one person who's a highly-skilled DBA, sysadmin, developer and who can interact affably with all levels of people in a company, including customers, at the drop of a hat. Often it's because they had one 'magical' person who did all that, though usually not very well, and the next person (or team) who comes in after on that project has to deal with a bundle of undocumented crap that never really 'worked', but worked well enough to keep some people happy. This scenario has been surprisingly common, and I worked with a company last year who was committed enough to hire dedicated dba and sysadmin positions, vs continuing to rely on app devs to handle all that stuff. The separation of concerns has worked out pretty well, though at first there was some concern about the cost of adding 'dedicated' people. In reality, all that work was being done anyway, often in ways that weren't terribly understandable by anyone outside the project. The time it takes to get feature X done, tested, db upgraded, rolled out, servers maintained, patches applied, etc... is going to be roughly the same. If that's going to take, say, 100 hours, it's either 3 weeks of one person, or 1 week of 3 people. It's not worked out quite that cut-and-dried, but it's coming close. And the ability for each person/team to focus on their core skills and let someone else handle the "other stuff" has meant that, generally, the quality of things is better all around.
- bmelton 15y agoPerhaps slightly offtopic, but at least in Software dev consulting, there's a market for the multi-hatted individual in consulting directly, and pay is generally commensurate with the number of hats you can speak intelligently on to a customer. This isn't necessarily true everywhere -- when I lived in Memphis, TN, I often felt that I was dooming my professional career as, every time I ran into a challenge, I'd fork my efforts and start tackling it. This means that I was getting skilled in (but not expert in) a wide variety of things. In short, my knowledge set was very wide, but somewhat shallow. It wasn't until I moved to the DC area that I realized there's not only a market for that type of person, but a fairly lucrative one. There are other tracks too, Architect, Management, whatever... whereas the scale repairman who can weld, machine and EE on a scale system is not exactly limited to just working on scale systems, but isn't necessarily opened up to as many positions as with software dev.
- randomStuff 15y agoThis is a total ploy by large companies to increase the H1-B visa cap. I have seen companies post job openings with starting salaries of 40K for experienced developer positions so they can then claim there were no American applicants. Its a self fulfilling prophecy, if companies outsource the jobs then people don't study those skills for fear of their job going to India; then the companies complain that there aren't enough Americans with CS degrees so they need more H1-B visas. The US needs tariffs in the IT industry to save our skill base so we can be competitive long term.
- yummyfajitas 15y agoProtecting the industry with trade barriers definitely sounds like the way to make it competitive. If we do this our IT industry might one day lead the world, like our sugar industry, automotive manufacturing and softwood lumber do.
- chadmalik 15y agoDespite your sarcasm, do you have any evidence of nations who have no import restrictions or government subsidies and have built up a competency? China and India are notorious for their import controls, so those are off the table for you to use. And don't use the US technology industry, which has been massively subsidized by government research, import controls, and purchasing. Hey, even the US sugar industry benefits massively from tariffs; without them it would have to compete with much lower priced product from warmer nations.
- v0cab 15y agoOr: Like the South Korean electronics industry does.
- gaius 15y agoOne other thing is that the traditional entry route to software for the non-traditional candidate was via the helpdesk or QA department. You got your foot in the door, impressed the established engineers by turning around tickets quickly or by writing comprehensive bug reports, then when they needed another developer, you got the tap on the shoulder. Some of the best engineers I've worked with have come in through this route, including the best manager I've worked for. If these kinds of positions are outsourced, how do we tap into that seam of talent anymore?
- ohyes 15y agoI think another issue is that no one is interested in doing 'on the job' training. Few people learn statistics, programming, and data-presentation in college (not all three anyway). Companies might consider finding someone smart with one or two of the skills and expecting them to learn the other skills on the job. And what is talent? To me, talent is the ability to learn to do something quite well. To say there is a lack of 'talent,' when you have only searched for people who /already/ have those specific skills is disingenuous. You have barely looked. Who is complaining about the lack of talent? Companies! Who produces talent? Companies! Who can fix this? Companies! Part of running a company efficiently is is hiring, grooming and retaining juniors (who will become seniors). For God knows what reason, few companies actually understand this. If the skills you are looking for are lacking, hire someone smart and teach them.
- _delirium 15y agoI agree somewhat, but retention is a fairly major problem. The shift away from career-length employment means that neither employers nor employees assume there will necessarily be much loyalty or longevity in the relationship. I think the decline in on-the-job training is directly related. Engineering firms used to be able to assume that it's okay to lose money on the first five years or so of an employee's work, if they built up skills that will make the company lots of money over the next 30-40 years of the employee's career. But if the company invests five years of significant training in a junior employee, and then the employee jumps ship to do freelance consulting or work for a competitor, the training never ends up paying for itself.
- chadmalik 15y agoWell maybe companies should start thinking about what it takes to retain employees? Perhaps start retirement programs that incentivize people to stay with them for a number of years. As far as I can see in the economy, you are wrong. The real issue is viewing "human resources" as disposable, plug-and-play things that you use to fill stopgap needs, not some desire by employees to job-hop all the time. A lot of people, especially when they start to settle down with a mortgage and/or kids, would probably love to have employment stability.
- jimmytucson 15y agoThat's a great point. In theory, having a shortage of X at price Y is nonsensical. It doesn't make any sense to say there's a shortage of worker drones with a mastery of multiple advanced disciplines. There are "enough" of them, they're just called consultants and they make like a thousand dollars an hour. Or if there aren't enough of them, offer a salary that will enable them to at least pay off the student loans they'll accumulate getting a Ph.D and a couple masters degrees on top. ...in theory.
- robomartin 15y agoI think you are very much on point with your comments. I am one of those people who are actually capable of going from MIG welding to designing websites, writing iOS apps, developing embedded hardware and software as well as GHz-range electronics with FPGA's, mechanical design and FEA. The only option for someone like me seems to be to run your own business. Nobody is likely to pay for the combined skill set. Which means that having these skills is both a blessing and a curse depending on your point of view. I can take any product from drawing to completion. My resume scares most employers. And, in many ways, rightly so. Hiring someone who can do the work of five people is very risky. You loose one person and your entire team is gone. And, of course, there is no way that one person can have the productivity offered by a team of specialists. In the end, someone with my skill set either ends-up doing their own thing or in a managerial position where the wide knowledge base and context gained from actually being able to do the work can be harnessed to guide and assist a team in achieving the required goals. As an entrepreneur, having a wide skill set can be priceless so long as you start letting go as soon as you can start hiring specialists. This can be hard for some. It's great to be able to do it all when you want to launch something and save a bunch of money. Once launched, you need to divest yourself from responsibilities as quickly as possible because you will hit productivity walls and you simply can't focus on everything at the same time. The age of the generalist is pretty far gone. If employment is the goal it is best to focus on one subject and become really good at it.
- deleted 15y ago[deleted]
- mturmon 15y agoGood take on the article. I.e., there's a sense that money is to be made through "big data" if only we could find the right, highly skilled person. Now, we notice that we're not making those gobs of money, as promised in the business press (and shown through a few example successes), and conclude that there must be a shortage of such people.