6 ms·

The real reason you were rejected, or anyone applying in a highly competitive field: an oversupply of qualified candidates.

ryandrake 10y ago
But, wait! I thought there was a "shortage of tech workers."
- x0x0 10y ago
  Which must necessarily be urgently addressed by open immigration for anyone who can write code! The future of silicon valley demands it!
ska 10y ago
In this particular case (data science) it is more an oversupply of candidates (qualified and not), plus difficulties defining and measuring "qualified", plus buzz. It's a difficult enough field to hire in when you understand what it is (and isn't) - and lots of companies are trying to do it with far more vague goals.
- throw_away_777 10y ago
  Why do you think candidates for data science jobs aren't qualified relative to other positions?
  - yummyfajitas 10y ago
    I've probably interviewed about 70-100 such people in the past year and a half. Exactly 1 such person was qualified (I hired him). The issue in my view is the following: people who know both statistics and computer science are extremely rare. People who actually understand statistics are rare. I can probably weed out 1/3 to 1/2 of candidates simply by asking what a p-value is, or what precision/recall are (this includes people who said they worked in search). Of the ones who know basic stats, most are neither good at nor interested in programming. They just want to use existing libraries to crunch numbers in a Jupyter notebook, then hand that off to the developers. Finding a person who can come up with a predictive model, understand what they did, optimize it without breaking it's statistical validity and deploy it to production is very hard. (If you can do this, I'm hiring in Pune and Delhi. Email in my profile.)
    - halflings 10y ago
      Ignoring what a p-value is does not mean that you don't know statistics. p-tests are not some inherent statistical property, they're just a useful model for significance. People coming from a CS background most likely didn't have to deal with p-values, but they can still be good at linear algebra or bayesian statistics. (not sure I can defend somebody that does not know what precision/recall are)
      - Bootvis 10y ago
        Regarding precision/recall, I've a background in financial econometrics and this is the first time I encounter the terms.
        halflings 10y ago
        That's OK. The article was talking about somebody interviewing for a search-related position (where precision and recall are usually what you are optimizing for). I guess they might be called differently in econometrics?
        kenjackson 10y ago
        I think the problem is that certain subfields use different terminology to mean similar or identical concepts. For example, while I'm in software, I tend to hear the terms sensitivity and specificity. They are historically medical terms. They aren't identical to recall/precision, but I think you can derive one set from the other.
        Bootvis 10y ago
        That certainly seems likely and a good thing to keep in mind when you're giving or taking an interview!
        antognini 10y ago
        The fundamental thing to know is the confusion matrix. There are about a dozen terms for various descriptors of the matrix, but they all can be calculated if you know the confusion matrix. The Wikipedia page has a great table to describe them all: https://en.wikipedia.org/wiki/Confusion_matrix https://en.wikipedia.org/wiki/Confusion_matrix You can see from that that sensitivity and recall are the same thing, but specificity and precision are not.
  - drxzcl 10y ago
    Not OP, but I think many companies aren't qualified to judge who is and who isn't a qualified candidate, at least for the first hires. This turns the whole thing into a "market for lemons". I've helped a few organisations solve this bootstrap problem by helping out with candidate selection and interviews, but many other just don't ask for help.
  - ska 10y ago
    Two main factors make data science stick out a little for me right now, although it isn't unique. One is that there is buzz & excitement around "data science" right now. Nothing specific to this area, but in my experiences this creates a large number of under- or un-qualified applicants. It also creates an environment for companies to desire to hire a role they are not well qualified to hire for. It is really difficult to hire well for roles you don't understand well. The second thing is that extremely few people are actually ready for this sort of job straight out of an academic program. A related Ph.D. or post doc plus a few years solid training in industry can make you a great candidate, but the academic work alone usually isn't even remotely close. There is confusion about this among both candidates (don't know what they don't know) and hiring managers (don't know what they are actually looking for). Add to that an oversupply of academic credentials relative to academic jobs and you have a problem. If you are a large company with a well defined data science program and a defined "entry level" data science role, if you take skill development and training seriously and have the senior staff for it, well then you are fine taking strong academic candidates and turning them into talented data scientists. If you are a less experienced company looking for scientists to solve a problem you don't fully understand, you may be in for a pretty rough ride.
- achompas 10y ago
  I think this is closer to reality. Tim is not competing with many folks like him -- he's a knowledgable, experienced, and capable data scientist with significant infrastructure experience, and a net positive to any team he joins. Some problems possibly originating from the company perspective (1) They are inundated with applications folks of all sorts of backgrounds: engineering, finance, academia, marketing, BI/analytics, etc. (2) They still haven't figured out hiring. To be fair, no one really has figured it out. Jeff Kolesky recently covered this as part of an excellent blog post. [0] (3) In addition to the typical variance in engineering interview processes, we now introduce variance in the definition of data science across companies, which just complicates things further. (4) Basically everything else Tim mentioned in his post: role or goals aren't clearly defined, remote data science is an unknown, etc. [0] http://kolesky.com/datums/job-search/ http://kolesky.com/datums/job-search/
emplynx 10y ago
But that offers no explanation why some are hired and some aren't.
- cortesoft 10y ago
  Because SOMEONE has to be hired? Let's supposed, for a moment, that the hiring was completely random. 100 people apply, they pick a random name out of a hat, and hire that person. Your name might never be chosen.

ryandrake 10y ago

But, wait! I thought there was a "shortage of tech workers."

x0x0 10y ago
Which must necessarily be urgently addressed by open immigration for anyone who can write code! The future of silicon valley demands it!

ska 10y ago

In this particular case (data science) it is more an oversupply of candidates (qualified and not), plus difficulties defining and measuring "qualified", plus buzz. It's a difficult enough field to hire in when you understand what it is (and isn't) - and lots of companies are trying to do it with far more vague goals.

throw_away_777 10y ago
Why do you think candidates for data science jobs aren't qualified relative to other positions?
- yummyfajitas 10y ago
  I've probably interviewed about 70-100 such people in the past year and a half. Exactly 1 such person was qualified (I hired him). The issue in my view is the following: people who know both statistics and computer science are extremely rare. People who actually understand statistics are rare. I can probably weed out 1/3 to 1/2 of candidates simply by asking what a p-value is, or what precision/recall are (this includes people who said they worked in search). Of the ones who know basic stats, most are neither good at nor interested in programming. They just want to use existing libraries to crunch numbers in a Jupyter notebook, then hand that off to the developers. Finding a person who can come up with a predictive model, understand what they did, optimize it without breaking it's statistical validity and deploy it to production is very hard. (If you can do this, I'm hiring in Pune and Delhi. Email in my profile.)
  - halflings 10y ago
    Ignoring what a p-value is does not mean that you don't know statistics. p-tests are not some inherent statistical property, they're just a useful model for significance. People coming from a CS background most likely didn't have to deal with p-values, but they can still be good at linear algebra or bayesian statistics. (not sure I can defend somebody that does not know what precision/recall are)
    - Bootvis 10y ago
      Regarding precision/recall, I've a background in financial econometrics and this is the first time I encounter the terms.
      - halflings 10y ago
        That's OK. The article was talking about somebody interviewing for a search-related position (where precision and recall are usually what you are optimizing for). I guess they might be called differently in econometrics?
      - kenjackson 10y ago
        I think the problem is that certain subfields use different terminology to mean similar or identical concepts. For example, while I'm in software, I tend to hear the terms sensitivity and specificity. They are historically medical terms. They aren't identical to recall/precision, but I think you can derive one set from the other.
        Bootvis 10y ago
        That certainly seems likely and a good thing to keep in mind when you're giving or taking an interview!
        antognini 10y ago
        The fundamental thing to know is the confusion matrix. There are about a dozen terms for various descriptors of the matrix, but they all can be calculated if you know the confusion matrix. The Wikipedia page has a great table to describe them all: https://en.wikipedia.org/wiki/Confusion_matrix https://en.wikipedia.org/wiki/Confusion_matrix You can see from that that sensitivity and recall are the same thing, but specificity and precision are not.
- drxzcl 10y ago
  Not OP, but I think many companies aren't qualified to judge who is and who isn't a qualified candidate, at least for the first hires. This turns the whole thing into a "market for lemons". I've helped a few organisations solve this bootstrap problem by helping out with candidate selection and interviews, but many other just don't ask for help.
- ska 10y ago
  Two main factors make data science stick out a little for me right now, although it isn't unique. One is that there is buzz & excitement around "data science" right now. Nothing specific to this area, but in my experiences this creates a large number of under- or un-qualified applicants. It also creates an environment for companies to desire to hire a role they are not well qualified to hire for. It is really difficult to hire well for roles you don't understand well. The second thing is that extremely few people are actually ready for this sort of job straight out of an academic program. A related Ph.D. or post doc plus a few years solid training in industry can make you a great candidate, but the academic work alone usually isn't even remotely close. There is confusion about this among both candidates (don't know what they don't know) and hiring managers (don't know what they are actually looking for). Add to that an oversupply of academic credentials relative to academic jobs and you have a problem. If you are a large company with a well defined data science program and a defined "entry level" data science role, if you take skill development and training seriously and have the senior staff for it, well then you are fine taking strong academic candidates and turning them into talented data scientists. If you are a less experienced company looking for scientists to solve a problem you don't fully understand, you may be in for a pretty rough ride.
achompas 10y ago
I think this is closer to reality. Tim is not competing with many folks like him -- he's a knowledgable, experienced, and capable data scientist with significant infrastructure experience, and a net positive to any team he joins. Some problems possibly originating from the company perspective (1) They are inundated with applications folks of all sorts of backgrounds: engineering, finance, academia, marketing, BI/analytics, etc. (2) They still haven't figured out hiring. To be fair, no one really has figured it out. Jeff Kolesky recently covered this as part of an excellent blog post. [0] (3) In addition to the typical variance in engineering interview processes, we now introduce variance in the definition of data science across companies, which just complicates things further. (4) Basically everything else Tim mentioned in his post: role or goals aren't clearly defined, remote data science is an unknown, etc. [0] http://kolesky.com/datums/job-search/ http://kolesky.com/datums/job-search/

emplynx 10y ago

But that offers no explanation why some are hired and some aren't.

cortesoft 10y ago
Because SOMEONE has to be hired? Let's supposed, for a moment, that the hiring was completely random. 100 people apply, they pick a random name out of a hat, and hire that person. Your name might never be chosen.