4 ms·

To make a specific point, without prejudice to the main points of the blog post: >One of the open secrets of working in technology is that technical interviews

by feral 10y ago

To make a specific point, without prejudice to the main points of the blog post: >One of the open secrets of working in technology is that technical interviews are completely worthless as a predictor for whether someone is a good hire or not. [...] >> We looked at tens of thousands of interviews, and everyone who had done the interviews and what what they scored the candidate, and how that person ultimately performed in their job. We found zero relationship. > It’s amazing to think that Google found zero relationship between an interviewer saying “Hire” and whether the candidate was actually a good hire or not. The fact [the scores of someone who gets hired] are not predictive of their eventual performance, does not mean that hiring process is completely random, or worthless. In order to tell that, you would need to look at the cohort of people who were 'no hire' and compare them to the 'hire' cohort; it's not enough to just look within the 'hire' cohort. In other words, the hiring process is (1) designed to be a Classifier, typically tuned with an emphasis on having a low false positive rate. Hoping that (2) the scores of the small subset of candidates who are classed 'hire', also act well as a regression to the candidate's eventual performance, is a big additional ask. Just because there is evidence that (2) is untrue, or particularly noisy, does not necessarily mean that (1) is broken. I don't see a great way of solving this, without a large company like Google also randomly hiring 'no hires' from different stages in their hiring process (and subsequently measuring their performance) in order to quantify the performance of subsequent steps in the hiring process - not aware of anyone who has done that; it would be really interesting to read how a company who does that gets on, but an expensive experiment.

vonmoltke 10y ago
> The fact [the scores of someone who gets hired] are not predictive of their eventual performance, does not mean that hiring process is completely random, or worthless. It does not automatically mean this, but it is a strong indicator. If the output of the classifier is not correlated with a given output sample it is reasonable to assume, in the absence of other data, that it is uncorrelated for other samples as well.
- feral 10y ago
  I do accept it is evidence - but I don't think I would say a strong indicator, and it doesn't justify the strong conclusions made that the process is completely arbitrary. - It's imbalanced classification: there are going to be more rejections than acceptances; so you have less statistical power, to find if there is a real relationship, when you just analyse the acceptances. Even apart from that, class imbalance can unintuitively affect post-hoc inferences. - Scores are in context of the other candidates at a given level, and stage in the interview process, so are fundamentally already residual in some sense. - I'm willing to bet, from hiring I've seen, that scores are distributed somewhat about the 'hire/no-hire' bar, and that extremes are rare - the process is going to be tuned, implicitly in the organisation, to make that 'hire/no-hire' decision, rather than predict performance. > it is reasonable to assume, in the absence of other data, that it is uncorrelated for other samples as well I think that's a very strong statement: it's not a random sample, its explicitly concentrated at one end of the distribution. I do accept it is evidence, just not near enough to justify the conclusion the whole process is arbitrary.
- statics 10y ago
  No, it's not a strong indicator that the process is random or worthless. That makes the assumption that the people rejected by the process would have scored just as well. It could be very good at determining who has the technical skills to work there. It might be that their performance scoring of hires is measuring something else- like how much their manager likes them.
  - vonmoltke 10y ago
    It could, but if technical skill is uncorrelated with performance scoring it will be really hard to determine how many people who make it through the process actually have the technical skill and how many don't.
rando289 10y ago
> it's not enough to just look within the 'hire' cohort. Enough for what? Why not? They specifically have a "score", not just a hire / no hire decision. Sure it would be better, but you don't make any convincing argument why their analysis is not valuable. > designed to be a Classifier, typically tuned with an emphasis on having a low false positive rate. classifier of what? Why is classifier capitalized? I don't think so.
- rco8786 10y ago
  statistics.
- Ntrails 10y ago
  >Enough for what? Why not? Let me throw out some made up numbers as an example. Company X hires 100 people out of 500 applicants. If only 30% of them perform well, you might argue the interview process is rubbish. However, if you can test and show that of the "un-hired" 400 only 5% would perform well, then clearly the interview process is better than picking at random. This is why just looking at hired performance is not indicative of whether the process works - because the task was not just to predict good performers but to predict a better rate of good performers than a random sample of applicants.
dominotw 10y ago
>One of the open secrets of working in technology is that technical interviews are completely worthless as a predictor for whether someone is a good hire or not. This is wrong. Apparently google has internally done much research and their interview techniques have strong collreation to success on the job.
- Bartweiss 10y ago
  Both "they work" and "they don't work" are unsupportably broad. Google has substantially reworked their interviewing process on at least one occasion, producing a big change in results. Story here: http://www.nytimes.com/2013/06/20/business/in-head-hunting-big-data-may-not-be-such-a-big-deal.html?pagewanted=all&_r=1 http://www.nytimes.com/2013/06/20/business/in-head-hunting-b... Meanwhile it's crucial and nigh-impossible to identify how much success is based on interview performance as opposed to selection effects in who interviews, and how interviewers see different candidates.
- mercutio2 10y ago
  I'm a big fan of Dare's writing, but on this one, I think he missed the mark wildly, for exactly the reason you cite: Google's research says almost the exact opposite of what he's claiming in this post. With that said, the large variation in ratios of disadvantaged minority employment at big tech companies does seem meaningful (and he cites that wide variation). I think intern and HR pipeline variation is likely very relevant to this difference. The problem is with his implication that throwing darts to pick from a pool of moderately qualified people would yield a reasonable result.
- Kinnard 10y ago
  Do you have a source with some of google's research and results on this matter?
chewyshine 10y ago
It is possible. Check out the difference between a concurrent and predictive validity study. Also, range restriction corrections can approximate the unrestricted population variance and validity coefficient.

by feral 10y ago

vonmoltke 10y ago

> The fact [the scores of someone who gets hired] are not predictive of their eventual performance, does not mean that hiring process is completely random, or worthless. It does not automatically mean this, but it is a strong indicator. If the output of the classifier is not correlated with a given output sample it is reasonable to assume, in the absence of other data, that it is uncorrelated for other samples as well.

feral 10y ago
I do accept it is evidence - but I don't think I would say a strong indicator, and it doesn't justify the strong conclusions made that the process is completely arbitrary. - It's imbalanced classification: there are going to be more rejections than acceptances; so you have less statistical power, to find if there is a real relationship, when you just analyse the acceptances. Even apart from that, class imbalance can unintuitively affect post-hoc inferences. - Scores are in context of the other candidates at a given level, and stage in the interview process, so are fundamentally already residual in some sense. - I'm willing to bet, from hiring I've seen, that scores are distributed somewhat about the 'hire/no-hire' bar, and that extremes are rare - the process is going to be tuned, implicitly in the organisation, to make that 'hire/no-hire' decision, rather than predict performance. > it is reasonable to assume, in the absence of other data, that it is uncorrelated for other samples as well I think that's a very strong statement: it's not a random sample, its explicitly concentrated at one end of the distribution. I do accept it is evidence, just not near enough to justify the conclusion the whole process is arbitrary.
statics 10y ago
No, it's not a strong indicator that the process is random or worthless. That makes the assumption that the people rejected by the process would have scored just as well. It could be very good at determining who has the technical skills to work there. It might be that their performance scoring of hires is measuring something else- like how much their manager likes them.
- vonmoltke 10y ago
  It could, but if technical skill is uncorrelated with performance scoring it will be really hard to determine how many people who make it through the process actually have the technical skill and how many don't.

rando289 10y ago

> it's not enough to just look within the 'hire' cohort. Enough for what? Why not? They specifically have a "score", not just a hire / no hire decision. Sure it would be better, but you don't make any convincing argument why their analysis is not valuable. > designed to be a Classifier, typically tuned with an emphasis on having a low false positive rate. classifier of what? Why is classifier capitalized? I don't think so.

rco8786 10y ago
statistics.
Ntrails 10y ago
>Enough for what? Why not? Let me throw out some made up numbers as an example. Company X hires 100 people out of 500 applicants. If only 30% of them perform well, you might argue the interview process is rubbish. However, if you can test and show that of the "un-hired" 400 only 5% would perform well, then clearly the interview process is better than picking at random. This is why just looking at hired performance is not indicative of whether the process works - because the task was not just to predict good performers but to predict a better rate of good performers than a random sample of applicants.

dominotw 10y ago

>One of the open secrets of working in technology is that technical interviews are completely worthless as a predictor for whether someone is a good hire or not. This is wrong. Apparently google has internally done much research and their interview techniques have strong collreation to success on the job.

Bartweiss 10y ago
Both "they work" and "they don't work" are unsupportably broad. Google has substantially reworked their interviewing process on at least one occasion, producing a big change in results. Story here: http://www.nytimes.com/2013/06/20/business/in-head-hunting-big-data-may-not-be-such-a-big-deal.html?pagewanted=all&_r=1 http://www.nytimes.com/2013/06/20/business/in-head-hunting-b... Meanwhile it's crucial and nigh-impossible to identify how much success is based on interview performance as opposed to selection effects in who interviews, and how interviewers see different candidates.
mercutio2 10y ago
I'm a big fan of Dare's writing, but on this one, I think he missed the mark wildly, for exactly the reason you cite: Google's research says almost the exact opposite of what he's claiming in this post. With that said, the large variation in ratios of disadvantaged minority employment at big tech companies does seem meaningful (and he cites that wide variation). I think intern and HR pipeline variation is likely very relevant to this difference. The problem is with his implication that throwing darts to pick from a pool of moderately qualified people would yield a reasonable result.
Kinnard 10y ago
Do you have a source with some of google's research and results on this matter?

chewyshine 10y ago

It is possible. Check out the difference between a concurrent and predictive validity study. Also, range restriction corrections can approximate the unrestricted population variance and validity coefficient.