7 ms·

(Tedious disclaimer: my opinion only, not speaking for anybody else. I'm an SRE at Google.) We expect and accept a high false-negative rate. Our interview proc

(Tedious disclaimer: my opinion only, not speaking for anybody else. I'm an SRE at Google.) We expect and accept a high false-negative rate. Our interview process is optimised for zero false-positives at the cost of many false-negatives. This is a deliberate choice. So yes, I would expect to see a significant rate of rejections of people who are clearly qualified. The sort of people that we want to hire are likely to come back for another try anyway, and the long-term effect of this process seems to be doing what it was supposed to.

joosters 10y ago
I've often heard this, and it seems a poor excuse for a terrible interview process/terrible hiring figures. Why do you not try to improve your interview process to lower the false-negative rate? Doing so in an intelligent way need not raise the false-positives rate (unless you somehow believe that the interview process is already perfected by Google...)
- evanws 10y ago
  They definitely try their best to give every candidate as fair a shot as possible, but there's always going to be error. It becomes a question of what type of error hurts more, accidentally rejecting someone who's qualified or accepting someone who isn't. They reasonably believe the latter is significantly more dangerous.
- pablasso 10y ago
  Oh, I can play this game too: Why don't we make software without bugs? if you do software in an intelligent way there's no need to create bugs. Of course everyone strives for better processes —at the end of the day, that means more revenue— is just that those problems seem easier to solve from the outside.
  - taspeotis 10y ago
    As an extreme example: NASA has done a pretty good job of making software without bugs [1]. Consider these stats : the last three versions of the program — each 420,000 lines long — had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors. [1] https://www.fastcompany.com/28121/they-write-right-stuff https://www.fastcompany.com/28121/they-write-right-stuff
    - hyperpape 10y ago
      Keep going, why don't we all write software the way NASA does? (Also, it's just an analogy. I think perfect hiring would be much harder than bug free software).
      - na85 10y ago
        Because companies are interested in profit; not top-quality software. Anyone who tells you differently is deluded or lying.
      - taspeotis 10y ago
        That's why it's an extreme example. I don't know if it's possible for anyone to answer this question but: does any organisation have a hiring process that's 5000:1 better than the average equivalent, like NASA's 5000:1 software defect rate?
    - alexeiz 10y ago
      I wouldn't be surprised if NASA had a different definition of "error" than a typical "bug" in commercial software. My video player plays every single mpeg/avi/mkv I through at it. Yet it has dozens if not hundreds of serious bugs. I just doubt those bugs fall under the NASA's definition of "error".
- asuffield 10y ago
  I am not allowed to share our data with you about how well the process is performing. But I would like to point out that of the two of us, only the one who doesn't know is suggesting that we have "terrible hiring figures" or a "terrible interview process".
  - asuffield 10y ago
    Hi, downvoters who aren't commenting. What exactly is your objection to this post?
    - PhantomGremlin 10y ago
      I didn't downvote, but I'll take a shot at answering. Your post had a touch of condescension, sort of: "well I work at Google and you don't, so I obviously know better than you ...". Kind of reminds me of this HN comment and rejoinder: https://news.ycombinator.com/item?id=35079 https://news.ycombinator.com/item?id=35079 Did you win the Putnam? Yes, I did. You generally don't know the history of the people you're interacting with here on HN, so it's not always appropriate to act so smug about where you work.
      - mwpmaybe 10y ago
        That thread is amazing. My favorite part is the inventors of Tarsnap and Dropbox basically saying to each other, "oh hey, I'm working on the same thing as you!"
        cperciva 10y ago
        That's a bit of a mischaracterization. Drew said that he was working on the same thing as me; I didn't respond, because my response would have been "no, you're not; I'm building a secure backup tool" and I didn't want to antagonize people any further.
    - ocdtrekkie 10y ago
      Basically "just take my word for it, I'm at Google".
eridius 10y ago
Why are you optimizing for zero false-positives? That's what I'd expect from an early startup, where every employee is critical, not from a large company.
- hetman 10y ago
  What exactly do they lose by doing this? I doubt they have a shortage of applicants.
  - wott 10y ago
    > What exactly do they lose by doing this? A huge amount of time/energy/money wasted in interviewing way too many people in a way too deep recruiting process.
  - YeGoblynQueenne 10y ago
    The point is that google is trying to hire "only the best". Let's say that "the best" are 1% of applicants (to make it simple). Now, imagine that google's interview process, optimised to reduce the false positive rate [1] to 0% as it purportedly is, rejects 10% of applicants that should be hired (i.e. it has a 10% "false negative rate"). How would you guarantee that this rejected 10% does not include the 1% that are "the best"? You can't find out because you've already ditched them, so you can't exactly compare them to the ones you hired. You can find out which of the ones you hired are "the best" but only compared to your other hires. There's no guarantee that you don't end up hiring mediocre people, just by consistently failing to hire the actual best every time. How likely is it that you'll ditch the 1% by chance? If you consistently reject 10% of candidates you actually should hire, then it's one out of ten, I'd say. So it depends on how high is google's "false negative rate". If it's as high as 50% they may well end up rejecting half of the people they're trying to hire. The google SRE user above mentions "many false-negatives". That sounds like worse than 50%. So, to answer your question with another question: what happens if you consistently miss most of the group you are trying to hire, week after week? _______________________ [1] Normally people look at true positive rate and true negative rate, the former being the proportion of all positive results that are correct, and accordingly for the latter. "False positive" is just the complement of "true positive". Also, note that a process may have a high TPR and high TNR at the same time, so a high TNR on its own is no guarantee of a good-quality process.
joewee 10y ago
This explanation summarizes why tech companies lack diversity. Sampling bias at its best. This logic is a smart persons way of saying you only want to hire people similar to those you've already hired.
- asuffield 10y ago
  All I am free to say about this is that we look hard at diversity issues in hiring and put a lot of effort into eliminating them, and that I personally believe we do a better job of stamping this out than any other company I have worked for in my career. I suspect, but cannot prove, that our process gives better diversity results than most of the ideas which are "popular" on HN at present. I do some work on this personally. I'm just not allowed to discuss the details at present. (The reasons why I'm not allowed to discuss this are due to tedious bureaucracy, not anything interesting. I have tried to get that changed, but it would require more effort than I am willing to expend on a problem that will go away in time.)
  - joewee 10y ago
    I perform a lot of data analysis and separately I've been involved in hiring decisions. I've seen two common issues: 1) humans make decisions based on statistically insignificant sample sizes 2) we are incapable of identifying our biases without data. Even when large organization do look at the data, they are reluctant to share it, even internally. I applaud your efforts and hope are tracking the results.
  - YeGoblynQueenne 10y ago
    It shouldn't be very hard to do well in diversity, however you choose to define it. For instance, say you notice your workforce is strongly biased against one population group, like female software engineers [1]. In that case: a) Set a target for the proportion of female software engineers you want to hire. b) Stop hiring male engineers at the point where hiring more would make you miss your female-engineer target. c) Keep hiring only female engineers until you hit your target. In principle, that might cause some concern among male engineers who could feel discriminated against. In practice however, google and all other tech companies are already employing that process, except they do so informally (one hopes) and the groups they hire for are not the ones usually included in "diversity"- for instance, according to [1] 72% of tech workers at google are male, vs 50% ish in the general population. Additionally, when it comes to google specifically, recruiters are supposed to actively go after "the best", so it shouldn't be a problem for that company in particular to go after the best female people. In fact, that google targets its hires and yet it ends up with a strong bias towards a specific kind of engineer is a very good example of how not to do diversity. [1] http://www.theverge.com/2015/8/20/9179853/tech-diversity-scorecard-apple-google-microsoft-facebook-intel-twitter-amazon http://www.theverge.com/2015/8/20/9179853/tech-diversity-sco...
Terr_ 10y ago
> The sort of people that we want to hire are likely to come back for another try anyway Starry-eyed kids with no sense of ownership to root them anywhere else, always ready to study and jump ship every 6 months? :p
aluminussoma 10y ago
Thanks for the response. From the replies, it is evident that this subject touches some raw nerves! I wanted to run that small experiment as a way of showing that the shortage of engineers is overstated (a common refrain in the H1-B visa debates). I am not sure if Google is one of the companies that says they can't find qualified engineers. Many peer companies to Google do say that. Having a very high false negative rate is something that should be taken into account with the current visa debates.
YeGoblynQueenne 10y ago
>> Our interview process is optimised for zero false-positives at the cost of many false-negatives. Why are false positives worse than false negatives? Optimising for either exclusively sounds like a very good way to fill up with mediocre people: either you miss too many of the best (what you do), or you hit too many of the worst (what you try to avoid doing). Of course, there's no hard-and-fast rule about the quality of a hire. You can hope to have zero false-positives, but you can't really count on it. What you should aim to optimise is the ratio of the people you really wanted to hire over the people you actually hired. This would allow you to improve the quality of your hires over time so that it approaches the high point of some measure of goodness. Realistically speaking, that's the best result you can expect to achieve. Any tactic that purports to give a better outcome (zero false negatives? Really?) should be regarded with suspicion.
firephreek 10y ago
That just makes Google sound risk averse. Given some of their business models and decisions over the last 5 years, I don't think that's the case. Given the quality of those products over the last 5 years, I might be inclined to question the nature and/or quality of the engineers being hired. Or maybe it's further up the ladder?
hkkWrites 10y ago
I say that false positives in recruitment are quite bad for any company, not just Google. But I don't agree on how Google (and others) go about preventing false positives. If Google is so worried about false positives, why don't they just ask much harder questions in interviews ? If the hard questions are solved, Google should proceed for recruitment and reject otherwise. No candidate feels puzzled by a rejection, since they know they failed too. I'd say the questions used by Google can be solved (or expect to be solved) by most of the CS grads coming out of top 50 world's universities and also various coding competitions/ top coder etc. The fact is that the output of those universities far exceeds what Google wants to recruit in a year. Yet Google complains on not finding enough talent ! Please learn to convert the false negatives to clear cut negatives while also preventing false positives. Why does Google set up multi-layered committees post interview to gloss over the interview discussions (actual interviewers not included) and then take decisions based on some fuzzy undisclosed logic ?

joosters 10y ago

I've often heard this, and it seems a poor excuse for a terrible interview process/terrible hiring figures. Why do you not try to improve your interview process to lower the false-negative rate? Doing so in an intelligent way need not raise the false-positives rate (unless you somehow believe that the interview process is already perfected by Google...)

evanws 10y ago
They definitely try their best to give every candidate as fair a shot as possible, but there's always going to be error. It becomes a question of what type of error hurts more, accidentally rejecting someone who's qualified or accepting someone who isn't. They reasonably believe the latter is significantly more dangerous.
pablasso 10y ago
Oh, I can play this game too: Why don't we make software without bugs? if you do software in an intelligent way there's no need to create bugs. Of course everyone strives for better processes —at the end of the day, that means more revenue— is just that those problems seem easier to solve from the outside.
- taspeotis 10y ago
  As an extreme example: NASA has done a pretty good job of making software without bugs [1]. Consider these stats : the last three versions of the program — each 420,000 lines long — had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors. [1] https://www.fastcompany.com/28121/they-write-right-stuff https://www.fastcompany.com/28121/they-write-right-stuff
  - hyperpape 10y ago
    Keep going, why don't we all write software the way NASA does? (Also, it's just an analogy. I think perfect hiring would be much harder than bug free software).
    - na85 10y ago
      Because companies are interested in profit; not top-quality software. Anyone who tells you differently is deluded or lying.
    - taspeotis 10y ago
      That's why it's an extreme example. I don't know if it's possible for anyone to answer this question but: does any organisation have a hiring process that's 5000:1 better than the average equivalent, like NASA's 5000:1 software defect rate?
  - alexeiz 10y ago
    I wouldn't be surprised if NASA had a different definition of "error" than a typical "bug" in commercial software. My video player plays every single mpeg/avi/mkv I through at it. Yet it has dozens if not hundreds of serious bugs. I just doubt those bugs fall under the NASA's definition of "error".
asuffield 10y ago
I am not allowed to share our data with you about how well the process is performing. But I would like to point out that of the two of us, only the one who doesn't know is suggesting that we have "terrible hiring figures" or a "terrible interview process".
- asuffield 10y ago
  Hi, downvoters who aren't commenting. What exactly is your objection to this post?
  - PhantomGremlin 10y ago
    I didn't downvote, but I'll take a shot at answering. Your post had a touch of condescension, sort of: "well I work at Google and you don't, so I obviously know better than you ...". Kind of reminds me of this HN comment and rejoinder: https://news.ycombinator.com/item?id=35079 https://news.ycombinator.com/item?id=35079 Did you win the Putnam? Yes, I did. You generally don't know the history of the people you're interacting with here on HN, so it's not always appropriate to act so smug about where you work.
    - mwpmaybe 10y ago
      That thread is amazing. My favorite part is the inventors of Tarsnap and Dropbox basically saying to each other, "oh hey, I'm working on the same thing as you!"
      - cperciva 10y ago
        That's a bit of a mischaracterization. Drew said that he was working on the same thing as me; I didn't respond, because my response would have been "no, you're not; I'm building a secure backup tool" and I didn't want to antagonize people any further.
  - ocdtrekkie 10y ago
    Basically "just take my word for it, I'm at Google".

eridius 10y ago

Why are you optimizing for zero false-positives? That's what I'd expect from an early startup, where every employee is critical, not from a large company.

hetman 10y ago
What exactly do they lose by doing this? I doubt they have a shortage of applicants.
- wott 10y ago
  > What exactly do they lose by doing this? A huge amount of time/energy/money wasted in interviewing way too many people in a way too deep recruiting process.
- YeGoblynQueenne 10y ago
  The point is that google is trying to hire "only the best". Let's say that "the best" are 1% of applicants (to make it simple). Now, imagine that google's interview process, optimised to reduce the false positive rate [1] to 0% as it purportedly is, rejects 10% of applicants that should be hired (i.e. it has a 10% "false negative rate"). How would you guarantee that this rejected 10% does not include the 1% that are "the best"? You can't find out because you've already ditched them, so you can't exactly compare them to the ones you hired. You can find out which of the ones you hired are "the best" but only compared to your other hires. There's no guarantee that you don't end up hiring mediocre people, just by consistently failing to hire the actual best every time. How likely is it that you'll ditch the 1% by chance? If you consistently reject 10% of candidates you actually should hire, then it's one out of ten, I'd say. So it depends on how high is google's "false negative rate". If it's as high as 50% they may well end up rejecting half of the people they're trying to hire. The google SRE user above mentions "many false-negatives". That sounds like worse than 50%. So, to answer your question with another question: what happens if you consistently miss most of the group you are trying to hire, week after week? _______________________ [1] Normally people look at true positive rate and true negative rate, the former being the proportion of all positive results that are correct, and accordingly for the latter. "False positive" is just the complement of "true positive". Also, note that a process may have a high TPR and high TNR at the same time, so a high TNR on its own is no guarantee of a good-quality process.

joewee 10y ago

This explanation summarizes why tech companies lack diversity. Sampling bias at its best. This logic is a smart persons way of saying you only want to hire people similar to those you've already hired.

asuffield 10y ago
All I am free to say about this is that we look hard at diversity issues in hiring and put a lot of effort into eliminating them, and that I personally believe we do a better job of stamping this out than any other company I have worked for in my career. I suspect, but cannot prove, that our process gives better diversity results than most of the ideas which are "popular" on HN at present. I do some work on this personally. I'm just not allowed to discuss the details at present. (The reasons why I'm not allowed to discuss this are due to tedious bureaucracy, not anything interesting. I have tried to get that changed, but it would require more effort than I am willing to expend on a problem that will go away in time.)
- joewee 10y ago
  I perform a lot of data analysis and separately I've been involved in hiring decisions. I've seen two common issues: 1) humans make decisions based on statistically insignificant sample sizes 2) we are incapable of identifying our biases without data. Even when large organization do look at the data, they are reluctant to share it, even internally. I applaud your efforts and hope are tracking the results.
- YeGoblynQueenne 10y ago
  It shouldn't be very hard to do well in diversity, however you choose to define it. For instance, say you notice your workforce is strongly biased against one population group, like female software engineers [1]. In that case: a) Set a target for the proportion of female software engineers you want to hire. b) Stop hiring male engineers at the point where hiring more would make you miss your female-engineer target. c) Keep hiring only female engineers until you hit your target. In principle, that might cause some concern among male engineers who could feel discriminated against. In practice however, google and all other tech companies are already employing that process, except they do so informally (one hopes) and the groups they hire for are not the ones usually included in "diversity"- for instance, according to [1] 72% of tech workers at google are male, vs 50% ish in the general population. Additionally, when it comes to google specifically, recruiters are supposed to actively go after "the best", so it shouldn't be a problem for that company in particular to go after the best female people. In fact, that google targets its hires and yet it ends up with a strong bias towards a specific kind of engineer is a very good example of how not to do diversity. [1] http://www.theverge.com/2015/8/20/9179853/tech-diversity-scorecard-apple-google-microsoft-facebook-intel-twitter-amazon http://www.theverge.com/2015/8/20/9179853/tech-diversity-sco...

Terr_ 10y ago

> The sort of people that we want to hire are likely to come back for another try anyway Starry-eyed kids with no sense of ownership to root them anywhere else, always ready to study and jump ship every 6 months? :p

aluminussoma 10y ago

Thanks for the response. From the replies, it is evident that this subject touches some raw nerves! I wanted to run that small experiment as a way of showing that the shortage of engineers is overstated (a common refrain in the H1-B visa debates). I am not sure if Google is one of the companies that says they can't find qualified engineers. Many peer companies to Google do say that. Having a very high false negative rate is something that should be taken into account with the current visa debates.

YeGoblynQueenne 10y ago

>> Our interview process is optimised for zero false-positives at the cost of many false-negatives. Why are false positives worse than false negatives? Optimising for either exclusively sounds like a very good way to fill up with mediocre people: either you miss too many of the best (what you do), or you hit too many of the worst (what you try to avoid doing). Of course, there's no hard-and-fast rule about the quality of a hire. You can hope to have zero false-positives, but you can't really count on it. What you should aim to optimise is the ratio of the people you really wanted to hire over the people you actually hired. This would allow you to improve the quality of your hires over time so that it approaches the high point of some measure of goodness. Realistically speaking, that's the best result you can expect to achieve. Any tactic that purports to give a better outcome (zero false negatives? Really?) should be regarded with suspicion.

firephreek 10y ago

That just makes Google sound risk averse. Given some of their business models and decisions over the last 5 years, I don't think that's the case. Given the quality of those products over the last 5 years, I might be inclined to question the nature and/or quality of the engineers being hired. Or maybe it's further up the ladder?

hkkWrites 10y ago

I say that false positives in recruitment are quite bad for any company, not just Google. But I don't agree on how Google (and others) go about preventing false positives. If Google is so worried about false positives, why don't they just ask much harder questions in interviews ? If the hard questions are solved, Google should proceed for recruitment and reject otherwise. No candidate feels puzzled by a rejection, since they know they failed too. I'd say the questions used by Google can be solved (or expect to be solved) by most of the CS grads coming out of top 50 world's universities and also various coding competitions/ top coder etc. The fact is that the output of those universities far exceeds what Google wants to recruit in a year. Yet Google complains on not finding enough talent ! Please learn to convert the false negatives to clear cut negatives while also preventing false positives. Why does Google set up multi-layered committees post interview to gloss over the interview discussions (actual interviewers not included) and then take decisions based on some fuzzy undisclosed logic ?