12 ms·

Think of the problem from Google's perspective though. At some point, you have tens of thousands of candidates and you need a system to quantify how good they a

Think of the problem from Google's perspective though. At some point, you have tens of thousands of candidates and you need a system to quantify how good they are. Further, it's reasonable to have false negatives (people you don't hire that should have been hired) but really bad to have false positives (people that you hire that you should not have). Together, these boil down into the de facto whiteboarding interview process.

aaggarwal 11y ago
Yes, exactly, since it's Google, they can afford to have strong filters which will definitely filter out a bad hire, may be at the expense of rejecting a good candidate. Their policy might expect good candidate to get absorbed in Google one way or other, but they definitely want to filter a bad hire. However, in the recent years, I have seen this to be not working as expected. Still, I don't think it matters to Google as an big organization.
- waterlesscloud 11y ago
  "they can afford to have strong filters which will definitely filter out a bad hire, may be at the expense of rejecting a good candidate" Can they, though? Obviously they believe they can, but are they correct? Google's not going away any time soon, but they could easily slip a long way down from the top of the heap. There are signs that's started to happen. I don't think they can be nearly as casual about losing top candidates as they seem to believe. In five or ten years we'll know, I suppose.
  - aaggarwal 11y ago
    In the long run, this could affect Google, but, then again, if they change their hiring policies for better even after long 5 years, the best talent that might had been rejected before would still go to work at Google. So, still I don't think Google really consider it as an issue for them. Though, I have been burned once myself by them.
    - loopbit 11y ago
      I don't consider myself "top talent", so I won't comment on my own experience through the interview process. But I do know that I don't want to work for google. Hell, I'm happier doing my own thing. I also know a few "top talent" engineers that rejected the google offer because they didn't like the process and the people they met. I know even more ex-Googlers that quit within the year of starting there. It's not an issue for them? From where I'm looking, most of the experienced top talent (that it's not working at google already) is keeping away from it, in greener pastures and most won't look back. Yep, google is not going anywhere anytime soon, but I'd be just a bit concerned if I were them.
    - eva1984 11y ago
      Google is not the Google it was. I mean I still consider that working for Google is attractive, but things I get from tens of my friends who work there are to say the best....mixed. Google might still have the vigor and excitement at its core, but unless you are an already exceptional engineer, the work you get assigned there, often times, ranges from bland to boring. Like I said, Google is still an amazing company, but it is by no means the only one out there. Talents won't come to work for it just because it is Google, without great compensation and interesting projects.
- kamaal 11y ago
  >>However, in the recent years, I have seen this to be not working as expected. It never worked, what do you think happens when you start asking these kind of questions in interviews? People just go out and spend insane amount of time memorizing solutions to interview questions on the internet. They continue the same after you have hired them. So their on the job productivity remains low. Please note, they need to prepare for their next job in the current one. So the only thing this results is in people spending insane amount of time preparing interviews doing very little work in their day jobs.
  - fsk 11y ago
    I'd rather spend a year learning Python or Go or whatever, instead of spending a year practicing interview questions. I never realized that some people put as much effort into practicing interview questions as I put into learning how to actually write software. Does this mean technical interviews, even a structured interview, are completely useless? I.e., if you follow an interview script, eventually candidates find out about your script, and then it's useless.
- lavamantis 11y ago
  "filter out a bad hire" Good teams have a balanced skill sets I've found. You need some uber-algorithm guys who can pass questions like this, but you also need engineers who can fill in all the rest of the tasks that make up a production worthy system. I have not found that the uber-algorithm guys always pay attention to detail, put in good comments, think of all the edge cases, communicate well with outside teams, etc. Those types of engineers are what you consider a "bad hire", and won't be working at Google.
  - unoti 11y ago
    > you also need engineers who can fill in all the rest of the tasks that make up a production worthy system. I have not found that the uber-algorithm guys always pay attention to detail, put in good comments, think of all the edge cases, communicate well with outside teams, etc. What's to keep a good engineer like you describe from buying a couple of books and spending a couple of months working on becoming the type of person that can do well on these questions?
    - laurencerowe 11y ago
      There is a glut of interesting opportunities out there. For me personally, there are just many more interesting ways to spend my time than revising the CS classes I took almost 15 years ago.
    - msandford 11y ago
      > What's to keep a good engineer like you describe from buying a couple of books and spending a couple of months working on becoming the type of person that can do well on these questions? Google doesn't make everyone multi-millionaires anymore. If you want me to jump through a bunch of hoops for you, you'd better have a pretty solid case for what the outcome will be. By actively selecting against people who don't want to answer bullshit questions, they're definitely turning away creative, talented people. They're selecting for different people instead. Perhaps that's what Google needs to feed the machine now that the company is mature, so I won't necessarily fault them for it. But that filtering or selection IS happening, even if they don't think it is.
falcolas 11y ago
If only whiteboarding interview processes actually weeded out false positives. In practice, they select for people with good memorization skills. If you can remember the details on a ten dozen different algorithms and data structures, you can pass one of these without having a single lick of creativity or skill. I say this as an employee who has worked alongside many unskilled drones who made it past the algorithmic interview process.
- chrismsnz 11y ago
  That's not what the interview process is like at all. They're more interested in how you approach real world issues (the questions I got asked were conceivably real-life issues a company like Google would face with its products). If you can solve that issue by applying efficient and well understood data structures and algorithms, then that indicates you understand the problem space and solutions that may apply. It's not like they get you in a room and ask you to draw a linked list or a binary tree.
  - falcolas 11y ago
    > It's not like they get you in a room and ask you to draw a linked list or a binary tree. And yet the OP was turned down for not being able to invert a binary tree (based on limited information, etc etc.)? What do you feel that is, if not a "whiteboard this algorithm" question? As for many of the other companies who try and emulate Google - this is exactly what they do.
    - chrismsnz 11y ago
      > And yet the OP was turned down for not being able to invert a binary tree Your source is a series of salty tweets.
      - falcolas 11y ago
        Well, for what it's worth, it's all we have to go on at the moment. And in either case, my original comment was directed at the idea that "algorithmic interviews provide false negatives, but not false positives".
PhasmaFelis 11y ago
It's true that evaluating candidates is a hard problem, but you'd think that at the very least they would try to evaluate them based on skills that will actually be used in the job. Judging programmers based on their whiteboard skills is like judging a ballet dancer based on how eloquently he can describe his training routine.
RogerL 11y ago
No, it doesn't. Say I pass the invert a tree question. Does that prove I am good? That I can design a product, listen to customer requirements, come up with ideas that elude others, read a research paper and turn it into something commercial, solder a circuit board, make a schedule, mentor junior engineers, write documentation? No. All inverting a tree tells them is I studied trees recently, and/or I'm at least superficially clever. Google lets go of plenty of people. They aren't making perfect hires.
- hinkley 11y ago
  They have publicly admitted there's no correlation between their hiring process and outcomes. I'm not even clear [why] we're still debating this. Other than to wonder why they still haven't done anything about it. (I interviewed there ~8 years ago, pre Android when they were looking for deep mobile expertise and got flunked out on a similar problem. I am not at all interested in working for them now)
  - alexfernandez 11y ago
    To show that there is a correlation you have to quantify both hiring process evaluation and job performance. Since both are elusive quantities that are hard to quantify, showing there is no correlation may just mean that measurements are done poorly. E.g. when job success is measured by boss satisfaction.
- reagency 11y ago
  None of the things you describe are part of the job for a rank and file bigco engineer. They already have filled all the leadership roles.
chrischen 11y ago
Problem is if your system is actually capable measuring up to the claimed degree of accuracy, or just stabbing in the dark after a certain point. IQ tests supposedly become inaccurate after 150/160, so you wouldn't really want to put too much emphasis on 180 vs 170 at those levels.
fsk 11y ago
This has got to be the biggest hiring fallacy I've ever heard. "It's better to reject a good candidate than hire a bad candidate." That's completely false, and anyone who says that is completely ignorant of Bayesian logic. Here are some simple numbers. Suppose that a "good" candidate is a 1-in-100 find. Suppose that a "bad" candidate has a 1% chance of tricking you into hiring them anyway. Every time you pass on a "good" candidate, that is greater opportunity for a "bad" candidate to trick you into hiring them! Counterintuitively, if you pass on too many of the "good" candidates, in your overzealousness to reject bad candidates, you're actually INCREASING THE ODDS OF A BAD HIRE. This is management porn. "It's better to reject a good candidate than hire a bad candidate." It makes the manager feel good. "Wow! That candidate seemed smart, but I rejected him anyway! I'm such a great leader! I make the tough decisions!" tl;dr summary Because "good" candidates are rare, every time you pass on a good candidate that increases your odds of making a bad hire! This is simple Bayesian reasoning!
- ricardobeat 11y ago
  This is not about numbers. A bad hire can easily cause more friction than if nobody was hired at all. Net effect for the business is negative. You're also probably severely overestimating the likelihood of a bad hire going through.
  - liberatus 11y ago
    Doesn't understand bayesian logic! Doesn't understand risk management!
    - fsk 11y ago
      Idiot: We don't want to make any bad hires! Therefore, we reject a lot of good candidates! FSK: If you pass on many good candidates, and you have a small chance of hiring any given bad candidate, then each good candidate you reject actually INCREASES your odds of making a bad hire. Idiot: We made a bad hire once! Never again! Now we reject lots of good candidates to avoid that repeat disaster! FSK: But, if you want to minimize your bad hire rate, you also have to minimize the number of good candidates you reject. Idiot: NO! NO! NO! The way you make sure you hire no bad candidates is to be so strict that you reject lots of good ones! That's what everyone told me so it must be right! In one ear and out the other. Why do people who don't understand statistics get to be managers? If you don't understand this statistics argument I made, you're unqualified to work in any sort of technical area.
mark-r 11y ago
"Google's perspective" shouldn't be a monolithic thing. Sure if you're sorting through candidates that were dropped on you then this process makes sense. But if you've discovered someone that by reputation and demonstrated code base has already excelled, and you're actively chasing them, then the interview process for that individual has to change. It's more about seeing if they'll thrive in your environment, and convincing them that you're an inevitable part of their career.
smsm42 11y ago
> you need a system to quantify how good they are But do you? How catastrophical would it be if Google accidentally hired not the top 0.001% but instead somebody from top 0.01%? I mean, same people that build everything else around us and the world still didn't collapse yet? I get it, you have to have standards, and you don't want to waste time and money on somebody that can't pass fizzbuzz test. But once you've moved past that - do you need to obsess so hard on ranking and quantifying it? There are probably some positions in Google that require people to invert binary trees (speaking figuratively) day in and day out, but my decades of experience in the industry shows that most positions aren't like that. I've not worked for google but I'm pretty sure it's also this way there. If you hire smart and competent people, it's OK even if they don't reverse a binary tree here or there.
Corrado 11y ago
Well, my suggestion would be to use Page Rank to sift through the resumes. If you are respected by respected computer programmers then your "score" goes up. They could measure that by mentions in a blog, twitter "conversations", Github commits on the same project, etc. "Hmmm... it looks like this guy really gets around. He's on the same project at GoodProgrammer A and GoodProgrammer B. He also tweets about functional programming and was mentioned in GoodProgrammer C's latest blog."
samkone 11y ago
This cracks me up a bit. I don't have anything against whiteboarding, even though I find it meaningless. With this you more often than not have false positives. And if Google had so many resumes they have to filter through, then why do they have all these recruiters literally spamming you to join? they have recruiting issues like many others tech companies, but the whiteboard is really a bad filter. I've seen very much capable,competent and experienced engineers being denied just because they didn't spend too much time learning cute algorithms or coding in the whiteboard, in front of interviewers sometimes more focused in showing off how smart they are.

aaggarwal 11y ago

Yes, exactly, since it's Google, they can afford to have strong filters which will definitely filter out a bad hire, may be at the expense of rejecting a good candidate. Their policy might expect good candidate to get absorbed in Google one way or other, but they definitely want to filter a bad hire. However, in the recent years, I have seen this to be not working as expected. Still, I don't think it matters to Google as an big organization.

waterlesscloud 11y ago
"they can afford to have strong filters which will definitely filter out a bad hire, may be at the expense of rejecting a good candidate" Can they, though? Obviously they believe they can, but are they correct? Google's not going away any time soon, but they could easily slip a long way down from the top of the heap. There are signs that's started to happen. I don't think they can be nearly as casual about losing top candidates as they seem to believe. In five or ten years we'll know, I suppose.
- aaggarwal 11y ago
  In the long run, this could affect Google, but, then again, if they change their hiring policies for better even after long 5 years, the best talent that might had been rejected before would still go to work at Google. So, still I don't think Google really consider it as an issue for them. Though, I have been burned once myself by them.
  - loopbit 11y ago
    I don't consider myself "top talent", so I won't comment on my own experience through the interview process. But I do know that I don't want to work for google. Hell, I'm happier doing my own thing. I also know a few "top talent" engineers that rejected the google offer because they didn't like the process and the people they met. I know even more ex-Googlers that quit within the year of starting there. It's not an issue for them? From where I'm looking, most of the experienced top talent (that it's not working at google already) is keeping away from it, in greener pastures and most won't look back. Yep, google is not going anywhere anytime soon, but I'd be just a bit concerned if I were them.
  - eva1984 11y ago
    Google is not the Google it was. I mean I still consider that working for Google is attractive, but things I get from tens of my friends who work there are to say the best....mixed. Google might still have the vigor and excitement at its core, but unless you are an already exceptional engineer, the work you get assigned there, often times, ranges from bland to boring. Like I said, Google is still an amazing company, but it is by no means the only one out there. Talents won't come to work for it just because it is Google, without great compensation and interesting projects.
kamaal 11y ago
>>However, in the recent years, I have seen this to be not working as expected. It never worked, what do you think happens when you start asking these kind of questions in interviews? People just go out and spend insane amount of time memorizing solutions to interview questions on the internet. They continue the same after you have hired them. So their on the job productivity remains low. Please note, they need to prepare for their next job in the current one. So the only thing this results is in people spending insane amount of time preparing interviews doing very little work in their day jobs.
- fsk 11y ago
  I'd rather spend a year learning Python or Go or whatever, instead of spending a year practicing interview questions. I never realized that some people put as much effort into practicing interview questions as I put into learning how to actually write software. Does this mean technical interviews, even a structured interview, are completely useless? I.e., if you follow an interview script, eventually candidates find out about your script, and then it's useless.
lavamantis 11y ago
"filter out a bad hire" Good teams have a balanced skill sets I've found. You need some uber-algorithm guys who can pass questions like this, but you also need engineers who can fill in all the rest of the tasks that make up a production worthy system. I have not found that the uber-algorithm guys always pay attention to detail, put in good comments, think of all the edge cases, communicate well with outside teams, etc. Those types of engineers are what you consider a "bad hire", and won't be working at Google.
- unoti 11y ago
  > you also need engineers who can fill in all the rest of the tasks that make up a production worthy system. I have not found that the uber-algorithm guys always pay attention to detail, put in good comments, think of all the edge cases, communicate well with outside teams, etc. What's to keep a good engineer like you describe from buying a couple of books and spending a couple of months working on becoming the type of person that can do well on these questions?
  - laurencerowe 11y ago
    There is a glut of interesting opportunities out there. For me personally, there are just many more interesting ways to spend my time than revising the CS classes I took almost 15 years ago.
  - msandford 11y ago
    > What's to keep a good engineer like you describe from buying a couple of books and spending a couple of months working on becoming the type of person that can do well on these questions? Google doesn't make everyone multi-millionaires anymore. If you want me to jump through a bunch of hoops for you, you'd better have a pretty solid case for what the outcome will be. By actively selecting against people who don't want to answer bullshit questions, they're definitely turning away creative, talented people. They're selecting for different people instead. Perhaps that's what Google needs to feed the machine now that the company is mature, so I won't necessarily fault them for it. But that filtering or selection IS happening, even if they don't think it is.

falcolas 11y ago

If only whiteboarding interview processes actually weeded out false positives. In practice, they select for people with good memorization skills. If you can remember the details on a ten dozen different algorithms and data structures, you can pass one of these without having a single lick of creativity or skill. I say this as an employee who has worked alongside many unskilled drones who made it past the algorithmic interview process.

chrismsnz 11y ago
That's not what the interview process is like at all. They're more interested in how you approach real world issues (the questions I got asked were conceivably real-life issues a company like Google would face with its products). If you can solve that issue by applying efficient and well understood data structures and algorithms, then that indicates you understand the problem space and solutions that may apply. It's not like they get you in a room and ask you to draw a linked list or a binary tree.
- falcolas 11y ago
  > It's not like they get you in a room and ask you to draw a linked list or a binary tree. And yet the OP was turned down for not being able to invert a binary tree (based on limited information, etc etc.)? What do you feel that is, if not a "whiteboard this algorithm" question? As for many of the other companies who try and emulate Google - this is exactly what they do.
  - chrismsnz 11y ago
    > And yet the OP was turned down for not being able to invert a binary tree Your source is a series of salty tweets.
    - falcolas 11y ago
      Well, for what it's worth, it's all we have to go on at the moment. And in either case, my original comment was directed at the idea that "algorithmic interviews provide false negatives, but not false positives".

PhasmaFelis 11y ago

It's true that evaluating candidates is a hard problem, but you'd think that at the very least they would try to evaluate them based on skills that will actually be used in the job. Judging programmers based on their whiteboard skills is like judging a ballet dancer based on how eloquently he can describe his training routine.

RogerL 11y ago

No, it doesn't. Say I pass the invert a tree question. Does that prove I am good? That I can design a product, listen to customer requirements, come up with ideas that elude others, read a research paper and turn it into something commercial, solder a circuit board, make a schedule, mentor junior engineers, write documentation? No. All inverting a tree tells them is I studied trees recently, and/or I'm at least superficially clever. Google lets go of plenty of people. They aren't making perfect hires.

hinkley 11y ago
They have publicly admitted there's no correlation between their hiring process and outcomes. I'm not even clear [why] we're still debating this. Other than to wonder why they still haven't done anything about it. (I interviewed there ~8 years ago, pre Android when they were looking for deep mobile expertise and got flunked out on a similar problem. I am not at all interested in working for them now)
- alexfernandez 11y ago
  To show that there is a correlation you have to quantify both hiring process evaluation and job performance. Since both are elusive quantities that are hard to quantify, showing there is no correlation may just mean that measurements are done poorly. E.g. when job success is measured by boss satisfaction.
reagency 11y ago
None of the things you describe are part of the job for a rank and file bigco engineer. They already have filled all the leadership roles.

chrischen 11y ago

Problem is if your system is actually capable measuring up to the claimed degree of accuracy, or just stabbing in the dark after a certain point. IQ tests supposedly become inaccurate after 150/160, so you wouldn't really want to put too much emphasis on 180 vs 170 at those levels.

fsk 11y ago

This has got to be the biggest hiring fallacy I've ever heard. "It's better to reject a good candidate than hire a bad candidate." That's completely false, and anyone who says that is completely ignorant of Bayesian logic. Here are some simple numbers. Suppose that a "good" candidate is a 1-in-100 find. Suppose that a "bad" candidate has a 1% chance of tricking you into hiring them anyway. Every time you pass on a "good" candidate, that is greater opportunity for a "bad" candidate to trick you into hiring them! Counterintuitively, if you pass on too many of the "good" candidates, in your overzealousness to reject bad candidates, you're actually INCREASING THE ODDS OF A BAD HIRE. This is management porn. "It's better to reject a good candidate than hire a bad candidate." It makes the manager feel good. "Wow! That candidate seemed smart, but I rejected him anyway! I'm such a great leader! I make the tough decisions!" tl;dr summary Because "good" candidates are rare, every time you pass on a good candidate that increases your odds of making a bad hire! This is simple Bayesian reasoning!

ricardobeat 11y ago
This is not about numbers. A bad hire can easily cause more friction than if nobody was hired at all. Net effect for the business is negative. You're also probably severely overestimating the likelihood of a bad hire going through.
- liberatus 11y ago
  Doesn't understand bayesian logic! Doesn't understand risk management!
  - fsk 11y ago
    Idiot: We don't want to make any bad hires! Therefore, we reject a lot of good candidates! FSK: If you pass on many good candidates, and you have a small chance of hiring any given bad candidate, then each good candidate you reject actually INCREASES your odds of making a bad hire. Idiot: We made a bad hire once! Never again! Now we reject lots of good candidates to avoid that repeat disaster! FSK: But, if you want to minimize your bad hire rate, you also have to minimize the number of good candidates you reject. Idiot: NO! NO! NO! The way you make sure you hire no bad candidates is to be so strict that you reject lots of good ones! That's what everyone told me so it must be right! In one ear and out the other. Why do people who don't understand statistics get to be managers? If you don't understand this statistics argument I made, you're unqualified to work in any sort of technical area.

mark-r 11y ago

"Google's perspective" shouldn't be a monolithic thing. Sure if you're sorting through candidates that were dropped on you then this process makes sense. But if you've discovered someone that by reputation and demonstrated code base has already excelled, and you're actively chasing them, then the interview process for that individual has to change. It's more about seeing if they'll thrive in your environment, and convincing them that you're an inevitable part of their career.

smsm42 11y ago

> you need a system to quantify how good they are But do you? How catastrophical would it be if Google accidentally hired not the top 0.001% but instead somebody from top 0.01%? I mean, same people that build everything else around us and the world still didn't collapse yet? I get it, you have to have standards, and you don't want to waste time and money on somebody that can't pass fizzbuzz test. But once you've moved past that - do you need to obsess so hard on ranking and quantifying it? There are probably some positions in Google that require people to invert binary trees (speaking figuratively) day in and day out, but my decades of experience in the industry shows that most positions aren't like that. I've not worked for google but I'm pretty sure it's also this way there. If you hire smart and competent people, it's OK even if they don't reverse a binary tree here or there.

Corrado 11y ago

Well, my suggestion would be to use Page Rank to sift through the resumes. If you are respected by respected computer programmers then your "score" goes up. They could measure that by mentions in a blog, twitter "conversations", Github commits on the same project, etc. "Hmmm... it looks like this guy really gets around. He's on the same project at GoodProgrammer A and GoodProgrammer B. He also tweets about functional programming and was mentioned in GoodProgrammer C's latest blog."

samkone 11y ago

This cracks me up a bit. I don't have anything against whiteboarding, even though I find it meaningless. With this you more often than not have false positives. And if Google had so many resumes they have to filter through, then why do they have all these recruiters literally spamming you to join? they have recruiting issues like many others tech companies, but the whiteboard is really a bad filter. I've seen very much capable,competent and experienced engineers being denied just because they didn't spend too much time learning cute algorithms or coding in the whiteboard, in front of interviewers sometimes more focused in showing off how smart they are.