5 ms·

Laszlo Bock studied the predictive value of technical interviews when he was in charge of people operations at Google. He reported finding that they had zero pr

Laszlo Bock studied the predictive value of technical interviews when he was in charge of people operations at Google. He reported finding that they had zero predictive power. To put it another way, we should expect that picking names out of a hat would do just as well as Google's infamously rigorous interview process. If I recall correctly, this result is generally consistent with other attempts to study hiring with technical requirements. So if technical interviews have no predictive power, is there anything else that does? Yes: standardized testing. So why doesn't everyone use standardized testing? Well, it's a lot of work. You have to accurately identify personal attributes that are relevant to success in your business. They have to be things you can objectively measure. You have to encode them in a standardized test, or, failing that, find an existing test that measures something sufficiently similar to be relevant. You have to administer it in a way that preserves a reliable signal. You probably need to engage professional specialists if you really want it to work. At the very least, you need to get someone in who can ask the right questions about your business and your goals to identify some existing standardized test that will match your needs reasonably well. Maybe you don't think you can afford to do that. I mean, that basically means you can't afford to do better than random chance in hiring, but if you can't, you can't. So what do you do instead? Well, hiring is risky. A bad hire can cost your business a lot. Uncontrolled risk makes people anxious. Anxiety is bad for productivity. Anxious people want to increase their control--or, failing that, their feeling of control. So how can you feel more in control than by pulling names out of a hat? How about technical interviews? Maybe they don't have any predictive power, but they give you a very strong feeling of control. You control the candidate-selection process. You control the interview format and procedures. You decide the criteria, and who gets an offer and who doesn't. Sure, the end result may be the same as pulling names out of a hat, but the process is much more soothing. So technical interviews it is.

handrous 5y ago
> To put it another way, we should expect that picking names out of a hat would do just as well as Google's infamously rigorous interview process. If I recall correctly, this result is generally consistent with other attempts to study hiring with technical requirements. The notoriously-drawn-out-and-draining interviews that practically require at least some study in advance, and probably quite a bit, strongly affect which names are in the hat though, no? I'd assumed that was about half the reason they did them in the first place (the other half is that hazing actually is pretty good at creating "team spirit" out of nothing)
- mikelevins 5y ago
  Sure there's probably some filtering effect. I wonder what it actually filters for? If Bock's results are to believed, it doesn't filter for suitability to the job. Maybe it's just a way to somewhat reduce the size of the candidate pool? And maybe your observation about hazing is on the money, too.
  - handrous 5y ago
    A huge percentage of developers never bother applying to any FAANG (or similar) because they either don't believe they could pass so it'd be pointless, or they don't want to put hundreds of hours of study and go through several very painful days of soul-sucking interviews & travel for something that's not a sure thing, when they're already fairly comfortable as far as income goes (but probably not FAANG-comfortable). That's a pretty significant filter.
    - mikelevins 5y ago
      Good point. So, then, it's filtering for what? Starry-eyed optimism? That might actually be useful to the prospective employer.
      - handrous 5y ago
        It filters for several things, including a very high level of confidence in technical abilities, and a strong enough desire to work in a FAANG(-alike) to put up with a whole lot of dull/painful unpaid bullshit, but the point is that the "hat" doesn't contain a random sample of developers. I think the notoriety of the interviews probably does do a good job of ensuring that the pool almost entirely contains people that, so far as tech chops go, are good enough to do the job—it excludes a bunch more of those from bothering to apply, of course, but the point is that I don't find it surprising that choosing randomly from their already selection-biased "hat" yields results of similar quality to actually conducting the interviews. The problem is that, if they loosen up the interviews, that will (I expect) stop being true. Look at it like this: if you have a club that is notorious for only letting very pretty people in, I bet a random sampling of the line outside skews incredibly good-looking, but they can't just start randomly choosing people from the line without regard for their appearance, or that will change in a hurry. Meanwhile, it's also true that some very pretty club-goers don't get in that line in the first place, for a bunch of reasons possibly including that they don't want to be part of a scene like that, or a lack of confidence or anxiety over being harshly judged, or not thinking it's worth standing in line for an hour when they could go down the block to another club with lower standards and no line, and nearly all people who might be able to get in on an exceptionally weak night don't even think of trying, since more often than not it's pointless—but despite the loss of those folks, the line is still 95+% very pretty people, so it looks like they could do away with the high standards and harsh judgement and just let in a random sample by lottery, if you consider only that in isolation and not the effect the process itself had on the composition of the line. (FWIW, this is a non-FAANG perspective—I'm 100% certain I'd struggle to even get past the phone screen without prep; about 95% sure I could do the actual work to an acceptable level; but judge it only about 50% likely I could eventually pass an interview with one or another after a great deal of practice and many, many tries, given probably 18-24 months of on-and-off effort and tens of extremely unpleasant days; so, I'm in the "unwilling to spend weeks of my life doing nothing but unwrapping Wonka Bars, when I'm not very sure there's a Golden Ticket in one of them" crowd)
thedufer 5y ago
> Laszlo Bock studied the predictive value of technical interviews when he was in charge of people operations at Google. He reported finding that they had zero predictive power. Mind citing a source here? That would be a pretty big deal, both because it flies in the face of the entire industry, and because of the implications about Google if it determined the interviews were useless and then continued spending an incredible amount of money conducting them.
- mikelevins 5y ago
  I was relying on my own memory, so feel free to disregard my remarks. If you want to know what Bock has said about the subject, Google for Laszlo Bock along with some terms like "hiring" and "predictive". His book on the subject is here: https://www.goodreads.com/book/show/22875447-work-rules https://www.goodreads.com/book/show/22875447-work-rules It says that brain teasers and unstructured interviews don't work. Structured interviews do. So what's a structured interview? It's a standardized set of predefined factual questions determined by relevant domain experts to be directly relevant to the job. In effect, a custom standardized test. He says the problem with them is that interviewers don't like using them. Oh, and he says the problem with unstructured interviews--which interviewers prefer--is that the best predictor of the interview's result is an impression that the interviewer forms in the first ten seconds, and that impression has zero predictive power about the candidate's suitability for the job. Google a little more widely to find more general research in this area. For example: https://www.researchgate.net/publication/240237107_The_Role_of_General_Cognitive_Ability_and_Job_Performance_Why_There_Cannot_Be_a_Debate https://www.researchgate.net/publication/240237107_The_Role_... ...which says the best predictor overall is general cognitive ability (what we usually call IQ) and https://www.academia.edu/32446971/Big_Five_Personality_Dimensions_and_Job_Performance_in_Army_and_Civil_Occupations_A_European_Perspective https://www.academia.edu/32446971/Big_Five_Personality_Dimen... ...which says that two of the Big Five personality traits (namely, Conscientiousness and Emotional Stability) are good predictors of good job performance. IQ, by the way, is strongly correlated with another of the Big Five, generally called Openness. So if we believe all of this research, then what you sould be looking for is objectively-verifiable skills directly related to what you expect the job to actually entail, combined with general intelligence, conscientiousness, and emotional stability. By the way, I score low in emotional stability, so I have to rely on the other traits. If you find that my characterizations are grossly wrong, feel free to trash me over them and I will thank you for improving my understanding and humbly apologize for wasting your time.
edmundsauto 5y ago
Did his work look only at people who passed the Google process? I would guess so, since he probably didn't have the resources to track down the careers of people outside of Google. The statement "Amongst people who passed the Google interview, the technical screen had no connection with Google career success" is a very different statement than "no connection between technical interview performance and career performance" due to the selection bias.
- mikelevins 5y ago
  You're right on both counts. Good point.
ergocoder 5y ago
Zero prediction power is an extremely exaggeration. If this was true, Google could have just hired everyone who failed and rejected everyone who passed the interview? If Google started doing that, the company would collapse pretty quickly.
- skinkestek 5y ago
  Agree that it is an extreme exaggeration. Some people just can't code and there is a baseline. > If Google started doing that, the company would collapse pretty quickly. I'd mention that they are already crumbling: 1.) Search has been deteriorating since 2007 and is now at the level of the engines they replaced (yep, Yahoo, Alta Vista etc actually worked, it was just that Google was extremely much better back then). 2.) They cannot maintain anything except the core services. 3.) Previously everyone including me wanted to work there because it was cool, now I at least don't think of it as cool anymore so I guess they have a harder time recruiting the best and brightest.

handrous 5y ago

> To put it another way, we should expect that picking names out of a hat would do just as well as Google's infamously rigorous interview process. If I recall correctly, this result is generally consistent with other attempts to study hiring with technical requirements. The notoriously-drawn-out-and-draining interviews that practically require at least some study in advance, and probably quite a bit, strongly affect which names are in the hat though, no? I'd assumed that was about half the reason they did them in the first place (the other half is that hazing actually is pretty good at creating "team spirit" out of nothing)

mikelevins 5y ago
Sure there's probably some filtering effect. I wonder what it actually filters for? If Bock's results are to believed, it doesn't filter for suitability to the job. Maybe it's just a way to somewhat reduce the size of the candidate pool? And maybe your observation about hazing is on the money, too.
- handrous 5y ago
  A huge percentage of developers never bother applying to any FAANG (or similar) because they either don't believe they could pass so it'd be pointless, or they don't want to put hundreds of hours of study and go through several very painful days of soul-sucking interviews & travel for something that's not a sure thing, when they're already fairly comfortable as far as income goes (but probably not FAANG-comfortable). That's a pretty significant filter.
  - mikelevins 5y ago
    Good point. So, then, it's filtering for what? Starry-eyed optimism? That might actually be useful to the prospective employer.
    - handrous 5y ago
      It filters for several things, including a very high level of confidence in technical abilities, and a strong enough desire to work in a FAANG(-alike) to put up with a whole lot of dull/painful unpaid bullshit, but the point is that the "hat" doesn't contain a random sample of developers. I think the notoriety of the interviews probably does do a good job of ensuring that the pool almost entirely contains people that, so far as tech chops go, are good enough to do the job—it excludes a bunch more of those from bothering to apply, of course, but the point is that I don't find it surprising that choosing randomly from their already selection-biased "hat" yields results of similar quality to actually conducting the interviews. The problem is that, if they loosen up the interviews, that will (I expect) stop being true. Look at it like this: if you have a club that is notorious for only letting very pretty people in, I bet a random sampling of the line outside skews incredibly good-looking, but they can't just start randomly choosing people from the line without regard for their appearance, or that will change in a hurry. Meanwhile, it's also true that some very pretty club-goers don't get in that line in the first place, for a bunch of reasons possibly including that they don't want to be part of a scene like that, or a lack of confidence or anxiety over being harshly judged, or not thinking it's worth standing in line for an hour when they could go down the block to another club with lower standards and no line, and nearly all people who might be able to get in on an exceptionally weak night don't even think of trying, since more often than not it's pointless—but despite the loss of those folks, the line is still 95+% very pretty people, so it looks like they could do away with the high standards and harsh judgement and just let in a random sample by lottery, if you consider only that in isolation and not the effect the process itself had on the composition of the line. (FWIW, this is a non-FAANG perspective—I'm 100% certain I'd struggle to even get past the phone screen without prep; about 95% sure I could do the actual work to an acceptable level; but judge it only about 50% likely I could eventually pass an interview with one or another after a great deal of practice and many, many tries, given probably 18-24 months of on-and-off effort and tens of extremely unpleasant days; so, I'm in the "unwilling to spend weeks of my life doing nothing but unwrapping Wonka Bars, when I'm not very sure there's a Golden Ticket in one of them" crowd)

thedufer 5y ago

> Laszlo Bock studied the predictive value of technical interviews when he was in charge of people operations at Google. He reported finding that they had zero predictive power. Mind citing a source here? That would be a pretty big deal, both because it flies in the face of the entire industry, and because of the implications about Google if it determined the interviews were useless and then continued spending an incredible amount of money conducting them.

mikelevins 5y ago
I was relying on my own memory, so feel free to disregard my remarks. If you want to know what Bock has said about the subject, Google for Laszlo Bock along with some terms like "hiring" and "predictive". His book on the subject is here: https://www.goodreads.com/book/show/22875447-work-rules https://www.goodreads.com/book/show/22875447-work-rules It says that brain teasers and unstructured interviews don't work. Structured interviews do. So what's a structured interview? It's a standardized set of predefined factual questions determined by relevant domain experts to be directly relevant to the job. In effect, a custom standardized test. He says the problem with them is that interviewers don't like using them. Oh, and he says the problem with unstructured interviews--which interviewers prefer--is that the best predictor of the interview's result is an impression that the interviewer forms in the first ten seconds, and that impression has zero predictive power about the candidate's suitability for the job. Google a little more widely to find more general research in this area. For example: https://www.researchgate.net/publication/240237107_The_Role_of_General_Cognitive_Ability_and_Job_Performance_Why_There_Cannot_Be_a_Debate https://www.researchgate.net/publication/240237107_The_Role_... ...which says the best predictor overall is general cognitive ability (what we usually call IQ) and https://www.academia.edu/32446971/Big_Five_Personality_Dimensions_and_Job_Performance_in_Army_and_Civil_Occupations_A_European_Perspective https://www.academia.edu/32446971/Big_Five_Personality_Dimen... ...which says that two of the Big Five personality traits (namely, Conscientiousness and Emotional Stability) are good predictors of good job performance. IQ, by the way, is strongly correlated with another of the Big Five, generally called Openness. So if we believe all of this research, then what you sould be looking for is objectively-verifiable skills directly related to what you expect the job to actually entail, combined with general intelligence, conscientiousness, and emotional stability. By the way, I score low in emotional stability, so I have to rely on the other traits. If you find that my characterizations are grossly wrong, feel free to trash me over them and I will thank you for improving my understanding and humbly apologize for wasting your time.

edmundsauto 5y ago

Did his work look only at people who passed the Google process? I would guess so, since he probably didn't have the resources to track down the careers of people outside of Google. The statement "Amongst people who passed the Google interview, the technical screen had no connection with Google career success" is a very different statement than "no connection between technical interview performance and career performance" due to the selection bias.

mikelevins 5y ago
You're right on both counts. Good point.

ergocoder 5y ago

Zero prediction power is an extremely exaggeration. If this was true, Google could have just hired everyone who failed and rejected everyone who passed the interview? If Google started doing that, the company would collapse pretty quickly.

skinkestek 5y ago
Agree that it is an extreme exaggeration. Some people just can't code and there is a baseline. > If Google started doing that, the company would collapse pretty quickly. I'd mention that they are already crumbling: 1.) Search has been deteriorating since 2007 and is now at the level of the engines they replaced (yep, Yahoo, Alta Vista etc actually worked, it was just that Google was extremely much better back then). 2.) They cannot maintain anything except the core services. 3.) Previously everyone including me wanted to work there because it was cool, now I at least don't think of it as cool anymore so I guess they have a harder time recruiting the best and brightest.