10 ms·

Technical phone screen superforecasters

78 points by leeny 6y ago

choppaface 6y ago
“Was this statistically significant, or are you conducting pseudoscience? Definitely pseudoscience.” That’s all you need to know. They quantified interviewer quality in terms of offer rate. This article is about self-celebrating a hiring binge without regard to the impact on the team or company.
- nickff 6y ago
  No it's not. The pair of charts in the post are really illustrative of how inconsistent some of the interviewers were; this post contains some valuable insights which merit further study.
  - gleb 6y ago
    Noise. They didn’t have enough data to run the analysis they wanted to run. I would guess they need 50-250 actual hires per interviewer/group to get meaningful results.
    - nickff 6y ago
      It might be noise; I wouldn't try to publish these results in an academic journal, but I'd keep digging. The title uses the word 'superforecasters', which makes me think the author has read some of Philip Tetlock's work, and is trying to apply it to their situation, which I find admirable.
    - fractionalhare 6y ago
      Eh, no. I'd expect a dramatic enough effect size that they could make it rigorous with 10 or so actual hires per interviewer. But I agree what's presented here is noise.
      - pmiller2 6y ago
        I agree. You could probably do it with 3-5 hires per interviewer even, if you drastically increased the number of interviewers.
- jackblemming 6y ago
  Yes I was surprised they didn't include any analysis on how the hired employees actually performed. Nor did they test their system by doing blind interviews where already hired and well performing employees would go through the pipeline. I don't know if they actually wanted to take a good hard look at their own system.
  - pmiller2 6y ago
    That's hard to do, considering any fixed test has a ceiling on how much it can test for, and you don't get access to rejected candidates as a control group. Although, testing current team members with the same test could offer a sort of pseudo-control group, I suppose.
- hyperpape 6y ago
  They quantified phone screen effectiveness by yield of candidates who passed onsites. I don’t know if they said, but I think those were conducted by different engineers. That’s not infallible, but they’re committed to it being an ok metric—-it’s already what determines who they’re hiring. If you think that their decisions in on-sites are sufficiently bad, you probably should give up on the idea of optimizing how phone screens are conducted--it would be wasted effort, and you'd have no idea how to make sure it worked. You can easily imagine that these results are just noise, but it’s more difficult (though hardly impossible) to spin a story in which it’s making their process worse. I think it would require that these interviewers have a knack for selecting weaker candidates who can still pass the on-site.
  - deleted 6y ago
    [deleted]
ipnon 6y ago
Phone screens are tests of sociability. Sociability is an inherently bipolar or multipolar activity, where participants speech, body language and tone are interpreted by other participants, digested, then fed back to the original participant in the form of more speech, body language and tone. We deem those who would score high in sociability to be charming, to have charisma. We deem them so because they have the ability to raise the sociability of those around them. By dent of their mannerisms, they make others feel confident in expressing themselves, and they are willingly to let minor faux pas slide. We would thus expect a candidate being interviewed by a charismatic interviewer to score more highly than they would in the presence of a social dolt. This state of affairs brings into question the utility of phone interviews as measures of job performance.
- fractionalhare 6y ago
  I don't think I've ever taken or given a phone screen where sociability was the metric being tested for. If I was far below a certain baseline of sociability I'm sure I'd fail, but beyond that I don't think phone screens were distinguishing my sociability or charisma whatsoever.
  - xxpor 6y ago
    I don't think OP is implying it's a conscious thing.
    - deleted 6y ago
      [deleted]
  - ThePadawan 6y ago
    Really? I generally found that in a sequence of ~3 interviews for a position, the order will generally be * a personal interview (i.e. the one for sociability - mostly with HR people) * a more technical interview (generally with tech people who don't answer questions about non-technical stuff) * an interview with stakeholders (i.e. a potential future boss, or a department head who can answer questions about long-term career development, perks, etc.. This also tests for sociability)
ericsoderstrom 6y ago
I wonder how much of this comes down to problem selection, rather than effectiveness of the interviewer? Interviewers usually stick to a couple of problems, so their predictive power might depend primarily on the problem set. I've been going through some technical phone screens lately, and am pretty surprised by how bad the questions are that some people choose. Questions that I wouldn't think would provide any signal whatsoever for hiring
- xxpor 6y ago
  Personally speaking, our phone screens are more focused on if an "on-site" (we really need a new name) interview will just be a complete waste of time, especially for the interviewee themselves!
  - deleted 6y ago
    [deleted]
  - pmiller2 6y ago
    You mean like FizzBuzz, but at a little higher level?
    - dannyw 6y ago
      Something like that. If we have a frontend engineering role and a candidate is not able to explain what Promises are, then we know their resume is BS.
unkleaboki 6y ago
Nice one
pydry 6y ago
I wonder how much eliminating hazers contributed. I suspect a lot.
- pmiller2 6y ago
  Depends what you mean by "hazers," I suppose. When I was interviewing candidates at a company that didn't have a structured interviewing process, I used to ask people to write "the world's shittiest JSON encoder." What that is is a very limited JSON encoder that could handle strings, numbers, None, dicts, and arrays, without regard to performance, on their laptop or a loaner, with full access to the internet, and a pinky promise that the candidate wouldn't just look up "how to write JSON encoder," or anything similar. I would give the problem in stages, starting with primitive types and working up to arrays and dicts. If we finished that, we could talk about other stuff, like how to write the world's shittiest JSON encoder, adding additional types, testing, and maybe a little optimization. I thought it was a good problem, and we got a good candidate or 2 out of it through interview loops I participated in. There were lots of twists and turns to the problem, but it wasn't some commonly asked question like LRU cache, and it didn't demand a lot of code. I assume you wouldn't have classified me as a "hazer," right?
  - tcoff91 6y ago
    No that sounds great. Definitely not hazing. Hazing is like where you get asked to solve problems and are expected to produce an algorithm that was originally someone’s PHD thesis and there’s no way you would come up with it unless you memorized it from leetcode and literally did the same problem already. Oh and you gotta also write all the code out by hand on a whiteboard while the person smirks and makes tsk tsk sounds.
    - stephen_cagle 6y ago
      This -> https://leetcode.com/problems/maximal-rectangle/solution/ https://leetcode.com/problems/maximal-rectangle/solution/ is a example of a hazzer interview. Could be mistaken, but I think I found that this is the most common hard Dynamic Programming problem at Google interviews. The idea that anyone should be able to solve a problem like this in 45 minutes as a demonstration of their skill as a software engineer is just ludicrous. If you can solve this problem it means you have studied this or similar problems. All it means is that you have the luxury of study. This has almost nothing to do with being a software engineer.
that_guy_iain 6y ago
Personally, I think these sort of approaches are the problem with the tech job market. We currently have a massive shortage in talent to do the job, yet complains seem to think we need to prove we're worth their time when in realitiy there are more companies looking to hire than developers so the competition is not with developers with other developers for jobs but with companies for talent. So that is the first part I think is fundmentally wrong. I've literally had companies tell me people spend days on their tech test, I instantly told them I wouldn't be doing that tech test and if they wanted to test me I could come in again and they could give me a production issue to fix, code review, etc but any test would need to have a reasonable time limit. They never got back to me and honestly I'm really happy they didn't. The next part is the entire tech screening process. I remember back in the day when we used to just ask people if they could do FizzBuzz now we're looking at if they used an interface to wrap their ORM usage. That's something that can get picked up at code review and taught. Not everyone is going to code by default at the style of a company but they can learn to do the things the company wants. Then there is the obivous, lets test people for things they aren't going to do. Google and co made this the in thing and the fact we now have books upon books just designed so people who are good at tech can pass a tech interview is a sign in itself that there is something rotten. Hell, there was one guy that took almost a year off to study for the Google exam, err I mean interview. In a world where the shortage of tech employee means they're on the eligble for a visa list in pretty much every country in the world. I've literally had someone be failed on a test before they did far more than expected therefore didn't have time to finish the test. No, that's the person you 100% want. They exceeded expecations. I've made pretty much every mistake in interviewing others, I've had these mistakes while I was interviewing. I've come to realise what is important is, can they do the basic level programming required? If so get them in and if they really suck let them go during the trial period. Our worst developer at the company I work for interviewed amazingly, he falls asleep during meetings. A bunch of companies keep trying to improve the hiring in tech, the problem is, they are the problem.
- bsder 6y ago
  > Google and co made this the in thing The problem was "graph algorithms" used to be like "C pointer indirections"--a task that indicated that you were one notch above the average. Of course, once Google made this a metric, it lost all meaning. If you put a condition between someone and $500K, they WILL figure out how to convince you that they passed the condition.
rahimiali 6y ago
Please don’t implement this in your hiring pipeline. What’s missing from this is an analysis of the candidates that didn’t make it past the screens of these superscreeners. What kind of bias are these superscreeners introducing into the candidate pool? Are they good predictors because they inadvertently exploit a bias in the interviewers?
- bleah1000 6y ago
  While the author didn't put that data in the article, the very last bullet point claimed that this method increased diversity. I would be a bit skeptical of that claim without any evidence, but it doesn't look like they ignored that could be a problem.
- amf12 6y ago
  And also they are optimizing for TP while ignoring the FN. Is it possible that some of the candidates they rejected could have been interviewed by other interviewers and made it to offer stage? It probably doesn't matter to them as long as they get their positions filled, but its prolonging their hiring.
deleted 6y ago
[deleted]
dkdk8283 6y ago
Do we really have a talent shortage? I think the market may be flooded - and we have the luxury to match personality with technical ability.
- confidantlake 6y ago
  We do not. I agree with you.
username90 6y ago
So they selected phone screeners with the same bias as the on-site screeners? Shouldn't the aim be to reduce correlation in order to increase the signal? Doing the same interview 5 times would make hiring even cheaper, but that isn't really the most important metric you should optimize for.

78 points by leeny 6y ago

“Was this statistically significant, or are you conducting pseudoscience? Definitely pseudoscience.” That’s all you need to know. They quantified interviewer quality in terms of offer rate. This article is about self-celebrating a hiring binge without regard to the impact on the team or company.

nickff 6y ago
No it's not. The pair of charts in the post are really illustrative of how inconsistent some of the interviewers were; this post contains some valuable insights which merit further study.
- gleb 6y ago
  Noise. They didn’t have enough data to run the analysis they wanted to run. I would guess they need 50-250 actual hires per interviewer/group to get meaningful results.
  - nickff 6y ago
    It might be noise; I wouldn't try to publish these results in an academic journal, but I'd keep digging. The title uses the word 'superforecasters', which makes me think the author has read some of Philip Tetlock's work, and is trying to apply it to their situation, which I find admirable.
  - fractionalhare 6y ago
    Eh, no. I'd expect a dramatic enough effect size that they could make it rigorous with 10 or so actual hires per interviewer. But I agree what's presented here is noise.
    - pmiller2 6y ago
      I agree. You could probably do it with 3-5 hires per interviewer even, if you drastically increased the number of interviewers.
jackblemming 6y ago
Yes I was surprised they didn't include any analysis on how the hired employees actually performed. Nor did they test their system by doing blind interviews where already hired and well performing employees would go through the pipeline. I don't know if they actually wanted to take a good hard look at their own system.
- pmiller2 6y ago
  That's hard to do, considering any fixed test has a ceiling on how much it can test for, and you don't get access to rejected candidates as a control group. Although, testing current team members with the same test could offer a sort of pseudo-control group, I suppose.
hyperpape 6y ago
They quantified phone screen effectiveness by yield of candidates who passed onsites. I don’t know if they said, but I think those were conducted by different engineers. That’s not infallible, but they’re committed to it being an ok metric—-it’s already what determines who they’re hiring. If you think that their decisions in on-sites are sufficiently bad, you probably should give up on the idea of optimizing how phone screens are conducted--it would be wasted effort, and you'd have no idea how to make sure it worked. You can easily imagine that these results are just noise, but it’s more difficult (though hardly impossible) to spin a story in which it’s making their process worse. I think it would require that these interviewers have a knack for selecting weaker candidates who can still pass the on-site.
- deleted 6y ago
  [deleted]

ipnon 6y ago

Phone screens are tests of sociability. Sociability is an inherently bipolar or multipolar activity, where participants speech, body language and tone are interpreted by other participants, digested, then fed back to the original participant in the form of more speech, body language and tone. We deem those who would score high in sociability to be charming, to have charisma. We deem them so because they have the ability to raise the sociability of those around them. By dent of their mannerisms, they make others feel confident in expressing themselves, and they are willingly to let minor faux pas slide. We would thus expect a candidate being interviewed by a charismatic interviewer to score more highly than they would in the presence of a social dolt. This state of affairs brings into question the utility of phone interviews as measures of job performance.

fractionalhare 6y ago
I don't think I've ever taken or given a phone screen where sociability was the metric being tested for. If I was far below a certain baseline of sociability I'm sure I'd fail, but beyond that I don't think phone screens were distinguishing my sociability or charisma whatsoever.
- xxpor 6y ago
  I don't think OP is implying it's a conscious thing.
  - deleted 6y ago
    [deleted]
- ThePadawan 6y ago
  Really? I generally found that in a sequence of ~3 interviews for a position, the order will generally be * a personal interview (i.e. the one for sociability - mostly with HR people) * a more technical interview (generally with tech people who don't answer questions about non-technical stuff) * an interview with stakeholders (i.e. a potential future boss, or a department head who can answer questions about long-term career development, perks, etc.. This also tests for sociability)

ericsoderstrom 6y ago

I wonder how much of this comes down to problem selection, rather than effectiveness of the interviewer? Interviewers usually stick to a couple of problems, so their predictive power might depend primarily on the problem set. I've been going through some technical phone screens lately, and am pretty surprised by how bad the questions are that some people choose. Questions that I wouldn't think would provide any signal whatsoever for hiring

xxpor 6y ago
Personally speaking, our phone screens are more focused on if an "on-site" (we really need a new name) interview will just be a complete waste of time, especially for the interviewee themselves!
- deleted 6y ago
  [deleted]
- pmiller2 6y ago
  You mean like FizzBuzz, but at a little higher level?
  - dannyw 6y ago
    Something like that. If we have a frontend engineering role and a candidate is not able to explain what Promises are, then we know their resume is BS.

unkleaboki 6y ago

Nice one

pydry 6y ago

I wonder how much eliminating hazers contributed. I suspect a lot.

pmiller2 6y ago
Depends what you mean by "hazers," I suppose. When I was interviewing candidates at a company that didn't have a structured interviewing process, I used to ask people to write "the world's shittiest JSON encoder." What that is is a very limited JSON encoder that could handle strings, numbers, None, dicts, and arrays, without regard to performance, on their laptop or a loaner, with full access to the internet, and a pinky promise that the candidate wouldn't just look up "how to write JSON encoder," or anything similar. I would give the problem in stages, starting with primitive types and working up to arrays and dicts. If we finished that, we could talk about other stuff, like how to write the world's shittiest JSON encoder, adding additional types, testing, and maybe a little optimization. I thought it was a good problem, and we got a good candidate or 2 out of it through interview loops I participated in. There were lots of twists and turns to the problem, but it wasn't some commonly asked question like LRU cache, and it didn't demand a lot of code. I assume you wouldn't have classified me as a "hazer," right?
- tcoff91 6y ago
  No that sounds great. Definitely not hazing. Hazing is like where you get asked to solve problems and are expected to produce an algorithm that was originally someone’s PHD thesis and there’s no way you would come up with it unless you memorized it from leetcode and literally did the same problem already. Oh and you gotta also write all the code out by hand on a whiteboard while the person smirks and makes tsk tsk sounds.
  - stephen_cagle 6y ago
    This -> https://leetcode.com/problems/maximal-rectangle/solution/ https://leetcode.com/problems/maximal-rectangle/solution/ is a example of a hazzer interview. Could be mistaken, but I think I found that this is the most common hard Dynamic Programming problem at Google interviews. The idea that anyone should be able to solve a problem like this in 45 minutes as a demonstration of their skill as a software engineer is just ludicrous. If you can solve this problem it means you have studied this or similar problems. All it means is that you have the luxury of study. This has almost nothing to do with being a software engineer.

that_guy_iain 6y ago

Personally, I think these sort of approaches are the problem with the tech job market. We currently have a massive shortage in talent to do the job, yet complains seem to think we need to prove we're worth their time when in realitiy there are more companies looking to hire than developers so the competition is not with developers with other developers for jobs but with companies for talent. So that is the first part I think is fundmentally wrong. I've literally had companies tell me people spend days on their tech test, I instantly told them I wouldn't be doing that tech test and if they wanted to test me I could come in again and they could give me a production issue to fix, code review, etc but any test would need to have a reasonable time limit. They never got back to me and honestly I'm really happy they didn't. The next part is the entire tech screening process. I remember back in the day when we used to just ask people if they could do FizzBuzz now we're looking at if they used an interface to wrap their ORM usage. That's something that can get picked up at code review and taught. Not everyone is going to code by default at the style of a company but they can learn to do the things the company wants. Then there is the obivous, lets test people for things they aren't going to do. Google and co made this the in thing and the fact we now have books upon books just designed so people who are good at tech can pass a tech interview is a sign in itself that there is something rotten. Hell, there was one guy that took almost a year off to study for the Google exam, err I mean interview. In a world where the shortage of tech employee means they're on the eligble for a visa list in pretty much every country in the world. I've literally had someone be failed on a test before they did far more than expected therefore didn't have time to finish the test. No, that's the person you 100% want. They exceeded expecations. I've made pretty much every mistake in interviewing others, I've had these mistakes while I was interviewing. I've come to realise what is important is, can they do the basic level programming required? If so get them in and if they really suck let them go during the trial period. Our worst developer at the company I work for interviewed amazingly, he falls asleep during meetings. A bunch of companies keep trying to improve the hiring in tech, the problem is, they are the problem.

bsder 6y ago
> Google and co made this the in thing The problem was "graph algorithms" used to be like "C pointer indirections"--a task that indicated that you were one notch above the average. Of course, once Google made this a metric, it lost all meaning. If you put a condition between someone and $500K, they WILL figure out how to convince you that they passed the condition.

rahimiali 6y ago

Please don’t implement this in your hiring pipeline. What’s missing from this is an analysis of the candidates that didn’t make it past the screens of these superscreeners. What kind of bias are these superscreeners introducing into the candidate pool? Are they good predictors because they inadvertently exploit a bias in the interviewers?

bleah1000 6y ago
While the author didn't put that data in the article, the very last bullet point claimed that this method increased diversity. I would be a bit skeptical of that claim without any evidence, but it doesn't look like they ignored that could be a problem.
amf12 6y ago
And also they are optimizing for TP while ignoring the FN. Is it possible that some of the candidates they rejected could have been interviewed by other interviewers and made it to offer stage? It probably doesn't matter to them as long as they get their positions filled, but its prolonging their hiring.

deleted 6y ago

[deleted]

dkdk8283 6y ago

Do we really have a talent shortage? I think the market may be flooded - and we have the luxury to match personality with technical ability.

confidantlake 6y ago
We do not. I agree with you.

username90 6y ago

So they selected phone screeners with the same bias as the on-site screeners? Shouldn't the aim be to reduce correlation in order to increase the signal? Doing the same interview 5 times would make hiring even cheaper, but that isn't really the most important metric you should optimize for.