5 ms·

“Was this statistically significant, or are you conducting pseudoscience? Definitely pseudoscience.” That’s all you need to know. They quantified interviewer

“Was this statistically significant, or are you conducting pseudoscience? Definitely pseudoscience.” That’s all you need to know. They quantified interviewer quality in terms of offer rate. This article is about self-celebrating a hiring binge without regard to the impact on the team or company.

nickff 6y ago
No it's not. The pair of charts in the post are really illustrative of how inconsistent some of the interviewers were; this post contains some valuable insights which merit further study.
- gleb 6y ago
  Noise. They didn’t have enough data to run the analysis they wanted to run. I would guess they need 50-250 actual hires per interviewer/group to get meaningful results.
  - nickff 6y ago
    It might be noise; I wouldn't try to publish these results in an academic journal, but I'd keep digging. The title uses the word 'superforecasters', which makes me think the author has read some of Philip Tetlock's work, and is trying to apply it to their situation, which I find admirable.
  - fractionalhare 6y ago
    Eh, no. I'd expect a dramatic enough effect size that they could make it rigorous with 10 or so actual hires per interviewer. But I agree what's presented here is noise.
    - pmiller2 6y ago
      I agree. You could probably do it with 3-5 hires per interviewer even, if you drastically increased the number of interviewers.
jackblemming 6y ago
Yes I was surprised they didn't include any analysis on how the hired employees actually performed. Nor did they test their system by doing blind interviews where already hired and well performing employees would go through the pipeline. I don't know if they actually wanted to take a good hard look at their own system.
- pmiller2 6y ago
  That's hard to do, considering any fixed test has a ceiling on how much it can test for, and you don't get access to rejected candidates as a control group. Although, testing current team members with the same test could offer a sort of pseudo-control group, I suppose.
hyperpape 6y ago
They quantified phone screen effectiveness by yield of candidates who passed onsites. I don’t know if they said, but I think those were conducted by different engineers. That’s not infallible, but they’re committed to it being an ok metric—-it’s already what determines who they’re hiring. If you think that their decisions in on-sites are sufficiently bad, you probably should give up on the idea of optimizing how phone screens are conducted--it would be wasted effort, and you'd have no idea how to make sure it worked. You can easily imagine that these results are just noise, but it’s more difficult (though hardly impossible) to spin a story in which it’s making their process worse. I think it would require that these interviewers have a knack for selecting weaker candidates who can still pass the on-site.
- deleted 6y ago
  [deleted]

nickff 6y ago

No it's not. The pair of charts in the post are really illustrative of how inconsistent some of the interviewers were; this post contains some valuable insights which merit further study.

gleb 6y ago
Noise. They didn’t have enough data to run the analysis they wanted to run. I would guess they need 50-250 actual hires per interviewer/group to get meaningful results.
- nickff 6y ago
  It might be noise; I wouldn't try to publish these results in an academic journal, but I'd keep digging. The title uses the word 'superforecasters', which makes me think the author has read some of Philip Tetlock's work, and is trying to apply it to their situation, which I find admirable.
- fractionalhare 6y ago
  Eh, no. I'd expect a dramatic enough effect size that they could make it rigorous with 10 or so actual hires per interviewer. But I agree what's presented here is noise.
  - pmiller2 6y ago
    I agree. You could probably do it with 3-5 hires per interviewer even, if you drastically increased the number of interviewers.

jackblemming 6y ago

Yes I was surprised they didn't include any analysis on how the hired employees actually performed. Nor did they test their system by doing blind interviews where already hired and well performing employees would go through the pipeline. I don't know if they actually wanted to take a good hard look at their own system.

pmiller2 6y ago
That's hard to do, considering any fixed test has a ceiling on how much it can test for, and you don't get access to rejected candidates as a control group. Although, testing current team members with the same test could offer a sort of pseudo-control group, I suppose.

hyperpape 6y ago

They quantified phone screen effectiveness by yield of candidates who passed onsites. I don’t know if they said, but I think those were conducted by different engineers. That’s not infallible, but they’re committed to it being an ok metric—-it’s already what determines who they’re hiring. If you think that their decisions in on-sites are sufficiently bad, you probably should give up on the idea of optimizing how phone screens are conducted--it would be wasted effort, and you'd have no idea how to make sure it worked. You can easily imagine that these results are just noise, but it’s more difficult (though hardly impossible) to spin a story in which it’s making their process worse. I think it would require that these interviewers have a knack for selecting weaker candidates who can still pass the on-site.

deleted 6y ago
[deleted]