3 ms·

I mostly focus on consistency in this post (how to make a group of interviewers more consistent). Of course, what actually matters is accuracy (predictive utili

by ammon 8y ago

I mostly focus on consistency in this post (how to make a group of interviewers more consistent). Of course, what actually matters is accuracy (predictive utility of the interview). Obviously those are not the same thing (you could run 100% consistent interviews by just having everyone always grade “strong no”). However, I think (based on running the interview team at Triplebyte) that inconsistency is the primary obstacle to accuracy in practice. So I end up spending the majority of my time focusing on how to make interviewers more consistent.

sirsar 8y ago
Were you able to measure the outcome of this process? I.e., was there a significant reduction in inconsistency, inaccuracy, or similarity bias after the exercise?
joe_the_user 8y ago
Well, consistency and fairness might have the added benefit of goodwill towards the company from those folks don't necessarily get hired.
thaumasiotes 8y ago
In psychometrics, the field that studies the same thing you're trying to do, the concepts you're referring to as "consistency" and "accuracy" are known as "reliability" and "validity". It's somewhat striking to me that you seem so worried about these concepts but you don't seem to be aware of the normal terms for them. How much does TripleByte try to inform itself of the existing research in this field? To what extent does TripleByte seek to incorporate psychometric results about what kinds of tests are likely to have high reliability and construct validity? And one more more specific question: > what actually matters is accuracy (predictive utility of the interview) What is it that you're trying to predict? You could be trying to find employees who will be good employees, which would put TripleByte in the business of credentialing, or to find employees who will pass interviews at other companies, which would make TripleByte a recruiting agency. In the past, Harj has been explicit that what TripleByte wants to predict is whether a candidate will successfully pass the hiring process at another company, regardless of how well that hiring process performs. Is this still true?
- trishume 8y ago
  I’m guessing they know the terms but ‘consistency’ and ‘accuracy’ are just more common easier to understand terms without sacrificing meaning. I’ve talked to some TripleByte engineers and been very impressed by the sophistication of the statistics and experimental methodology they use.
  - thaumasiotes 8y ago
    Using an experimental methodology isn't something you should be impressed by in itself. Experimenting means you don't know what you should be doing. That's great if no one knows what you should be doing and you're trying to figure it out; it's less great if everyone else knows what you should be doing, but you don't. The psychometric literature is pretty robust.
    - trishume 8y ago
      One specific conversation I had was how they read the literature on the best adaptive testing systems and then developed improvements tailored to their specific data and the advantages they had as a real time online test. Another was on specifically the psychometric literature, and the big meta-analyses of the predictiveness of different testing factors on job performance, and how that influenced the experiments they did early on and how they honed in on what they do today. As well as downsides they discovered of various methods people commonly suggest they're ignoring. I came away from those conversations extremely impressed with TripleByte's employees and competence as an organization. They definitely think about this stuff.
taneq 8y ago
Ses ms reasonable, after all, precision isn't the same thing as accuracy but you can't get accuracy without precision.

by ammon 8y ago

sirsar 8y ago

Were you able to measure the outcome of this process? I.e., was there a significant reduction in inconsistency, inaccuracy, or similarity bias after the exercise?

joe_the_user 8y ago

Well, consistency and fairness might have the added benefit of goodwill towards the company from those folks don't necessarily get hired.

thaumasiotes 8y ago

In psychometrics, the field that studies the same thing you're trying to do, the concepts you're referring to as "consistency" and "accuracy" are known as "reliability" and "validity". It's somewhat striking to me that you seem so worried about these concepts but you don't seem to be aware of the normal terms for them. How much does TripleByte try to inform itself of the existing research in this field? To what extent does TripleByte seek to incorporate psychometric results about what kinds of tests are likely to have high reliability and construct validity? And one more more specific question: > what actually matters is accuracy (predictive utility of the interview) What is it that you're trying to predict? You could be trying to find employees who will be good employees, which would put TripleByte in the business of credentialing, or to find employees who will pass interviews at other companies, which would make TripleByte a recruiting agency. In the past, Harj has been explicit that what TripleByte wants to predict is whether a candidate will successfully pass the hiring process at another company, regardless of how well that hiring process performs. Is this still true?

trishume 8y ago
I’m guessing they know the terms but ‘consistency’ and ‘accuracy’ are just more common easier to understand terms without sacrificing meaning. I’ve talked to some TripleByte engineers and been very impressed by the sophistication of the statistics and experimental methodology they use.
- thaumasiotes 8y ago
  Using an experimental methodology isn't something you should be impressed by in itself. Experimenting means you don't know what you should be doing. That's great if no one knows what you should be doing and you're trying to figure it out; it's less great if everyone else knows what you should be doing, but you don't. The psychometric literature is pretty robust.
  - trishume 8y ago
    One specific conversation I had was how they read the literature on the best adaptive testing systems and then developed improvements tailored to their specific data and the advantages they had as a real time online test. Another was on specifically the psychometric literature, and the big meta-analyses of the predictiveness of different testing factors on job performance, and how that influenced the experiments they did early on and how they honed in on what they do today. As well as downsides they discovered of various methods people commonly suggest they're ignoring. I came away from those conversations extremely impressed with TripleByte's employees and competence as an organization. They definitely think about this stuff.

taneq 8y ago

Ses ms reasonable, after all, precision isn't the same thing as accuracy but you can't get accuracy without precision.