5 ms·

At Kellogg they used to say it's best to either be the first to be interviewed or the last—assuming you're a decent candidate and well prepared to begin with.

At Kellogg they used to say it's best to either be the first to be interviewed or the last—assuming you're a decent candidate and well prepared to begin with. First because you set the benchmark, and your outstanding qualities become "requirements" for the candidates that follow to meet that benchmark. Last because of recency bias, so whatever qualities you have are better recalled by interviewers. Everyone else becomes somewhat forgettable. (I would guess they mentioned a study about it, but it's been 10 years and I don't have the reference handy.) In my experience as an interviewer, everyone in the middle does get somewhat mixed up together, especially when I had less than 5 minutes to scribble notes and reset between candidates. But I would modify "first" and "last" to "towards the start" and "towards the end" e.g., I will (subconsciously) more easily benchmark candidates against a very strong "second interview of the day" than against a lackluster "very first candidate", if that makes sense. Said differently, whoever is the first candidate that hits it out of the park becomes the benchmark. And whoever happens to be the best relatively strong candidate is more easily recalled than other relatively strong candidates. Thank you for coming to my TED talk

6510 3y ago
[flagged]
cyanydeez 3y ago
now to make it interesting, place the middle candidates right after lunch to see if having lunch improves the middle stretch
bschne 3y ago
(how) do you (try to) correct for this with interviews?
- OJFord 3y ago
  Someone's bound to mention it in this thread (or I hope they do, because I thought it was great but can't remember the name or the details) - but there's some formula that's roughly like: N = total applicants 1. Interview N/10, decline no matter what 2. Hire the next person who's better than everyone seen so far As I recall it was slightly more complex than that, perhaps only to give it an impression of being rigorous theory, but that was the gist of it.
  - navane 3y ago
    I think you're looking for the Secretary problem. And the solution is almost what you said, but instead of N/10 it's a bit over N/3.
  - wiml 3y ago
    This only applies if you have to make a hire/skip decision immediately after each interview and can't go back later. (The formulation I usually see is a princess choosing a suitor — for pride/face reasons, she can't pass on a suitor and then go back if it turns out he was the best of the bunch.) That's not typically how job interviews work, though. I think the magic number is 1/e of the applicant pool (e = Napier's constant, 2.7...)
  - vikingerik 3y ago
    That mathematical problem (the secretary problem) is often cited, but it's not how real hiring works. The problem defines the only success criterion as hiring the single top candidate, and everything else fails. That's not at all what you're looking for in the real world - there are a spectrum of results, where any one in the top decile will be great and the next decile only slightly lesser and so on.
    - bschne 3y ago
      One-dimensional ranking is one aspect of how it breaks in real-world hiring, but I think almost the bigger might be that the secretary problem assumes you need to decide immediately and can't just wait around until you've seen the whole batch of applicants.
      - Dylan16807 3y ago
        The main difference is that you're trying to solve the problem of turning an interview into a way to choose between candidates, while the secretary problem assumes you already figured that out and is trying to solve something different entirely!
  - bschne 3y ago
    This is the secretary problem [1] from optimal stopping theory, but it doesn't apply to hiring because you can wait until the end to make a decision once you've seen everyone. The main problem here indicated by OP isn't explore/exploit, it's that the ordering of candidates seemingly distorts their rating of them. 1. https://en.wikipedia.org/wiki/Secretary_problem https://en.wikipedia.org/wiki/Secretary_problem
babyshake 3y ago
What's the conventional wisdom for something like a demo day? Try to go first?
- airstrike 3y ago
  Their recommendation was to either be first or last. I guess my own twist is you don't have to be the very first or last—you have to be great and to try not to be right in the middle of the pack. But above all, do well and don't overthink this stuff as there's no scientific rigor. It doesn't help to be first or last and suck at it.
weinzierl 3y ago
What I learned in my years as proposal manager was that it is always best to get the first slot amongst the other bidders for your presentation and the last for the negotiation.
dhosek 3y ago
I remember while hiring for a new team member commenting about the first candidate that we interviewed that if we had seen him later in the sequence we likely would have hired him but coming first, he just seemed “ok.”
makeitdouble 3y ago
A decent way to normalize this effect is to take relatively detailed notes of the exchanges, and make the decision a few hours or a day later rereading the notes. One of the issues we had interviewing very young candidates (like fresh out of college) was how they all looked awkward, even in their outfit, and few had any kind of confidence during the 15~20 min we spent with them, with some saying really weird things (stuff like "I'm really good at the internet"). But obviously non of that matters long term, we assumed they'd probably all fit in fine once in. Putting some distance and picking up the good and bad stuff from the transcript helped a lot to get past the weird impressions, including the order we saw them and how tired we were when we saw them. To some point.
- kjqgqkejbfefn 3y ago
  >a few hours or a day later rereading the notes. https://onlinelibrary.wiley.com/doi/abs/10.1002/ejsp.52 https://onlinelibrary.wiley.com/doi/abs/10.1002/ejsp.52 >The durability of anchoring effects >Recent research suggests that judgmental anchoring is mediated by a selective increase in the accessibility of knowledge about the judgmental target. Anchoring thus constitutes one instance of the judgmental effects of increased knowledge accessibility. Such knowledge accessibility effects have repeatedly been demonstrated to be fairly durable, which suggests that the effects of judgmental anchoring may also persist over time. Consistent with this assumption, three experiments demonstrate that judgmental anchors influence judgment even if they were present one week before the critical judgment is made. In fact, the magnitude of anchoring was found to remain undiminished over this period of time.
  - makeitdouble 3y ago
    Thanks, that was pretty interesting. I didn't get access to the full text, but had a look at other papers from the same researcher [0] on what kind of methodology they use. In the case of recruiting, I think the main factor when moving the decision further down the line is the change in information ("a selective increase in the accessibility of knowledge about the judgmental target"), in two specific ways: - we actually remember less about the subject, for better or worse. A candidate might have had a weird look, and the notes are probably impacted by that bias, but we can look back at their coding test without that impression and come out with a slightly different conclusion. - we get to compare to other subjects in a different order. In particular, that helps catching weird expectations. For instance if every candidates has been falling through the same trap, it helps give them a pass and assume the question was at fault. If we had to do that in real time, only the last few ones would get a kinder judgement. [0] https://www.researchgate.net/publication/11394075_The_Malleability_of_Anchoring_Effects https://www.researchgate.net/publication/11394075_The_Mallea...
kjqgqkejbfefn 3y ago
Here's a study by someone from Kellogg university on this very subject: https://www.jcr-admin.org/preprints/chernev-preprint.pdf https://www.jcr-admin.org/preprints/chernev-preprint.pdf >Semantic Anchoring in Sequential Evaluations of Vices and Virtues >How do people evaluate sequentially presented items? Prior research suggests that sequential evaluations are subject to anchoring biases, such that the values of subsequently evaluated alternatives are assimilated toward the initially considered option. The present research argues, however, that sequential valuations often lead to contrast rather than assimilation effects, whereby values of the subsequently estimated alternatives are distanced from the initially evaluated option. These contrast effects are attributed to semantic anchoring, which stems from evaluating conceptually related options classified into opposing categories (e.g., vices and virtues).
gumby 3y ago
Seems like any mathematician interviewing candidates would know the secretary problem (https://en.wikipedia.org/wiki/Secretary_problem https://en.wikipedia.org/wiki/Secretary_problem) and thus going first would be the worst option.
- selcuka 3y ago
  Fortunately real world employers usually don't have to hire on the spot.
- avianlyric 3y ago
  The secretary problem only applies if you have to reject candidates before you’ve interviewed them all. Specifically it’s formulated as each candidate needs to be either accepted or rejected immediately after their interview. That situation obviously doesn’t apply if you’re interviewing many candidates on the same day and comparing them to each other. As a result to optimal strategy for the secretary problem also doesn’t apply.
daxfohl 3y ago
Seems like you could interview half the candidates and get a lot of time back.
- johnnyanmac 3y ago
  Or none of them because you already had an internal hire ready in the wing anyway.
begueradj 3y ago
Interesting. Do you have any explanation for that ?
wodenokoto 3y ago
Kellog, as in cornflakes?
- airstrike 3y ago
  no, as in the Kellogg School of Management

6510 3y ago

[flagged]

cyanydeez 3y ago

now to make it interesting, place the middle candidates right after lunch to see if having lunch improves the middle stretch

bschne 3y ago

(how) do you (try to) correct for this with interviews?

OJFord 3y ago
Someone's bound to mention it in this thread (or I hope they do, because I thought it was great but can't remember the name or the details) - but there's some formula that's roughly like: N = total applicants 1. Interview N/10, decline no matter what 2. Hire the next person who's better than everyone seen so far As I recall it was slightly more complex than that, perhaps only to give it an impression of being rigorous theory, but that was the gist of it.
- navane 3y ago
  I think you're looking for the Secretary problem. And the solution is almost what you said, but instead of N/10 it's a bit over N/3.
- wiml 3y ago
  This only applies if you have to make a hire/skip decision immediately after each interview and can't go back later. (The formulation I usually see is a princess choosing a suitor — for pride/face reasons, she can't pass on a suitor and then go back if it turns out he was the best of the bunch.) That's not typically how job interviews work, though. I think the magic number is 1/e of the applicant pool (e = Napier's constant, 2.7...)
- vikingerik 3y ago
  That mathematical problem (the secretary problem) is often cited, but it's not how real hiring works. The problem defines the only success criterion as hiring the single top candidate, and everything else fails. That's not at all what you're looking for in the real world - there are a spectrum of results, where any one in the top decile will be great and the next decile only slightly lesser and so on.
  - bschne 3y ago
    One-dimensional ranking is one aspect of how it breaks in real-world hiring, but I think almost the bigger might be that the secretary problem assumes you need to decide immediately and can't just wait around until you've seen the whole batch of applicants.
    - Dylan16807 3y ago
      The main difference is that you're trying to solve the problem of turning an interview into a way to choose between candidates, while the secretary problem assumes you already figured that out and is trying to solve something different entirely!
- bschne 3y ago
  This is the secretary problem [1] from optimal stopping theory, but it doesn't apply to hiring because you can wait until the end to make a decision once you've seen everyone. The main problem here indicated by OP isn't explore/exploit, it's that the ordering of candidates seemingly distorts their rating of them. 1. https://en.wikipedia.org/wiki/Secretary_problem https://en.wikipedia.org/wiki/Secretary_problem

babyshake 3y ago

What's the conventional wisdom for something like a demo day? Try to go first?

airstrike 3y ago
Their recommendation was to either be first or last. I guess my own twist is you don't have to be the very first or last—you have to be great and to try not to be right in the middle of the pack. But above all, do well and don't overthink this stuff as there's no scientific rigor. It doesn't help to be first or last and suck at it.

weinzierl 3y ago

What I learned in my years as proposal manager was that it is always best to get the first slot amongst the other bidders for your presentation and the last for the negotiation.

dhosek 3y ago

I remember while hiring for a new team member commenting about the first candidate that we interviewed that if we had seen him later in the sequence we likely would have hired him but coming first, he just seemed “ok.”

makeitdouble 3y ago

A decent way to normalize this effect is to take relatively detailed notes of the exchanges, and make the decision a few hours or a day later rereading the notes. One of the issues we had interviewing very young candidates (like fresh out of college) was how they all looked awkward, even in their outfit, and few had any kind of confidence during the 15~20 min we spent with them, with some saying really weird things (stuff like "I'm really good at the internet"). But obviously non of that matters long term, we assumed they'd probably all fit in fine once in. Putting some distance and picking up the good and bad stuff from the transcript helped a lot to get past the weird impressions, including the order we saw them and how tired we were when we saw them. To some point.

kjqgqkejbfefn 3y ago
>a few hours or a day later rereading the notes. https://onlinelibrary.wiley.com/doi/abs/10.1002/ejsp.52 https://onlinelibrary.wiley.com/doi/abs/10.1002/ejsp.52 >The durability of anchoring effects >Recent research suggests that judgmental anchoring is mediated by a selective increase in the accessibility of knowledge about the judgmental target. Anchoring thus constitutes one instance of the judgmental effects of increased knowledge accessibility. Such knowledge accessibility effects have repeatedly been demonstrated to be fairly durable, which suggests that the effects of judgmental anchoring may also persist over time. Consistent with this assumption, three experiments demonstrate that judgmental anchors influence judgment even if they were present one week before the critical judgment is made. In fact, the magnitude of anchoring was found to remain undiminished over this period of time.
- makeitdouble 3y ago
  Thanks, that was pretty interesting. I didn't get access to the full text, but had a look at other papers from the same researcher [0] on what kind of methodology they use. In the case of recruiting, I think the main factor when moving the decision further down the line is the change in information ("a selective increase in the accessibility of knowledge about the judgmental target"), in two specific ways: - we actually remember less about the subject, for better or worse. A candidate might have had a weird look, and the notes are probably impacted by that bias, but we can look back at their coding test without that impression and come out with a slightly different conclusion. - we get to compare to other subjects in a different order. In particular, that helps catching weird expectations. For instance if every candidates has been falling through the same trap, it helps give them a pass and assume the question was at fault. If we had to do that in real time, only the last few ones would get a kinder judgement. [0] https://www.researchgate.net/publication/11394075_The_Malleability_of_Anchoring_Effects https://www.researchgate.net/publication/11394075_The_Mallea...

kjqgqkejbfefn 3y ago

Here's a study by someone from Kellogg university on this very subject: https://www.jcr-admin.org/preprints/chernev-preprint.pdf https://www.jcr-admin.org/preprints/chernev-preprint.pdf >Semantic Anchoring in Sequential Evaluations of Vices and Virtues >How do people evaluate sequentially presented items? Prior research suggests that sequential evaluations are subject to anchoring biases, such that the values of subsequently evaluated alternatives are assimilated toward the initially considered option. The present research argues, however, that sequential valuations often lead to contrast rather than assimilation effects, whereby values of the subsequently estimated alternatives are distanced from the initially evaluated option. These contrast effects are attributed to semantic anchoring, which stems from evaluating conceptually related options classified into opposing categories (e.g., vices and virtues).

gumby 3y ago

Seems like any mathematician interviewing candidates would know the secretary problem (https://en.wikipedia.org/wiki/Secretary_problem https://en.wikipedia.org/wiki/Secretary_problem) and thus going first would be the worst option.

selcuka 3y ago
Fortunately real world employers usually don't have to hire on the spot.
avianlyric 3y ago
The secretary problem only applies if you have to reject candidates before you’ve interviewed them all. Specifically it’s formulated as each candidate needs to be either accepted or rejected immediately after their interview. That situation obviously doesn’t apply if you’re interviewing many candidates on the same day and comparing them to each other. As a result to optimal strategy for the secretary problem also doesn’t apply.

daxfohl 3y ago

Seems like you could interview half the candidates and get a lot of time back.

johnnyanmac 3y ago
Or none of them because you already had an internal hire ready in the wing anyway.

begueradj 3y ago

Interesting. Do you have any explanation for that ?

wodenokoto 3y ago

Kellog, as in cornflakes?

airstrike 3y ago
no, as in the Kellogg School of Management