7 ms·

Engineers can’t gauge their own interview performance

79 points by leeny 11y ago

soham 11y ago
Terrific analysis, thank you! I love reading your posts. I also think that the dynamics differ based on the mode of interviewing viz. Faceless (like http://interviewing.io http://interviewing.io does) vs face-ful (like normal onsite or video interviews, or the way http://interviewkickstart.com http://interviewkickstart.com does). I have an observation to share for the latter kind of interviews: When I was hiring at Box for one of my teams (or even now), I'd generally go towards the end, to meet the candidate and talk to them about how the day went. I'd ask them about specific interviews, the interviewers and how they think they did on the given problems. I found that it was surprisingly rare to get a candidate who was aware enough of how things went (positive or otherwise). Digging into it, I realized that the correlation of what the candidate perceived, was simply to how expressive the interviewER was. If the interviewer was friendly and chatty, the candidate would feel like they have done well (or at least forgivably well). If the interviewer was more silent, the candidate would feel they haven't done very well. Now that I do interview training for a living, I try and hammer this point home: that interviews are like a date. It depends a lot on the other person (in this case, the interviewer). Understanding this well often works like magic.
vonmoltke 11y ago
Hopefully this gets more love than it did last time: https://news.ycombinator.com/item?id=10739343 https://news.ycombinator.com/item?id=10739343 I found the analysis very interesting and consistent with my personal bias (I almost always think I did worse than I actually did).
- dang 11y ago
  > Hopefully this gets more love than it did last time We invited the submitter to repost it for that reason. I'll try to merge the comments from there into here. Invited reposts are mostly deprecated now in favor of re-ups [1], but when it looks like the submitter might also be the author (as e.g. with Show HNs), we still send them. It's nice for an author to know that their post may still get discussed, and it's good for HN when an author jumps into the thread. [1] All the details at https://news.ycombinator.com/item?id=10705926 https://news.ycombinator.com/item?id=10705926 and previous posts linked there.
  - leeny 11y ago
    Author here. Very much appreciated.
deleted 11y ago
[deleted]
invalidfunction 11y ago
Eh - humans generally have a hard time rating things on a 5 star rating system. That's mainly because the discrete intervals between 1 and 5 stars aren't well defined so everyone will have a different opinion of what 3 stars "mean". I encountered this when I ran a rating system in Mechanical Turk. If you ask turkers to do something arbitrary like "rate the quality of this picture" it takes quite a number of ratings on the same image to get an accurate consensus. It would be interesting to see if you can replicate experiment but replace the 5 stars with 5 yes/no questions regarding their performance and see if those match up
- InclinedPlane 11y ago
  Right, it's like game ratings. Is the minimum rating an "F" or is 60% an "F"? Is a rating in the middle good or bad?
- zyxley 11y ago
  I'm a fan of the Siskel & Ebert system instead of five stars: two thumbs down (bad), one thumb up/one thumb down (okay), two thumbs up (good).
- pc86 11y ago
  There are text descriptions under the stars, which are coincidentally rated out of 4, not 5. 3/4 is "Solid" for someone's technical skills. If someone does below average or amazing it's unlikely that the interviewer is going to select 3/4, regardless of their internal idea of what 3/4 means, when it says "Solid" underneath that.
- saulrh 11y ago
  Even worse because 1 star and 5 stars mean different things to different people, or even sometimes to the same person under different circumstances. If you asked me to rate my interview performance without first defining some objective criteria, for example, I'd have to spent a few minutes deciding whether (1 vs 5) stars is ("didn't get the job" vs "got hired with a nice title and negotiated a bunch of extra pay") or ("was arrested for assaulting the interviewer" vs "got the job").
- dogma1138 11y ago
cryoshon 11y ago
I've found this with myself as well. In one of my worst-ever interviews, I had the flu really badly the entire time, and read the signals of all of my interviewers to be both negative regarding my technical chops and negative toward my weakened state. I got the job, and was very well liked. In another trainwreck of an interview where I seriously considered standing up and leaving after getting extreme negative signals, I was offered the job as well-- but I should have paid attention to the negative signals, as they were indicative of larger problems with that group. In other interviews, I've massively overestimated my own performance, seeing positive signals everywhere and being very impressed with myself-- of course, these didn't pull through, and my confidence was dashed. In another incidence, I did get the job offer. The most confusing are the interviews in which I don't have a strong feeling of doing well or poorly either way-- not the median case, but certainly it happens frequently enough that I've gotten a job where I didn't think that I stood out. If anything, these cases are a consolation that others may see something positive that I don't see in myself. This tells me that my ability to correctly predict my interview performance is no more efficacious than random chance. I think that there are a few confounding factors which make interviewee measuring interview performance as perceived by the interviewer difficult. Interviews are a time of endless posturing, flagrant lies, propaganda, and overt deception hopefully mingled in with actual mutual interest, excitement, and good will. There are many social signals to keep track of in addition to self-monitoring to ensure the correct outcome. For people who are not the strongest socially, I think that these social signals tend to get dropped intermittently as attention shifts inward (don't say the wrong thing) or outward (get this technical problem correct).
- vdnkh 11y ago
  It's a fools errand to try and predict interview performance. I've been interviewing for a short while now and haven't gotten an offer yet despite feeling really great (and for some, really crappy) about my interviews. Feedback from the interviewers is equally inaccurate. On my most recent interview, I was told that my solution to their coding challenge was one of the best they'd ever seen, and that the feedback so far was extremely positive. Two days later I got the "another direction" email. I don't really take rejection to heart anymore but I hate not knowing what I need to do better.
cjrjdjcnd 11y ago
Nobody on HN has taken the PE or FE exams.
- vonmoltke 11y ago
  Neither have most of the engineers working for large corporations, unless it was required by their degree program (mine did not).
  - cjrjdjcnd 11y ago
    Except that's wrong. If you want to get hired with a job title of engineer, you at least have to take the FE exam. That is unless you're a webdev, in which case you can call yourself head fruitloop and it would still be acceptable, apparently
    - vonmoltke 11y ago
      That is absolutely false. I graduated with a BSEE and worked as an electrical engineer for over four years without ever attempting the FE. Nobody I knew, regardless of discipline, who graduated in the mid-90s or later has done so either. An FE is required in order to be hired at a firm providing engineering services "to the public", such as a civil engineering or architecture practice. It is not required, or in some cases even valued, at product companies.
    - mhurron 11y ago
      > If you want to get hired with a job title of engineer At the very least, not in the US you don't. /has title inflated job title of Engineer, not a web developer.
    - jnbiche 11y ago
      What is it with software developers who take such offense at the use of the word "engineer"? Clearly, you're not a "real" engineer, or you'd know that your statement about the use of the word "engineer" in titles was wrong in the US (and Canada doesn't use the FE exam). I've never met a engineer who gave a crap what my title was, software developer or software engineer. Which isn't a surprise, since we pose no threat to the jobs of "real" engineers. Not sure why the word is so threatening to some software developers. Do railroad engineers have to put up with this?
kelukelugames 11y ago
Are they being nice to me because I killed it or is it because I did so poorly that they are showing pity? Are they being mean because they think I wasted their time or because I killed it and they want to push me?
shoyer 11y ago
I actually interpret their data quite differently. There is a statistically robust and positive correlation between perceived and actual ratings: - 44% of the time, the actual and perceived ranking was identical. - 91% of the time, the difference between the rankings was at most one star. - Only in 9% of cases was there was a strong disagreement about how well the interview went (2 or more stars out of four). - There is also a bias towards engineers thinking they did slightly worse than they actually did. Even in an alternative universe where the same interview was repeated, you would not expect rankings by the same individual to be perfect. A more meaningful benchmark (though probably unattainable) would be interviewer/interviewee consistency relative to interviewer self-consistency.
- lordnacho 11y ago
  That's what I thought as well. I'm not sure it's a great idea to fit a straight line and then look at the R2. After all there's only three other values it can be, rather than a normal distribution.
- SEMW 11y ago
  Agreed. I think it's arguable that, when the question you're asking is 'how well do X and Y two things correlate with each other', the statistic you should be giving is r (which here is 0.49), not r^2. r^2 is the proportion of the variance of Y that's explained by X -- not that that isn't an interesting thing, but ITSM the correlation coefficient is more to what people expect a measure of relationship to be. (IANAStatistician)
- andimus 11y ago
  The problem with that interpretation is that it ignores how much something deviates from the expected random. If everything was completely random: - 25% of the time, the actual and perceived ranking would be identical. - 62.5% of the time, the difference would be at most one. - Only in 37.5% of cases would there be a strong difference in scores (2 or more stars out of four). Would you say that 44% is a good score on a multiple choice test? Obviously candidates did better than random guessing, but not by enough to call it a strong correlation. Determining the strength of a correlation is what R-squared is for, which makes it the right choice for this analysis.
exelius 11y ago
Nobody can judge their own interview performance - engineer or not. The hard part is that you as an interviewee don't really know what the interviewer is looking for. You may have nailed the back half of the interview, but you don't have the one skill they're looking for. Or sometimes you don't blow it at all and they love you, but there's one other person they interviewed for the position that blew them away. Trying to guess how well you did in an interview is folly. Hell, one interview tactic I use with people is to push them until they break: I keep asking technical questions until I reach a point where you say "I don't know". This sometimes makes people think they "failed" the interview, when all I'm really trying to do is see how much you know while also seeing how much you'll bullshit me on topics you don't know as well. Nobody can be an expert on everything, and interviews are as much about behavioral profiling as they are subject matter expertise.
edw519 11y ago
I never cared about my own "interview performance". But I always cared about my interviewer's assessment of me. So much so that I've always reserved the last 10 minutes of every interview just to find that out. They almost always ask if I have any questions, but even if they don't I ask them anyway: Based on what you've just learned: - How well would you say I did in this interview? - How well do my skills fill your requirement? - Where do you see me in the first month? - Where do you see me in 3 years? - What's the first thing you'd have me work on? - What gaps do you think I have? - What are your 3 biggest concerns about me? - What can I do to address those concerns? and of course: - What is the next step? I have never had an interviewer afraid to honestly (or appear to be honest) answer these questions. See how easy?
- someone7x 11y ago
  When I'm an interviewee I do the same thing. Opportunities to get interview feedback are already scarce, best to maximize them with questions like this. When I'm interviewing, I always do a portion at the end to tell them at least one thing that impressed me and one thing that gave me pause.
  - woebtz 11y ago
    That's an incredibly empathetic approach to interviewing, I'll try to keep that method in mind. I tend to be overly critical to a fault. How do you phrase or bring up your praise/concerns?
- NickLarsen 11y ago
  The last 10 minutes where you ask questions is a bad time to waste asking someone to judge your 3 year plan when they just met you. Most of these questions are better asked when you fail an interview and request feedback as to why. I specifically state when I perform interviews that I will not answer the question "How well would you say I did in this interview?" The interview isn't over yet, and that's impossible to answer. You should really spend this time to see how much you would want to work at the company. Ask about their processes, the things that would absolutely turn you off to a company or the things that would absolutely turn you on to a company and make you forget about all their other bullshit. Ask about the problems they are trying to solve. Ask about how smart the other people are who you'll be working with. Don't waste time in the interview checking if you passed. You'll know very soon anyway and there is always the chance to get this feedback afterward.
lewisl9029 11y ago
> "resumes suck" Do they? Maybe resume's specifically do suck, but surely it's beneficial to have at least some way to see an overview of what a candidate has accomplished in the past. Especially when you're hiring for more senior roles where you need to place more value on experience and a proven track record than raw technical excellence. I can't really see these "anonymous" interviewing platforms working well for those kinds of roles.
- 6d0debc071 11y ago
  > Do they? I have seen hundreds of CVs that show no achievements of note - and tend to be more along the lines of describing the duties of the job and little else besides. I've also had hundreds of subsequent conversations with those people that revealed significant accomplishments. CVs are better than nothing. But there's no set way to write them that's widely known, and people tend to be rather reticent about their own accomplishments - perhaps not even thinking of them as such. There's a lot of value being left on the table.
  - vonmoltke 11y ago
    Furthermore, not everyone's accomplishments boil down to nice bullet points.
jMyles 11y ago
...another reasonable conclusion from this data is that interviewers have difficulty judging interview performance.
- deleted 11y ago
  [deleted]
- JFlash 11y ago
  Beat me to it. When I interviewed at Google there were multiple interviewers and then there were multiple committees and people who look at their feedback. Comparing interviewers to other interviewers makes a "He did above average" from a person who's hard to impress as good as a "She did really great!" from someone else.
  - robrenaud 11y ago
    Well, the hiring committees see the entire prior rating distribution from each interviewer as well as the hires/no hires for each previous rating. So the committees has a decent way of correcting for interviewers who use different scales.
    - JFlash 11y ago
      Right. You'd only know someone was "hard to impress" with that information.
- krick 11y ago
  Interviewers kinda define interview performance. That's what interview performance is: how good did you appear to the interviewer.
  - JFlash 11y ago
    Being easy to work with is certainly something interviewers should look for, but software doesn't care how nice you are. Your technical competency should ideally be measured outside of any subjectivity. Of course that's not possible but companies try their best.
joeax 11y ago
I was once in an interview where the interviewer was a hard ass. Asking all kids of hard questions about languages I knew, then he got deeper and asked about specifics into those languages. Then started asking about really abstract stuff like different sorting algorithms, OOP concepts like polymorphism, etc. I felt I was holding my own but as the interview went on the interviewer seemed to grow more impatient and irritable. Needless to say I left the interview thinking I blew it. Two days later I got a call from the recruiter saying I was their top choice and got an offer soon after. P.S. It turned out that guy was just grouchy because he had interviewed so many underwhelming candidates. He admitted he was just trying to break me.
- pgodzin 11y ago
  Trying to break a candidate seems to be very related issue to the one the article is talking about. Most people will take that kind of interviewer attitude as a negative experience that reflects poorly on the company and would make them less interested in taking the job. The goal of the interviewer should be to evaluate a candidate, not dig and dig until you find a weakness. If it takes that much effort it's not worth doing in the first place.
  - joeax 11y ago
    Agree. Luckily attitudes have shifted over the years and companies realize there's too much risk in alienating a good candidate.
- kvcrawford 11y ago
  > He admitted he was just trying to break me Sounds like a great guy to work with.
  - joeax 11y ago
    He softened up a bit after a year. Some people you just have to win their respect.
stcredzero 11y ago
So many of you 20-somethings interviewing really make me cringe! Basically, many of you are a stark mirror in which I can see how terrible I was as an interviewer in my 20's and 30's. Be honest now, do you do this? 1) See something or think of something 2) Decide that it's the most important point 3) Go on a "fishing" expedition to try and get the interviewee to see it 4) If the interviewee can't figure out what you're hinting at, decide s/he's completely worthless! (Or if she gets it, decide she's the ultimate!) A lot of you do this. Cut it out. It's idiotic. It presupposes a pretty arrogant view of the relative value of your own perceptions. What you really need to be doing is listening. I had the experience of someone doing this to me with concurrency, then concluding I'm an impostor, while I'm telling him about 3 concurrent systems I've worked on. The disconnect? He was "fishing" for the current undergrad spiel about scheduling. Sorry, we covered it, but that wasn't emphasized in the same way when I was an undergrad, and the poorly thought out puzzle you came up with on the spot wasn't such a great hint. Given the number of times I was mis-identified as an impostor programmer during interviews (a minority of instances, but still) there is clearly something going on. You guys are just like the young Ivy league grads in Mad Men. You just have different clothes and different language, but you are looking out for the signs of your own tribe just the same.
- ArkyBeagle 11y ago
  Ah, the password game. You don't wanna work for those guys anyway. It's interesting isn't it? This was all much easier in decades past. The questions don't matter. The answer doesn't matter. What matters are the body language cues.
- konradb 11y ago
  Out of interest what is this current undergrad spiel about scheduling?
lazyant 11y ago
So Dunning–Kruger effect (poor candidates over valuing themselves) and impostor syndrome (the opposite).
- skue 11y ago
  Dunning-Kruger actually accounts for both.[1] But agreed, this isn't specific to hiring. [1] https://en.m.wikipedia.org/wiki/Dunning–Kruger_effect https://en.m.wikipedia.org/wiki/Dunning–Kruger_effect
deleted 11y ago
[deleted]
imh 11y ago
>And by extension, it means that in every interview cycle, some portion of interviewees are losing interest in joining your company just because they didn’t think they did well, despite the fact that they actually did. I hate the assertion of causation here. As always, this relationship means that A causes B, B causes A, and/or A and B have a common cause (or even more possibilities, given that we're essentially conditioning on this event taking place). Aline argues that "some portion of interviewees are losing interest in joining your company just because they didn’t think they did well," but I think it's more plausible that there is something that happens in the interview that simultaneously makes the interviewer think they did poorly and makes them think poorly of the company. If you're rating me on my ability to name country capitals in a statistics interview, I'm going to think you don't want to hire me and I'll think you're an idiot I'd hate to work for. Or if you're being kind of a jerk or being impatient and checking your cell phone every 30 seconds, I'll think I'm doing poorly and that you're rude and I wouldn't like to work with you. Common cause seems much more likely to me.

79 points by leeny 11y ago

soham 11y ago

Terrific analysis, thank you! I love reading your posts. I also think that the dynamics differ based on the mode of interviewing viz. Faceless (like http://interviewing.io http://interviewing.io does) vs face-ful (like normal onsite or video interviews, or the way http://interviewkickstart.com http://interviewkickstart.com does). I have an observation to share for the latter kind of interviews: When I was hiring at Box for one of my teams (or even now), I'd generally go towards the end, to meet the candidate and talk to them about how the day went. I'd ask them about specific interviews, the interviewers and how they think they did on the given problems. I found that it was surprisingly rare to get a candidate who was aware enough of how things went (positive or otherwise). Digging into it, I realized that the correlation of what the candidate perceived, was simply to how expressive the interviewER was. If the interviewer was friendly and chatty, the candidate would feel like they have done well (or at least forgivably well). If the interviewer was more silent, the candidate would feel they haven't done very well. Now that I do interview training for a living, I try and hammer this point home: that interviews are like a date. It depends a lot on the other person (in this case, the interviewer). Understanding this well often works like magic.

vonmoltke 11y ago

Hopefully this gets more love than it did last time: https://news.ycombinator.com/item?id=10739343 https://news.ycombinator.com/item?id=10739343 I found the analysis very interesting and consistent with my personal bias (I almost always think I did worse than I actually did).

dang 11y ago
> Hopefully this gets more love than it did last time We invited the submitter to repost it for that reason. I'll try to merge the comments from there into here. Invited reposts are mostly deprecated now in favor of re-ups [1], but when it looks like the submitter might also be the author (as e.g. with Show HNs), we still send them. It's nice for an author to know that their post may still get discussed, and it's good for HN when an author jumps into the thread. [1] All the details at https://news.ycombinator.com/item?id=10705926 https://news.ycombinator.com/item?id=10705926 and previous posts linked there.
- leeny 11y ago
  Author here. Very much appreciated.

deleted 11y ago

[deleted]

invalidfunction 11y ago

Eh - humans generally have a hard time rating things on a 5 star rating system. That's mainly because the discrete intervals between 1 and 5 stars aren't well defined so everyone will have a different opinion of what 3 stars "mean". I encountered this when I ran a rating system in Mechanical Turk. If you ask turkers to do something arbitrary like "rate the quality of this picture" it takes quite a number of ratings on the same image to get an accurate consensus. It would be interesting to see if you can replicate experiment but replace the 5 stars with 5 yes/no questions regarding their performance and see if those match up

InclinedPlane 11y ago
Right, it's like game ratings. Is the minimum rating an "F" or is 60% an "F"? Is a rating in the middle good or bad?
zyxley 11y ago
I'm a fan of the Siskel & Ebert system instead of five stars: two thumbs down (bad), one thumb up/one thumb down (okay), two thumbs up (good).
pc86 11y ago
There are text descriptions under the stars, which are coincidentally rated out of 4, not 5. 3/4 is "Solid" for someone's technical skills. If someone does below average or amazing it's unlikely that the interviewer is going to select 3/4, regardless of their internal idea of what 3/4 means, when it says "Solid" underneath that.
saulrh 11y ago
Even worse because 1 star and 5 stars mean different things to different people, or even sometimes to the same person under different circumstances. If you asked me to rate my interview performance without first defining some objective criteria, for example, I'd have to spent a few minutes deciding whether (1 vs 5) stars is ("didn't get the job" vs "got hired with a nice title and negotiated a bunch of extra pay") or ("was arrested for assaulting the interviewer" vs "got the job").
dogma1138 11y ago

cryoshon 11y ago

I've found this with myself as well. In one of my worst-ever interviews, I had the flu really badly the entire time, and read the signals of all of my interviewers to be both negative regarding my technical chops and negative toward my weakened state. I got the job, and was very well liked. In another trainwreck of an interview where I seriously considered standing up and leaving after getting extreme negative signals, I was offered the job as well-- but I should have paid attention to the negative signals, as they were indicative of larger problems with that group. In other interviews, I've massively overestimated my own performance, seeing positive signals everywhere and being very impressed with myself-- of course, these didn't pull through, and my confidence was dashed. In another incidence, I did get the job offer. The most confusing are the interviews in which I don't have a strong feeling of doing well or poorly either way-- not the median case, but certainly it happens frequently enough that I've gotten a job where I didn't think that I stood out. If anything, these cases are a consolation that others may see something positive that I don't see in myself. This tells me that my ability to correctly predict my interview performance is no more efficacious than random chance. I think that there are a few confounding factors which make interviewee measuring interview performance as perceived by the interviewer difficult. Interviews are a time of endless posturing, flagrant lies, propaganda, and overt deception hopefully mingled in with actual mutual interest, excitement, and good will. There are many social signals to keep track of in addition to self-monitoring to ensure the correct outcome. For people who are not the strongest socially, I think that these social signals tend to get dropped intermittently as attention shifts inward (don't say the wrong thing) or outward (get this technical problem correct).

vdnkh 11y ago
It's a fools errand to try and predict interview performance. I've been interviewing for a short while now and haven't gotten an offer yet despite feeling really great (and for some, really crappy) about my interviews. Feedback from the interviewers is equally inaccurate. On my most recent interview, I was told that my solution to their coding challenge was one of the best they'd ever seen, and that the feedback so far was extremely positive. Two days later I got the "another direction" email. I don't really take rejection to heart anymore but I hate not knowing what I need to do better.

cjrjdjcnd 11y ago

Nobody on HN has taken the PE or FE exams.

vonmoltke 11y ago
Neither have most of the engineers working for large corporations, unless it was required by their degree program (mine did not).
- cjrjdjcnd 11y ago
  Except that's wrong. If you want to get hired with a job title of engineer, you at least have to take the FE exam. That is unless you're a webdev, in which case you can call yourself head fruitloop and it would still be acceptable, apparently
  - vonmoltke 11y ago
    That is absolutely false. I graduated with a BSEE and worked as an electrical engineer for over four years without ever attempting the FE. Nobody I knew, regardless of discipline, who graduated in the mid-90s or later has done so either. An FE is required in order to be hired at a firm providing engineering services "to the public", such as a civil engineering or architecture practice. It is not required, or in some cases even valued, at product companies.
  - mhurron 11y ago
    > If you want to get hired with a job title of engineer At the very least, not in the US you don't. /has title inflated job title of Engineer, not a web developer.
  - jnbiche 11y ago
    What is it with software developers who take such offense at the use of the word "engineer"? Clearly, you're not a "real" engineer, or you'd know that your statement about the use of the word "engineer" in titles was wrong in the US (and Canada doesn't use the FE exam). I've never met a engineer who gave a crap what my title was, software developer or software engineer. Which isn't a surprise, since we pose no threat to the jobs of "real" engineers. Not sure why the word is so threatening to some software developers. Do railroad engineers have to put up with this?

kelukelugames 11y ago

Are they being nice to me because I killed it or is it because I did so poorly that they are showing pity? Are they being mean because they think I wasted their time or because I killed it and they want to push me?

shoyer 11y ago

I actually interpret their data quite differently. There is a statistically robust and positive correlation between perceived and actual ratings: - 44% of the time, the actual and perceived ranking was identical. - 91% of the time, the difference between the rankings was at most one star. - Only in 9% of cases was there was a strong disagreement about how well the interview went (2 or more stars out of four). - There is also a bias towards engineers thinking they did slightly worse than they actually did. Even in an alternative universe where the same interview was repeated, you would not expect rankings by the same individual to be perfect. A more meaningful benchmark (though probably unattainable) would be interviewer/interviewee consistency relative to interviewer self-consistency.

lordnacho 11y ago
That's what I thought as well. I'm not sure it's a great idea to fit a straight line and then look at the R2. After all there's only three other values it can be, rather than a normal distribution.
SEMW 11y ago
Agreed. I think it's arguable that, when the question you're asking is 'how well do X and Y two things correlate with each other', the statistic you should be giving is r (which here is 0.49), not r^2. r^2 is the proportion of the variance of Y that's explained by X -- not that that isn't an interesting thing, but ITSM the correlation coefficient is more to what people expect a measure of relationship to be. (IANAStatistician)
andimus 11y ago
The problem with that interpretation is that it ignores how much something deviates from the expected random. If everything was completely random: - 25% of the time, the actual and perceived ranking would be identical. - 62.5% of the time, the difference would be at most one. - Only in 37.5% of cases would there be a strong difference in scores (2 or more stars out of four). Would you say that 44% is a good score on a multiple choice test? Obviously candidates did better than random guessing, but not by enough to call it a strong correlation. Determining the strength of a correlation is what R-squared is for, which makes it the right choice for this analysis.

exelius 11y ago

Nobody can judge their own interview performance - engineer or not. The hard part is that you as an interviewee don't really know what the interviewer is looking for. You may have nailed the back half of the interview, but you don't have the one skill they're looking for. Or sometimes you don't blow it at all and they love you, but there's one other person they interviewed for the position that blew them away. Trying to guess how well you did in an interview is folly. Hell, one interview tactic I use with people is to push them until they break: I keep asking technical questions until I reach a point where you say "I don't know". This sometimes makes people think they "failed" the interview, when all I'm really trying to do is see how much you know while also seeing how much you'll bullshit me on topics you don't know as well. Nobody can be an expert on everything, and interviews are as much about behavioral profiling as they are subject matter expertise.

edw519 11y ago

I never cared about my own "interview performance". But I always cared about my interviewer's assessment of me. So much so that I've always reserved the last 10 minutes of every interview just to find that out. They almost always ask if I have any questions, but even if they don't I ask them anyway: Based on what you've just learned: - How well would you say I did in this interview? - How well do my skills fill your requirement? - Where do you see me in the first month? - Where do you see me in 3 years? - What's the first thing you'd have me work on? - What gaps do you think I have? - What are your 3 biggest concerns about me? - What can I do to address those concerns? and of course: - What is the next step? I have never had an interviewer afraid to honestly (or appear to be honest) answer these questions. See how easy?

someone7x 11y ago
When I'm an interviewee I do the same thing. Opportunities to get interview feedback are already scarce, best to maximize them with questions like this. When I'm interviewing, I always do a portion at the end to tell them at least one thing that impressed me and one thing that gave me pause.
- woebtz 11y ago
  That's an incredibly empathetic approach to interviewing, I'll try to keep that method in mind. I tend to be overly critical to a fault. How do you phrase or bring up your praise/concerns?
NickLarsen 11y ago
The last 10 minutes where you ask questions is a bad time to waste asking someone to judge your 3 year plan when they just met you. Most of these questions are better asked when you fail an interview and request feedback as to why. I specifically state when I perform interviews that I will not answer the question "How well would you say I did in this interview?" The interview isn't over yet, and that's impossible to answer. You should really spend this time to see how much you would want to work at the company. Ask about their processes, the things that would absolutely turn you off to a company or the things that would absolutely turn you on to a company and make you forget about all their other bullshit. Ask about the problems they are trying to solve. Ask about how smart the other people are who you'll be working with. Don't waste time in the interview checking if you passed. You'll know very soon anyway and there is always the chance to get this feedback afterward.

lewisl9029 11y ago

> "resumes suck" Do they? Maybe resume's specifically do suck, but surely it's beneficial to have at least some way to see an overview of what a candidate has accomplished in the past. Especially when you're hiring for more senior roles where you need to place more value on experience and a proven track record than raw technical excellence. I can't really see these "anonymous" interviewing platforms working well for those kinds of roles.

6d0debc071 11y ago
> Do they? I have seen hundreds of CVs that show no achievements of note - and tend to be more along the lines of describing the duties of the job and little else besides. I've also had hundreds of subsequent conversations with those people that revealed significant accomplishments. CVs are better than nothing. But there's no set way to write them that's widely known, and people tend to be rather reticent about their own accomplishments - perhaps not even thinking of them as such. There's a lot of value being left on the table.
- vonmoltke 11y ago
  Furthermore, not everyone's accomplishments boil down to nice bullet points.

jMyles 11y ago

...another reasonable conclusion from this data is that interviewers have difficulty judging interview performance.

deleted 11y ago
[deleted]
JFlash 11y ago
Beat me to it. When I interviewed at Google there were multiple interviewers and then there were multiple committees and people who look at their feedback. Comparing interviewers to other interviewers makes a "He did above average" from a person who's hard to impress as good as a "She did really great!" from someone else.
- robrenaud 11y ago
  Well, the hiring committees see the entire prior rating distribution from each interviewer as well as the hires/no hires for each previous rating. So the committees has a decent way of correcting for interviewers who use different scales.
  - JFlash 11y ago
    Right. You'd only know someone was "hard to impress" with that information.
krick 11y ago
Interviewers kinda define interview performance. That's what interview performance is: how good did you appear to the interviewer.
- JFlash 11y ago
  Being easy to work with is certainly something interviewers should look for, but software doesn't care how nice you are. Your technical competency should ideally be measured outside of any subjectivity. Of course that's not possible but companies try their best.

joeax 11y ago

I was once in an interview where the interviewer was a hard ass. Asking all kids of hard questions about languages I knew, then he got deeper and asked about specifics into those languages. Then started asking about really abstract stuff like different sorting algorithms, OOP concepts like polymorphism, etc. I felt I was holding my own but as the interview went on the interviewer seemed to grow more impatient and irritable. Needless to say I left the interview thinking I blew it. Two days later I got a call from the recruiter saying I was their top choice and got an offer soon after. P.S. It turned out that guy was just grouchy because he had interviewed so many underwhelming candidates. He admitted he was just trying to break me.

pgodzin 11y ago
Trying to break a candidate seems to be very related issue to the one the article is talking about. Most people will take that kind of interviewer attitude as a negative experience that reflects poorly on the company and would make them less interested in taking the job. The goal of the interviewer should be to evaluate a candidate, not dig and dig until you find a weakness. If it takes that much effort it's not worth doing in the first place.
- joeax 11y ago
  Agree. Luckily attitudes have shifted over the years and companies realize there's too much risk in alienating a good candidate.
kvcrawford 11y ago
> He admitted he was just trying to break me Sounds like a great guy to work with.
- joeax 11y ago
  He softened up a bit after a year. Some people you just have to win their respect.

stcredzero 11y ago

So many of you 20-somethings interviewing really make me cringe! Basically, many of you are a stark mirror in which I can see how terrible I was as an interviewer in my 20's and 30's. Be honest now, do you do this? 1) See something or think of something 2) Decide that it's the most important point 3) Go on a "fishing" expedition to try and get the interviewee to see it 4) If the interviewee can't figure out what you're hinting at, decide s/he's completely worthless! (Or if she gets it, decide she's the ultimate!) A lot of you do this. Cut it out. It's idiotic. It presupposes a pretty arrogant view of the relative value of your own perceptions. What you really need to be doing is listening. I had the experience of someone doing this to me with concurrency, then concluding I'm an impostor, while I'm telling him about 3 concurrent systems I've worked on. The disconnect? He was "fishing" for the current undergrad spiel about scheduling. Sorry, we covered it, but that wasn't emphasized in the same way when I was an undergrad, and the poorly thought out puzzle you came up with on the spot wasn't such a great hint. Given the number of times I was mis-identified as an impostor programmer during interviews (a minority of instances, but still) there is clearly something going on. You guys are just like the young Ivy league grads in Mad Men. You just have different clothes and different language, but you are looking out for the signs of your own tribe just the same.

ArkyBeagle 11y ago
Ah, the password game. You don't wanna work for those guys anyway. It's interesting isn't it? This was all much easier in decades past. The questions don't matter. The answer doesn't matter. What matters are the body language cues.
konradb 11y ago
Out of interest what is this current undergrad spiel about scheduling?

lazyant 11y ago

So Dunning–Kruger effect (poor candidates over valuing themselves) and impostor syndrome (the opposite).

skue 11y ago
Dunning-Kruger actually accounts for both.[1] But agreed, this isn't specific to hiring. [1] https://en.m.wikipedia.org/wiki/Dunning–Kruger_effect https://en.m.wikipedia.org/wiki/Dunning–Kruger_effect

deleted 11y ago

[deleted]

imh 11y ago

>And by extension, it means that in every interview cycle, some portion of interviewees are losing interest in joining your company just because they didn’t think they did well, despite the fact that they actually did. I hate the assertion of causation here. As always, this relationship means that A causes B, B causes A, and/or A and B have a common cause (or even more possibilities, given that we're essentially conditioning on this event taking place). Aline argues that "some portion of interviewees are losing interest in joining your company just because they didn’t think they did well," but I think it's more plausible that there is something that happens in the interview that simultaneously makes the interviewer think they did poorly and makes them think poorly of the company. If you're rating me on my ability to name country capitals in a statistics interview, I'm going to think you don't want to hire me and I'll think you're an idiot I'd hate to work for. Or if you're being kind of a jerk or being impatient and checking your cell phone every 30 seconds, I'll think I'm doing poorly and that you're rude and I wouldn't like to work with you. Common cause seems much more likely to me.