6 ms·

Encouraging interviewees to say “I don’t know” improves performance

45 points by dncrane 13y ago

triplesec 13y ago
Corollary: "There's no such thing as a stupid question", because you're not afraid to be wrong. All creative and innovative people are like this, without exception, IMO. As Richard Feynman said in the title of an autobiography: "What do you care what other people think?" Everybody wins when people leave their egos behind.
- ape4 13y ago
  Sure. But its not so hard to come up with a question that sounds pretty stupid. What is Monday? Can I eat your shoe for lunch? Why do you have to interview me before hiring me?
croddin 13y ago
When reading the title, I assumed this was talking about job interviews, a much different type of interview. How do these ideas apply to job interviews? Is it a good idea for an interviewer in a job interview to encourage interviewees to say I don't know, and should you be more ok with saying I don't know and then elaborating in a job interview?
- Peroni 13y ago
  Purely anecdotal of course but I always set the scene for interviewee's that "I don't know" is a completely acceptable answer. If the question could be attempted with an educated guess then "I don't know" is perfectly fine as long as an educated guess is then attempted. If you only get "I don't know" with no attempt at a guess or explanation for why they don't know then you may have issues.
- bennyg 13y ago
  On the job interview point: I'm always as truthful as possible in a job interview - especially technical ones. My background is not in tech, however I'm an autodidact with programming and whenever technical questions are asked that I may not know, I'll say it. And then I'll say how I think it should/would be done, or the steps I'd take to figure out the answer and solve the problem. Not sugarcoating some of my technical inability has faired me well so far. And then after I hear the question, think about the steps to do it, and/or get coached by the interviewee - I know the answer, and am ready to use it in my technical/creative arsenal. I feel like it's a win-win to not bullshit about something I don't know.
- TheChineseGirl 13y ago
  Without encouragement, candidates answer with "I don't know" during a job interview easily stands out. If we are interviewing for a position, aren't we searching for just ONE outstanding specific individual? Should we really explicitly encourage?
- elq 13y ago
  A perfectly acceptable answer to me is "I don't know, but this is what I would do to find out", go on to describe what books you'd refer to, what google searches, etc.
- jfarmer 13y ago
  Those are two separate questions, IMO. In the context of a job interview, however, I believe giving the interviewee permission to say "I don't know" makes the interview go better because they'll spend more time focusing on the interview and less time trying to figure out "what the interviewer wants." I usually give a little preamble telling them to feel comfortable blurting out nonsense. I'd rather have them blurt out nonsense than sit there for 30 seconds in silence because they're afraid to say something stupid. You can see the interviewee relax almost immediately.
- bcbrown 13y ago
  My view is that if an interviewer dings me for saying "I don't know", that's a sign I wouldn't want to work there anyway.
agumonkey 13y ago
lucidity as the fastest path to clarity ?
samograd 13y ago
I'll usually ask at least one question in an interview to see if the candidate will answer with "I don't know" or ramble on incoherently making something up. Guess which I prefer.
asgard1024 13y ago
I have long suspected this to be true. That's my argument against mandatory voting, because if people can choose not to vote, they can express they don't know and hopefully leave the result on people who feel they do know. However, it's a bit unclear how this ties together with Dunning-Kruger effect (which is also used as an argument against democracy). The D-K effect would suggest that if you allow people to express doubt, the experts will doubt more, and overall performance will decrease. It would be interesting if someone did a psychology experiment on that. (I am actually in favor of doing these kinds of demonstrations in high school, because it's in my opinion important for people to understand how democratic voting works, for example the fact that voting won't get you a simple average of the results - which is usually used as an argument against democracy.)
- mistercow 13y ago
  I'd actually be pretty surprised if the Dunning-Kruger effect applied significantly to politics. There are so many tribal pressures involved that I think that the phenomena involved with Dunning-Kruger would be overwhelmed.
  - asgard1024 13y ago
    In representative democracy, probably yes. But D-K effect is commonly used as an argument against (semi)direct democracy.
- forktheif 13y ago
  You can have mandatory voting with a "none of the above" option on the ballot.
  - asgard1024 13y ago
    And the point of such a system is? To spend more money and time of everyone?
    - anonymfus 13y ago
judk 13y ago
This is obvious. See also the child abuse panic of the 80s, where kids were pressured to make up stories of abuse. And the general fact that eyewitness testimony is unreliable.
beat 13y ago
One of my best technical interviews was from a lead engineer who simply asked more and more difficult questions until he found your limit (believe me, he knew more than anyone he would interview). His interest wasn't in what you knew so much as how you reacted when you hit the limits of your knowledge. There's a lot of value in that approach. I don't use it myself when interviewing people, but I do try to accomplish the same end.
- mistercow 13y ago
  >believe me, he knew more than anyone he would interview He wouldn't really need to. By asking questions about very domain-specific knowledge, he could still achieve the same effect with anyone who didn't have a freakishly similar set of expertise.
pron 13y ago
It's a nice idea, and it makes sense, but the sample size – 26 people per group, multiplied by 2 for two experiments – is too small to reach any definite conclusions.
gcb0 13y ago
One thing i always ask on that chat before the trivia questions, is what tech news sites the candidate read. And where he usually goes to were he wants to know the correct argument order of a function he does not use too often. That usually gives me a better grasp of how he will solve a real world problem than the trivia questions. But when hiring for teams that need to get specific work done fast, i do not underestimate the value of the trivia questions as it will show me the ramp up time to get that work done. Of course i apply that on top of the other means to get an idea if the candidate is a good fit overall. There is no silver bullet for hiring.
WhitneyLand 13y ago
How about just ditching contrived interview questions because companies have no good science or data to establish the correlation with hiring the best people? It seems past performance would be more reliable. What have you created and what was your role in it? Can you articulate your passion for the problems we are solving? Can you show that your colleagues respected your work and benefited from collaborating with you?
lmkg 13y ago
I think part of this conditioning from taking tests in school. For twelve or more years of your life, in any situation where you are asked to demonstrate your knowledge, a null response is treated in the same negative fashion as an incorrect response, and getting outside help is disallowed. No wonder we try and guess, we've literally been trained for years that it's the best approach! Compounding that, an interview is the one situation in "the real world" that most resembles a testing situation from grade school, which will bring back all those bad habits.
- tootie 13y ago
  Actually, the SAT (American college entrance exam) penalizes wrong answers more than non-answers. A non-answer gets you no points, but a wrong answer gets points subtracted.
  - j_baker 13y ago
    The SAT is the exception rather than the rule. Most standardized and non-standardized tests reward you for guessing.
    - mathattack 13y ago
      The GMAT as well. It's interesting when you work with little kids - you have to encourage them to guess. They're more likely to admit what they don't know.
  - iyulaev 13y ago
    Barely. IIRC it's 0 points if blank, -0.25 if wrong, so if you eliminate 1 choice it's worth it to guess. I'd like to see it be -5 points if wrong, since this is more reflective of the real world. When I entered the industry as an electrical engineer this was a difficult transition for me. In school, you're incentivized to guess and BS. But in the real world, the penalty for being wrong is HUGE.
    - wwweston 13y ago
patrickyeon 13y ago
Some commenters, and possibly the author, want this advice to extend to job interviews, but the reason an interviewer is pushing you to answer a question isn't that they want you to be right or wrong. A good interviewer wants to get a look into how you think and how you approach a problem. An even better interviewer will make that clear, by telling you "there's no trick here", "I don't know the answer myself, let's see what we can figure out", or "there's no one right answer, I just want to know what you can see here". A trick I've employed a fair bit lately to get a reserved interviewee to start working with a question is "What is the worst solution we could provide to attack this problem?" I'll even possibly go as far as offering my own horrible solution, and asking them where we fo to improve on this. And I do mean worst. I haven't met a candidate yet who can't at least throw out ideas on how to improve my horrible solutions, and at that point the ball is rolling.
- RokStdy 13y ago
  Asking somebody for the worst possible solution seems like a really clever ice-breaker. I'm stealing that technique! Thanks patrickyeon! I interviewed some folks a while ago for a simple desktop support position and I had a lot of trouble getting anything out of the candidates. It's possible that they were all just terrible, but I'm guessing it was more probably my approach. I gave them a very simple test[1], and I was met with sort of deer-in-headlights stares. Hence forth I think I'll start with a problem and an example horrible solution. [1]Presented the applicant with a screenshot from a desktop computer that couldn't connect to the internet. The screenshot showed several possible indicators (impossible network settings, cable unplugged, etc...).
- j_baker 13y ago
  > A good interviewer wants to get a look into how you think and how you approach a problem. I consider that a red flag. In this case, the interviewer is almost certainly less concerned with whether or not you can do the job than they are concerned with whether you think like them. Though I do like your suggestion to try offering a terrible solution to get the conversation started. Usually it's breaking the ice that's difficult.
  - 13y ago
m0th87 13y ago
A good interview question should give space for exploration - it shouldn't involve esoteric questions or things that are easily google-able. Good interview questions will give you insight into the candidate's thought process, not what they've memorized beyond fundamentals. So the results of this study seem irrelevant.
Spoom 13y ago
I've found that the ease with which someone answers "I don't know" has a positive correlation with their wisdom in general. (Of course, if they answer "I don't know" to everything, they probably just don't care, and that's a separate problem.)
- TheChineseGirl 13y ago
  I second this notion.
- yeukhon 13y ago
  I don't necessarily disagree with you. I think it is true that having too many I don't know means the person is probably not familiar with the area he's facing. The I don't care part is probably a bit rare. You probably have to look at body language (i.e. facial expression, for example).
ufmace 13y ago
Being comfortable with saying "I don't know" a lot is something I had to kind of train myself to do. It's part of realizing that this is a truly enormous field that we're in, and nobody could possibly know more than a fraction of it. Anyone who doesn't say it much is either spending all of their time in a subset of the field that they know very well, or is throwing around bullshit, either from fear or bravado.
mathattack 13y ago
The answers may get better, but don't you want to find out which candidates are more likely to bluff when they're wrong? The point of the interview isn't for candidates to give you answers that are as correct as possible. If this were the case, you'd mail them a list of questions, and ask them to get back to you in a week. One goal of interviews is to find out how people behave when they don't know. This is why I try to take people to the end of their knowledge, no matter where that end is.
yeukhon 13y ago
Nice study. I think it's quite intuitive. When you are forced to spell you are force to say something. So by saying "I don't know" it is natural to continue with "I don't know but I think" and then it continues with more words. Whereas to tell someone in advance "don't you dare to tell me I don't know" the person will just hold on to the thoughts until he or she has to spell that thought out (because he or she has to say something to get out of the misery).
adamconroy 13y ago
In an interview i have never had a problem saying that I don't know something or that I do know about something but can't recall the specific answer. I normally try and turn that scenario into a quick discussion where I hopefully let the interviewer display their vast knowledge and insight. Assuming the question is relevant I am always geniuenly interested in finding out about something I don't know. Seeing that I am not being paid to take the interview I feel it is only fair that I get something out of it.

45 points by dncrane 13y ago

Corollary: "There's no such thing as a stupid question", because you're not afraid to be wrong. All creative and innovative people are like this, without exception, IMO. As Richard Feynman said in the title of an autobiography: "What do you care what other people think?" Everybody wins when people leave their egos behind.

ape4 13y ago
Sure. But its not so hard to come up with a question that sounds pretty stupid. What is Monday? Can I eat your shoe for lunch? Why do you have to interview me before hiring me?

croddin 13y ago

When reading the title, I assumed this was talking about job interviews, a much different type of interview. How do these ideas apply to job interviews? Is it a good idea for an interviewer in a job interview to encourage interviewees to say I don't know, and should you be more ok with saying I don't know and then elaborating in a job interview?

Peroni 13y ago
Purely anecdotal of course but I always set the scene for interviewee's that "I don't know" is a completely acceptable answer. If the question could be attempted with an educated guess then "I don't know" is perfectly fine as long as an educated guess is then attempted. If you only get "I don't know" with no attempt at a guess or explanation for why they don't know then you may have issues.
bennyg 13y ago
On the job interview point: I'm always as truthful as possible in a job interview - especially technical ones. My background is not in tech, however I'm an autodidact with programming and whenever technical questions are asked that I may not know, I'll say it. And then I'll say how I think it should/would be done, or the steps I'd take to figure out the answer and solve the problem. Not sugarcoating some of my technical inability has faired me well so far. And then after I hear the question, think about the steps to do it, and/or get coached by the interviewee - I know the answer, and am ready to use it in my technical/creative arsenal. I feel like it's a win-win to not bullshit about something I don't know.
TheChineseGirl 13y ago
Without encouragement, candidates answer with "I don't know" during a job interview easily stands out. If we are interviewing for a position, aren't we searching for just ONE outstanding specific individual? Should we really explicitly encourage?
elq 13y ago
A perfectly acceptable answer to me is "I don't know, but this is what I would do to find out", go on to describe what books you'd refer to, what google searches, etc.
jfarmer 13y ago
Those are two separate questions, IMO. In the context of a job interview, however, I believe giving the interviewee permission to say "I don't know" makes the interview go better because they'll spend more time focusing on the interview and less time trying to figure out "what the interviewer wants." I usually give a little preamble telling them to feel comfortable blurting out nonsense. I'd rather have them blurt out nonsense than sit there for 30 seconds in silence because they're afraid to say something stupid. You can see the interviewee relax almost immediately.
bcbrown 13y ago
My view is that if an interviewer dings me for saying "I don't know", that's a sign I wouldn't want to work there anyway.

agumonkey 13y ago

lucidity as the fastest path to clarity ?

samograd 13y ago

I'll usually ask at least one question in an interview to see if the candidate will answer with "I don't know" or ramble on incoherently making something up. Guess which I prefer.

asgard1024 13y ago

I have long suspected this to be true. That's my argument against mandatory voting, because if people can choose not to vote, they can express they don't know and hopefully leave the result on people who feel they do know. However, it's a bit unclear how this ties together with Dunning-Kruger effect (which is also used as an argument against democracy). The D-K effect would suggest that if you allow people to express doubt, the experts will doubt more, and overall performance will decrease. It would be interesting if someone did a psychology experiment on that. (I am actually in favor of doing these kinds of demonstrations in high school, because it's in my opinion important for people to understand how democratic voting works, for example the fact that voting won't get you a simple average of the results - which is usually used as an argument against democracy.)

mistercow 13y ago
I'd actually be pretty surprised if the Dunning-Kruger effect applied significantly to politics. There are so many tribal pressures involved that I think that the phenomena involved with Dunning-Kruger would be overwhelmed.
- asgard1024 13y ago
  In representative democracy, probably yes. But D-K effect is commonly used as an argument against (semi)direct democracy.
forktheif 13y ago
You can have mandatory voting with a "none of the above" option on the ballot.
- asgard1024 13y ago
  And the point of such a system is? To spend more money and time of everyone?
  - anonymfus 13y ago

judk 13y ago

This is obvious. See also the child abuse panic of the 80s, where kids were pressured to make up stories of abuse. And the general fact that eyewitness testimony is unreliable.

beat 13y ago

One of my best technical interviews was from a lead engineer who simply asked more and more difficult questions until he found your limit (believe me, he knew more than anyone he would interview). His interest wasn't in what you knew so much as how you reacted when you hit the limits of your knowledge. There's a lot of value in that approach. I don't use it myself when interviewing people, but I do try to accomplish the same end.

mistercow 13y ago
>believe me, he knew more than anyone he would interview He wouldn't really need to. By asking questions about very domain-specific knowledge, he could still achieve the same effect with anyone who didn't have a freakishly similar set of expertise.

pron 13y ago

It's a nice idea, and it makes sense, but the sample size – 26 people per group, multiplied by 2 for two experiments – is too small to reach any definite conclusions.

gcb0 13y ago

One thing i always ask on that chat before the trivia questions, is what tech news sites the candidate read. And where he usually goes to were he wants to know the correct argument order of a function he does not use too often. That usually gives me a better grasp of how he will solve a real world problem than the trivia questions. But when hiring for teams that need to get specific work done fast, i do not underestimate the value of the trivia questions as it will show me the ramp up time to get that work done. Of course i apply that on top of the other means to get an idea if the candidate is a good fit overall. There is no silver bullet for hiring.

WhitneyLand 13y ago

How about just ditching contrived interview questions because companies have no good science or data to establish the correlation with hiring the best people? It seems past performance would be more reliable. What have you created and what was your role in it? Can you articulate your passion for the problems we are solving? Can you show that your colleagues respected your work and benefited from collaborating with you?

lmkg 13y ago

I think part of this conditioning from taking tests in school. For twelve or more years of your life, in any situation where you are asked to demonstrate your knowledge, a null response is treated in the same negative fashion as an incorrect response, and getting outside help is disallowed. No wonder we try and guess, we've literally been trained for years that it's the best approach! Compounding that, an interview is the one situation in "the real world" that most resembles a testing situation from grade school, which will bring back all those bad habits.

tootie 13y ago
Actually, the SAT (American college entrance exam) penalizes wrong answers more than non-answers. A non-answer gets you no points, but a wrong answer gets points subtracted.
- j_baker 13y ago
  The SAT is the exception rather than the rule. Most standardized and non-standardized tests reward you for guessing.
  - mathattack 13y ago
    The GMAT as well. It's interesting when you work with little kids - you have to encourage them to guess. They're more likely to admit what they don't know.
- iyulaev 13y ago
  Barely. IIRC it's 0 points if blank, -0.25 if wrong, so if you eliminate 1 choice it's worth it to guess. I'd like to see it be -5 points if wrong, since this is more reflective of the real world. When I entered the industry as an electrical engineer this was a difficult transition for me. In school, you're incentivized to guess and BS. But in the real world, the penalty for being wrong is HUGE.
  - wwweston 13y ago

patrickyeon 13y ago

Some commenters, and possibly the author, want this advice to extend to job interviews, but the reason an interviewer is pushing you to answer a question isn't that they want you to be right or wrong. A good interviewer wants to get a look into how you think and how you approach a problem. An even better interviewer will make that clear, by telling you "there's no trick here", "I don't know the answer myself, let's see what we can figure out", or "there's no one right answer, I just want to know what you can see here". A trick I've employed a fair bit lately to get a reserved interviewee to start working with a question is "What is the worst solution we could provide to attack this problem?" I'll even possibly go as far as offering my own horrible solution, and asking them where we fo to improve on this. And I do mean worst. I haven't met a candidate yet who can't at least throw out ideas on how to improve my horrible solutions, and at that point the ball is rolling.

RokStdy 13y ago
Asking somebody for the worst possible solution seems like a really clever ice-breaker. I'm stealing that technique! Thanks patrickyeon! I interviewed some folks a while ago for a simple desktop support position and I had a lot of trouble getting anything out of the candidates. It's possible that they were all just terrible, but I'm guessing it was more probably my approach. I gave them a very simple test[1], and I was met with sort of deer-in-headlights stares. Hence forth I think I'll start with a problem and an example horrible solution. [1]Presented the applicant with a screenshot from a desktop computer that couldn't connect to the internet. The screenshot showed several possible indicators (impossible network settings, cable unplugged, etc...).
j_baker 13y ago
> A good interviewer wants to get a look into how you think and how you approach a problem. I consider that a red flag. In this case, the interviewer is almost certainly less concerned with whether or not you can do the job than they are concerned with whether you think like them. Though I do like your suggestion to try offering a terrible solution to get the conversation started. Usually it's breaking the ice that's difficult.
- 13y ago

m0th87 13y ago

A good interview question should give space for exploration - it shouldn't involve esoteric questions or things that are easily google-able. Good interview questions will give you insight into the candidate's thought process, not what they've memorized beyond fundamentals. So the results of this study seem irrelevant.

Spoom 13y ago

I've found that the ease with which someone answers "I don't know" has a positive correlation with their wisdom in general. (Of course, if they answer "I don't know" to everything, they probably just don't care, and that's a separate problem.)

TheChineseGirl 13y ago
I second this notion.
yeukhon 13y ago
I don't necessarily disagree with you. I think it is true that having too many I don't know means the person is probably not familiar with the area he's facing. The I don't care part is probably a bit rare. You probably have to look at body language (i.e. facial expression, for example).

ufmace 13y ago

Being comfortable with saying "I don't know" a lot is something I had to kind of train myself to do. It's part of realizing that this is a truly enormous field that we're in, and nobody could possibly know more than a fraction of it. Anyone who doesn't say it much is either spending all of their time in a subset of the field that they know very well, or is throwing around bullshit, either from fear or bravado.

mathattack 13y ago

The answers may get better, but don't you want to find out which candidates are more likely to bluff when they're wrong? The point of the interview isn't for candidates to give you answers that are as correct as possible. If this were the case, you'd mail them a list of questions, and ask them to get back to you in a week. One goal of interviews is to find out how people behave when they don't know. This is why I try to take people to the end of their knowledge, no matter where that end is.

yeukhon 13y ago

Nice study. I think it's quite intuitive. When you are forced to spell you are force to say something. So by saying "I don't know" it is natural to continue with "I don't know but I think" and then it continues with more words. Whereas to tell someone in advance "don't you dare to tell me I don't know" the person will just hold on to the thoughts until he or she has to spell that thought out (because he or she has to say something to get out of the misery).

adamconroy 13y ago

In an interview i have never had a problem saying that I don't know something or that I do know about something but can't recall the specific answer. I normally try and turn that scenario into a quick discussion where I hopefully let the interviewer display their vast knowledge and insight. Assuming the question is relevant I am always geniuenly interested in finding out about something I don't know. Seeing that I am not being paid to take the interview I feel it is only fair that I get something out of it.