8 ms·

I find it quite problematic that researchers get to talk about their own research and present it as facts without anyone taking a critical look. Over time I’ve

by schmit 9y ago

I find it quite problematic that researchers get to talk about their own research and present it as facts without anyone taking a critical look. Over time I’ve become more skeptical about this kind of psychology research (as more studies fail to replicate) and, as is often the case, here the sample size is quite small (76 students, split across 3 groups), predicting something relatively noisy as GPA. It is unclear to me that one would be able to detect reasonable effects. Furthermore, some claims that make it into the piece are at odds with the data: > Strikingly, not one interviewer reported noticing that he or she was conducting a random interview. More striking still, the students who conducted random interviews rated the degree to which they “got to know” the interviewee slightly higher on average than those who conducted honest interviews. While Table 3 in the paper shows that there is no statistical evidence for this claim as the effects are swamped by the variance. My point is not that this article is wrong; verifying/debunking the claims would take much more time than my quick glance. But that ought to be the responsibility of the newspaper, and not individual readers. Politicians don’t get to write about the successes of their own policies. While there is a difference between researchers and politicians, I think we ought to be a bit more critical.

trishume 9y ago
The claim you mention about getting to know the person better in random interviews as well as "People can’t help seeing signals, even in noise." is misleading and unsurprising. People ask questions in interviews that they want to know the answer to and that could go either way. All questions have equivalent expected surprise, either both answers are unsurprising, or one answer is surprising, but you think you already know the answer is the other one. If interviewers were asking questions like "is 2+2=4" they would have detected random interviewers way easier, but they wouldn't be trying very hard to get to know the person. As for getting to know the person, the more surprising someone's answers are, if you believe they are telling the truth, the more distinguishes them from "average person who gave answers I expected", so you say you "got to know" them. This is unsurprising. This isn't to defend unstructured interviews, other studies for a long time have shown them to be worse than structured interviews and test scores. If I had to guess the only reason the research in the article got published as novel was the random interview part. Edit: Here's a table from a meta-analysis of lots of studies on correlation between different factors and job performance: http://imgur.com/a/YRFTh http://imgur.com/a/YRFTh. Basically unstructured interviews aren't as good as structured ones, but they are better than nothing. Work samples are the best, structured interviews and IQ tests tie for second. Note that this meta-analysis is combining a bunch of different fields to yield general observations, a specific field may have different results. But in expectation for a randomly selected field these are fairly solid results, and I don't expect fields vary from them too much.
emodendroket 9y ago
Maybe, but doesn't a claim like "a 30-minute conversation with someone is a strong indicator of job performance" deserve scrutiny of its own?
- dsjoerg 9y ago
  Of course it does. However the article and perhaps the underlying paper may be as useless as the interviews they are criticizing.
- curun1r 9y ago
  Unless you're talking about a very high-level position that either requires substantial leadership traits or very specialized knowledge, interviewing has never been about finding the right person for the job, it's been about finding a right person for the job. The reality is that most jobs can be done successfully by a large number of people. The accuracy of the assessment of the candidate's abilities is, to my mind, a secondary concern in the interview process and yet, just as in this study, is the only part of the interview process studied and critiqued. But what I find far more important in the interview process is involving current team members in the process of selecting new coworkers. It's one way of getting teams to be bought into a feeling of shared purpose and is the first step in establishing a working relationship. If you don't give at least some of the current team a role in the hiring process, teams will feel imposed upon by those hiring and won't be as understanding about flaws in those added to the teams. Focusing on selecting the "right candidate" is really myopic in a situation where there are likely many right candidates. We should, instead, be focusing on not selecting a wrong candidate and fostering the right team dynamic. We already know that strong teams significantly outperform strong individual performers who don't cooperate. Yet hiring still seems focused on optimizing for strong individual contributions. And I've yet to see a study that looks at the flaws of the interview process in building ineffective teams.
  - emodendroket 9y ago
    I think you may be misconstruing the argument here. I agree that any number of people will fit -- but the problem is, what if interviews aren't even selecting "a" right person for the job?
ljw1001 9y ago
Politicians _always_ promote the success of their own policies.
- derefr 9y ago
  I think the parent means, we don't take as objective fact a politician's claims that their policies worked; we interpret it as self-aggrandizement and scrutinize the claims quite heavily. And that—given the push to publish and the fact that null-result studies aren't very publishable—we should likely do the same for research conclusions.
  - ljw1001 9y ago
    I understood, but I think recent evidence in the US suggests that a great many people absolutely take as objective fact what a politician claims. A similar number probably absolutely believe the opposite with little or no evidence. Net, I think citing politicians was perhaps not the best analogy.
    - schmit 9y ago
      Sorry I was not more clear. What I meant was that the NY Times employs journalists and fact checkers and editors to validate stories, say on politics to make sure, to the best of their abilities, that the articles they post are correct. Why is that not the case here, where a professor is allowed to sell his own work? It is as if Obama is the NYT reporter for Obamacare. That people believe politicians blindly is a topic for another day :)
xapata 9y ago
> Politicians don’t get to write about the successes of their own policies Politicians boast about successes constantly. I don't understand what you mean by "get to".
- BigJono 9y ago
  They also run expensive ad campaigns promoting their own policies, at least here in Australia. They don't even have to be successful policies to be able to talk about them...
bnegreve 9y ago
Of course we should be more critical, the methodology here is far from perfect. But right now, people have much faith in interviewing, and this research suggests that this faith may not be justified. As you mentioned, this is not the only research that reaches this conclusion. There is also the work by Daniel Kahneman[1] which I find pretty rigorous and draw the same conclusions about interviewing. So obviously, the title of this article should be "Maybe interviewing is not that useful" instead of "The utter uselessness of job interviews", but besides this I find your comment unjustified. In fact, it's quite the opposite, I believe this type of work contributed to make us more critical by questioning some basic facts about interviewing, that i would have never questioned just a couple of years ago. > Over time I’ve become more skeptical about this kind of psychology research (as more studies fail to replicate) ok, this is interesting, where is it mentioned? [1] Think fast and think slow. There is this short article which mention some of the results and has been discussed on HN a couple of times already. http://www.nytimes.com/2011/10/23/magazine/dont-blink-the-hazards-of-confidence.html http://www.nytimes.com/2011/10/23/magazine/dont-blink-the-ha...
- mooreds 9y ago
  > ok, this is interesting, where is it mentioned https://mobile.nytimes.com/2015/08/28/science/many-social-science-findings-not-as-strong-as-claimed-study-says.html https://mobile.nytimes.com/2015/08/28/science/many-social-sc... among other places
- andrewmutz 9y ago
  It's easy to present data that shows flaws with today's interviewing. It's a lot harder to present a better way of predicting candidate performance in the workplace, along with substantial data that indicates it's better than today's methods. Corporations would love more effective ways to determine effectiveness/performance before hiring. Interviewing is terrible, but that doesn't mean there is a better option.
  - emodendroket 9y ago
    That feels a bit like saying we should just stick to blood letting and leeches because we don't know any more effective way of treating disease. Just because we don't have a superior alternative doesn't actually mean the current method is effective.
howitworks88 9y ago
> I find it quite problematic that researchers get to talk about their own research and present it as facts without anyone taking a critical look. Are you suggesting anything out of the ordinary is going on here? Doesn't one have to present their work as legitimate and wait for feedback? So long as they are open to that feedback, I don't really get this argument? > Over time I’ve become more skeptical about this kind of psychology research (as more studies fail to replicate) and, as is often the case, here the sample size is quite small (76 students, split across 3 groups), predicting something relatively noisy as GPA. It is unclear to me that one would be able to detect reasonable effects. So present that back them as a refutation? Are you waiting for someone else to do it? Why are you debating the reliability here?
hobls 9y ago
It's also worth noting that those students are probably very inexperienced interviewers.
- AstralStorm 9y ago
  Indeed, warrants further study.
valuearb 9y ago
Psychology is a science like astrology is a science. Sample sizes are the least relevant of the flaws with its studies.
jernfrost 9y ago
I don't see the problem. That researchers haven't done a perfect study in every possible way is hardly the issue here, but rather the blind trust in our ability to interview and hire people. This research paper does not come out of the blue. This is something that has been observed by many organizations and people for quite some time, that interviews are broken. In fact there is an awful lot of wrong stuff society keeps doing despite evidence over many year suggesting it is stupid. The stupidity of high CEO salaries e.g. has been proven quite well, yet business keeps a blind faith in it. Getting employees in complicated jobs to perform by rewarding them on extremely narrow metric has also proven itself counterproductive yet the MBA crowd refuse to believe it doesn't work. Business practices often seem to be more like religion than founded on reality. It is a GOOD thing that some researchers are trying to do their bit to correct this flawed picture.
ppidugu 9y ago
You are absolutely right!
kem 9y ago
There are many, many studies showing the same thing. This study is just one example. See, e.g., meta-analyses of your choice: http://psycnet.apa.org/journals/apl/79/4/599/ http://psycnet.apa.org/journals/apl/79/4/599/ They're predictive, but not very. Structured interviews are better, but not really by much. The problem with all of these things is they work, but not very well, and people make too much of any one thing. An interview is a very small slice of behavior, even if it's structured very well. People make too much of them.

by schmit 9y ago

trishume 9y ago

The claim you mention about getting to know the person better in random interviews as well as "People can’t help seeing signals, even in noise." is misleading and unsurprising. People ask questions in interviews that they want to know the answer to and that could go either way. All questions have equivalent expected surprise, either both answers are unsurprising, or one answer is surprising, but you think you already know the answer is the other one. If interviewers were asking questions like "is 2+2=4" they would have detected random interviewers way easier, but they wouldn't be trying very hard to get to know the person. As for getting to know the person, the more surprising someone's answers are, if you believe they are telling the truth, the more distinguishes them from "average person who gave answers I expected", so you say you "got to know" them. This is unsurprising. This isn't to defend unstructured interviews, other studies for a long time have shown them to be worse than structured interviews and test scores. If I had to guess the only reason the research in the article got published as novel was the random interview part. Edit: Here's a table from a meta-analysis of lots of studies on correlation between different factors and job performance: http://imgur.com/a/YRFTh http://imgur.com/a/YRFTh. Basically unstructured interviews aren't as good as structured ones, but they are better than nothing. Work samples are the best, structured interviews and IQ tests tie for second. Note that this meta-analysis is combining a bunch of different fields to yield general observations, a specific field may have different results. But in expectation for a randomly selected field these are fairly solid results, and I don't expect fields vary from them too much.

emodendroket 9y ago

Maybe, but doesn't a claim like "a 30-minute conversation with someone is a strong indicator of job performance" deserve scrutiny of its own?

dsjoerg 9y ago
Of course it does. However the article and perhaps the underlying paper may be as useless as the interviews they are criticizing.
curun1r 9y ago
Unless you're talking about a very high-level position that either requires substantial leadership traits or very specialized knowledge, interviewing has never been about finding the right person for the job, it's been about finding a right person for the job. The reality is that most jobs can be done successfully by a large number of people. The accuracy of the assessment of the candidate's abilities is, to my mind, a secondary concern in the interview process and yet, just as in this study, is the only part of the interview process studied and critiqued. But what I find far more important in the interview process is involving current team members in the process of selecting new coworkers. It's one way of getting teams to be bought into a feeling of shared purpose and is the first step in establishing a working relationship. If you don't give at least some of the current team a role in the hiring process, teams will feel imposed upon by those hiring and won't be as understanding about flaws in those added to the teams. Focusing on selecting the "right candidate" is really myopic in a situation where there are likely many right candidates. We should, instead, be focusing on not selecting a wrong candidate and fostering the right team dynamic. We already know that strong teams significantly outperform strong individual performers who don't cooperate. Yet hiring still seems focused on optimizing for strong individual contributions. And I've yet to see a study that looks at the flaws of the interview process in building ineffective teams.
- emodendroket 9y ago
  I think you may be misconstruing the argument here. I agree that any number of people will fit -- but the problem is, what if interviews aren't even selecting "a" right person for the job?

ljw1001 9y ago

Politicians _always_ promote the success of their own policies.

derefr 9y ago
I think the parent means, we don't take as objective fact a politician's claims that their policies worked; we interpret it as self-aggrandizement and scrutinize the claims quite heavily. And that—given the push to publish and the fact that null-result studies aren't very publishable—we should likely do the same for research conclusions.
- ljw1001 9y ago
  I understood, but I think recent evidence in the US suggests that a great many people absolutely take as objective fact what a politician claims. A similar number probably absolutely believe the opposite with little or no evidence. Net, I think citing politicians was perhaps not the best analogy.
  - schmit 9y ago
    Sorry I was not more clear. What I meant was that the NY Times employs journalists and fact checkers and editors to validate stories, say on politics to make sure, to the best of their abilities, that the articles they post are correct. Why is that not the case here, where a professor is allowed to sell his own work? It is as if Obama is the NYT reporter for Obamacare. That people believe politicians blindly is a topic for another day :)

xapata 9y ago

> Politicians don’t get to write about the successes of their own policies Politicians boast about successes constantly. I don't understand what you mean by "get to".

BigJono 9y ago
They also run expensive ad campaigns promoting their own policies, at least here in Australia. They don't even have to be successful policies to be able to talk about them...

bnegreve 9y ago

Of course we should be more critical, the methodology here is far from perfect. But right now, people have much faith in interviewing, and this research suggests that this faith may not be justified. As you mentioned, this is not the only research that reaches this conclusion. There is also the work by Daniel Kahneman[1] which I find pretty rigorous and draw the same conclusions about interviewing. So obviously, the title of this article should be "Maybe interviewing is not that useful" instead of "The utter uselessness of job interviews", but besides this I find your comment unjustified. In fact, it's quite the opposite, I believe this type of work contributed to make us more critical by questioning some basic facts about interviewing, that i would have never questioned just a couple of years ago. > Over time I’ve become more skeptical about this kind of psychology research (as more studies fail to replicate) ok, this is interesting, where is it mentioned? [1] Think fast and think slow. There is this short article which mention some of the results and has been discussed on HN a couple of times already. http://www.nytimes.com/2011/10/23/magazine/dont-blink-the-hazards-of-confidence.html http://www.nytimes.com/2011/10/23/magazine/dont-blink-the-ha...

mooreds 9y ago
> ok, this is interesting, where is it mentioned https://mobile.nytimes.com/2015/08/28/science/many-social-science-findings-not-as-strong-as-claimed-study-says.html https://mobile.nytimes.com/2015/08/28/science/many-social-sc... among other places
andrewmutz 9y ago
It's easy to present data that shows flaws with today's interviewing. It's a lot harder to present a better way of predicting candidate performance in the workplace, along with substantial data that indicates it's better than today's methods. Corporations would love more effective ways to determine effectiveness/performance before hiring. Interviewing is terrible, but that doesn't mean there is a better option.
- emodendroket 9y ago
  That feels a bit like saying we should just stick to blood letting and leeches because we don't know any more effective way of treating disease. Just because we don't have a superior alternative doesn't actually mean the current method is effective.

howitworks88 9y ago

> I find it quite problematic that researchers get to talk about their own research and present it as facts without anyone taking a critical look. Are you suggesting anything out of the ordinary is going on here? Doesn't one have to present their work as legitimate and wait for feedback? So long as they are open to that feedback, I don't really get this argument? > Over time I’ve become more skeptical about this kind of psychology research (as more studies fail to replicate) and, as is often the case, here the sample size is quite small (76 students, split across 3 groups), predicting something relatively noisy as GPA. It is unclear to me that one would be able to detect reasonable effects. So present that back them as a refutation? Are you waiting for someone else to do it? Why are you debating the reliability here?

hobls 9y ago

It's also worth noting that those students are probably very inexperienced interviewers.

AstralStorm 9y ago
Indeed, warrants further study.

valuearb 9y ago

Psychology is a science like astrology is a science. Sample sizes are the least relevant of the flaws with its studies.

jernfrost 9y ago

I don't see the problem. That researchers haven't done a perfect study in every possible way is hardly the issue here, but rather the blind trust in our ability to interview and hire people. This research paper does not come out of the blue. This is something that has been observed by many organizations and people for quite some time, that interviews are broken. In fact there is an awful lot of wrong stuff society keeps doing despite evidence over many year suggesting it is stupid. The stupidity of high CEO salaries e.g. has been proven quite well, yet business keeps a blind faith in it. Getting employees in complicated jobs to perform by rewarding them on extremely narrow metric has also proven itself counterproductive yet the MBA crowd refuse to believe it doesn't work. Business practices often seem to be more like religion than founded on reality. It is a GOOD thing that some researchers are trying to do their bit to correct this flawed picture.

ppidugu 9y ago

You are absolutely right!

kem 9y ago

There are many, many studies showing the same thing. This study is just one example. See, e.g., meta-analyses of your choice: http://psycnet.apa.org/journals/apl/79/4/599/ http://psycnet.apa.org/journals/apl/79/4/599/ They're predictive, but not very. Structured interviews are better, but not really by much. The problem with all of these things is they work, but not very well, and people make too much of any one thing. An interview is a very small slice of behavior, even if it's structured very well. People make too much of them.