4 ms·

I actually interpret their data quite differently. There is a statistically robust and positive correlation between perceived and actual ratings: - 44% of the

by shoyer 11y ago

I actually interpret their data quite differently. There is a statistically robust and positive correlation between perceived and actual ratings: - 44% of the time, the actual and perceived ranking was identical. - 91% of the time, the difference between the rankings was at most one star. - Only in 9% of cases was there was a strong disagreement about how well the interview went (2 or more stars out of four). - There is also a bias towards engineers thinking they did slightly worse than they actually did. Even in an alternative universe where the same interview was repeated, you would not expect rankings by the same individual to be perfect. A more meaningful benchmark (though probably unattainable) would be interviewer/interviewee consistency relative to interviewer self-consistency.

lordnacho 11y ago
That's what I thought as well. I'm not sure it's a great idea to fit a straight line and then look at the R2. After all there's only three other values it can be, rather than a normal distribution.
SEMW 11y ago
Agreed. I think it's arguable that, when the question you're asking is 'how well do X and Y two things correlate with each other', the statistic you should be giving is r (which here is 0.49), not r^2. r^2 is the proportion of the variance of Y that's explained by X -- not that that isn't an interesting thing, but ITSM the correlation coefficient is more to what people expect a measure of relationship to be. (IANAStatistician)
andimus 11y ago
The problem with that interpretation is that it ignores how much something deviates from the expected random. If everything was completely random: - 25% of the time, the actual and perceived ranking would be identical. - 62.5% of the time, the difference would be at most one. - Only in 37.5% of cases would there be a strong difference in scores (2 or more stars out of four). Would you say that 44% is a good score on a multiple choice test? Obviously candidates did better than random guessing, but not by enough to call it a strong correlation. Determining the strength of a correlation is what R-squared is for, which makes it the right choice for this analysis.
- mbizzle88 11y ago
  But the problem with R-squared in this case is that a linear model isn't appropriate for ordinal data. These rankings are ordered categories, not numbers.
  - andimus 11y ago
    The comment I was replying to was not about ordinal data and I stand by it in its context. But on your point, the Jury's still out on what to do with Likert-type data like this. Purists do say that it's ordinal and therefore you can't do any numeric analysis on it, but practically speaking you can learn a lot from it by treating it as interval data. In my opinion, statistical analysis like this is ALWAYS two parts speculation and one part science (if you're lucky). It's designed to give insight without declaring fact so you're allowed to bend the rules a little if it's in the spirit of the math. My "gut" feeling agrees with the original post. If we treat the data as ordinal data, then we can only look at the accuracy, and 44% is, if anything, more compelling. Cards on the table: I was involved in analyzing the data in the first place and have a vested interest in things-- but I am saying what I believe without any intentional bias.
mbizzle88 11y ago
I think the problem with the author's analysis is that she treated the rankings as numerical values when they are in fact only ordinal.

by shoyer 11y ago

lordnacho 11y ago

That's what I thought as well. I'm not sure it's a great idea to fit a straight line and then look at the R2. After all there's only three other values it can be, rather than a normal distribution.

SEMW 11y ago

Agreed. I think it's arguable that, when the question you're asking is 'how well do X and Y two things correlate with each other', the statistic you should be giving is r (which here is 0.49), not r^2. r^2 is the proportion of the variance of Y that's explained by X -- not that that isn't an interesting thing, but ITSM the correlation coefficient is more to what people expect a measure of relationship to be. (IANAStatistician)

andimus 11y ago

The problem with that interpretation is that it ignores how much something deviates from the expected random. If everything was completely random: - 25% of the time, the actual and perceived ranking would be identical. - 62.5% of the time, the difference would be at most one. - Only in 37.5% of cases would there be a strong difference in scores (2 or more stars out of four). Would you say that 44% is a good score on a multiple choice test? Obviously candidates did better than random guessing, but not by enough to call it a strong correlation. Determining the strength of a correlation is what R-squared is for, which makes it the right choice for this analysis.

mbizzle88 11y ago
But the problem with R-squared in this case is that a linear model isn't appropriate for ordinal data. These rankings are ordered categories, not numbers.
- andimus 11y ago
  The comment I was replying to was not about ordinal data and I stand by it in its context. But on your point, the Jury's still out on what to do with Likert-type data like this. Purists do say that it's ordinal and therefore you can't do any numeric analysis on it, but practically speaking you can learn a lot from it by treating it as interval data. In my opinion, statistical analysis like this is ALWAYS two parts speculation and one part science (if you're lucky). It's designed to give insight without declaring fact so you're allowed to bend the rules a little if it's in the spirit of the math. My "gut" feeling agrees with the original post. If we treat the data as ordinal data, then we can only look at the accuracy, and 44% is, if anything, more compelling. Cards on the table: I was involved in analyzing the data in the first place and have a vested interest in things-- but I am saying what I believe without any intentional bias.

mbizzle88 11y ago

I think the problem with the author's analysis is that she treated the rankings as numerical values when they are in fact only ordinal.