3 ms·

I was surprised by the figure from the original article, imho that's the strongest rebuttal: perceived ability grows strictly mononotonically with actual abilit

by t_mann 3y ago

I was surprised by the figure from the original article, imho that's the strongest rebuttal: perceived ability grows strictly mononotonically with actual ability, no sign of the famous non-monotonic U-curve. Yeah, the slope is less than one, and it grows a bit faster from the second to the third quartile than from the first to the second, but none of that changes the fact that people tend to slot themselves correctly. The chart is interesting in that it confirms that everyone perceives themselves to be slightly above average in terms of ability, which of course can't be true in practice. But what it also shows is that when they think they'll be below or above that (false) baseline, they're actually correct about it. So pretty much the exact opposite of what the Dunning-Kruger effect claims.

jampekka 3y ago
The slope will be less than one if there's e.g. any random guessing in the test even if the self-assesment is perfect (apart from whether they know if their guess is right or wrong of course) [1]. I think this is the effect that the post is dancing around, but doesn't seem to really understand (and how "autocorrelation" and indepence are discussed is very nonstandard to be charitable). [1] https://en.m.wikipedia.org/wiki/Regression_dilution https://en.m.wikipedia.org/wiki/Regression_dilution
- t_mann 3y ago
  I agree, the statistical analysis in the original post makes me very uneasy. I think it could be a case where the conclusion is correct, even though argument isn't necessarily. And yes, the fact that the slope is less than one is fairly uninteresting. The real problem here is that the Dunning-Kruger effect, as it's classically stated, claims that if you asked four people to rank themselves in terms of ability, the result would be 1-3-2-4, ie the people who know a little would put themselves above the people who know a lot but aren't quite experts. The problem is the data shows that they'd actually rank themselves correctly 1-2-3-4. But such a boring finding probably wouldn't have made the authors quite as famous, which might be why they tried bit of data mangling, and they found this really cool story that everyone would secretly love to be true. Which is a shame, because I think the fact that the mean of perceived ability is too high (and the variance too low) is really interesting too, and perfectly supported by the raw data.
  - jampekka 3y ago
    Yes. The methodology in the original D&K is quite shoddy, and vulnerable to e.g. good old regression to the mean, and the interpretations are too strong. This is sadly very common in psychology (and many other fields I'd guess) and even researchers don't care so much if the story is juicy enough. The pop version of the DK effect seems to be something like a 4-3-2-1 ranking, which is obviously not supported by the data.
  - tempestn 3y ago
    But they wouldn't. They'd rank themselves something like 1,2,2,3. We're not dealing with a population collaborating to all rank themselves in order, but rather each person individually estimating where their abilities lie in the population. The point is that if you ask someone in the, say, 5th percentile of ability what their ability is compared to the population, they might say 25th percentile. Ask someone at the 25th,and they might say 40th. At the 40th they could say 55th. And at the 90th, maybe they'll say 80th. So yes, if you order their guesses, they will be in roughly the correct order. But, crucially, that doesn't mean that they are ranking themselves correctly!
tempestn 3y ago
> The chart is interesting in that it confirms that everyone perceives themselves to be slightly above average in terms of ability, which of course can't be true in practice. No, everyone biases their self-assessments toward a point slightly above the mean. That's not the same as saying everyone thinks they're slightly above average, nor that people's self-assessments have no predictive power whatsoever. The lowest performers still think they're below average, just not as much as they should. The highest performers still think they're considerably above average. But they all have a bias toward (slightly above) the middle. So yes, people are generally correct in the direction that they deviate from that median self-assessment, but that just shows that people's self-assessments aren't completely without basis. Which D-K certainly didn't claim.
- t_mann 3y ago
  D-K claim a non-monotonic relationship, which simply isn't supported by that data, as you yourself point out: people rank themselves correctly (ordinally). I didn't mean to say that all self-assessments are the same, if that was the misunderstanding. My point is that the self-assessments indeed are meaningful, even more so than D-K claim.
  - RevEng 3y ago
    Check the original paper by D-K. Fix only focused on the first plot which has a monotonically increasing trend. The later plots show varying degrees of nonmonotonicity, though sadly they don't include error bars to indicate how statistically significant the differences between groups is.
- zeroonetwothree 3y ago
  But we don’t know their true ability, only the results on one test. It could be they accurately predicted their ability but because of random chance they did better/worse than their guess. Then you would get the exact data that is observed.
  - lokar 3y ago
    I thought they were estimating their performance on the test relative to others. There was no “real world” element.

by t_mann 3y ago

jampekka 3y ago

The slope will be less than one if there's e.g. any random guessing in the test even if the self-assesment is perfect (apart from whether they know if their guess is right or wrong of course) [1]. I think this is the effect that the post is dancing around, but doesn't seem to really understand (and how "autocorrelation" and indepence are discussed is very nonstandard to be charitable). [1] https://en.m.wikipedia.org/wiki/Regression_dilution https://en.m.wikipedia.org/wiki/Regression_dilution

t_mann 3y ago
I agree, the statistical analysis in the original post makes me very uneasy. I think it could be a case where the conclusion is correct, even though argument isn't necessarily. And yes, the fact that the slope is less than one is fairly uninteresting. The real problem here is that the Dunning-Kruger effect, as it's classically stated, claims that if you asked four people to rank themselves in terms of ability, the result would be 1-3-2-4, ie the people who know a little would put themselves above the people who know a lot but aren't quite experts. The problem is the data shows that they'd actually rank themselves correctly 1-2-3-4. But such a boring finding probably wouldn't have made the authors quite as famous, which might be why they tried bit of data mangling, and they found this really cool story that everyone would secretly love to be true. Which is a shame, because I think the fact that the mean of perceived ability is too high (and the variance too low) is really interesting too, and perfectly supported by the raw data.
- jampekka 3y ago
  Yes. The methodology in the original D&K is quite shoddy, and vulnerable to e.g. good old regression to the mean, and the interpretations are too strong. This is sadly very common in psychology (and many other fields I'd guess) and even researchers don't care so much if the story is juicy enough. The pop version of the DK effect seems to be something like a 4-3-2-1 ranking, which is obviously not supported by the data.
- tempestn 3y ago
  But they wouldn't. They'd rank themselves something like 1,2,2,3. We're not dealing with a population collaborating to all rank themselves in order, but rather each person individually estimating where their abilities lie in the population. The point is that if you ask someone in the, say, 5th percentile of ability what their ability is compared to the population, they might say 25th percentile. Ask someone at the 25th,and they might say 40th. At the 40th they could say 55th. And at the 90th, maybe they'll say 80th. So yes, if you order their guesses, they will be in roughly the correct order. But, crucially, that doesn't mean that they are ranking themselves correctly!

tempestn 3y ago

> The chart is interesting in that it confirms that everyone perceives themselves to be slightly above average in terms of ability, which of course can't be true in practice. No, everyone biases their self-assessments toward a point slightly above the mean. That's not the same as saying everyone thinks they're slightly above average, nor that people's self-assessments have no predictive power whatsoever. The lowest performers still think they're below average, just not as much as they should. The highest performers still think they're considerably above average. But they all have a bias toward (slightly above) the middle. So yes, people are generally correct in the direction that they deviate from that median self-assessment, but that just shows that people's self-assessments aren't completely without basis. Which D-K certainly didn't claim.

t_mann 3y ago
D-K claim a non-monotonic relationship, which simply isn't supported by that data, as you yourself point out: people rank themselves correctly (ordinally). I didn't mean to say that all self-assessments are the same, if that was the misunderstanding. My point is that the self-assessments indeed are meaningful, even more so than D-K claim.
- RevEng 3y ago
  Check the original paper by D-K. Fix only focused on the first plot which has a monotonically increasing trend. The later plots show varying degrees of nonmonotonicity, though sadly they don't include error bars to indicate how statistically significant the differences between groups is.
zeroonetwothree 3y ago
But we don’t know their true ability, only the results on one test. It could be they accurately predicted their ability but because of random chance they did better/worse than their guess. Then you would get the exact data that is observed.
- lokar 3y ago
  I thought they were estimating their performance on the test relative to others. There was no “real world” element.