3 ms·
Yup. Assuming the sample sizes are statistically significant, the original paper clearly shows: - On average, people estimate their ability around the 65th per
by crazygringo 3y ago
Yup. Assuming the sample sizes are statistically significant, the original paper clearly shows:
- On average, people estimate their ability around the 65th percentile (actual results) rather than the 50th (simulated random results) -- a significant difference
- That people's self-estimation increases with their actual ability, but only by a surprisingly small degree (actual results show a slight upwards trend, simulated random results are flat) -- another significant difference
The author's entire discussion of "autocorrelation" is a red herring that has nothing to do with anything. Their randomly-generated results do not match what the original paper shows.
None of this really sheds much light on to what degree the results can be or have been robustly replicated, of course. But there's nothing inherently problematic whatsoever about the way it's visualized. (It would be nice to see bars for variance, though.)
- ketozhang 3y agoThe autocorrelation is important to show that it's transformation to D-K plot will always give you the D-K affect for independent variables. However, the focus on autocorrelation is not very illuminating. We can explain the behaviors found quite easily: - If everyone's self-assessment score are (uniformally) random guesses, then the average self-assessment score for any quantile is 50%. Then of course those of lower quantile (less skilled) are overestimating. - If self-assessment score vs actual score are dependent proportionally, then the average of each quantile is always at least it's quantile value. This is the D-K effect, which is weaker as the correlation grows. -The opposite is true for disproportional relation. So, the D-K plot is extremely sensitive to correlations and can easily over-exaggerate the weakest of correlations.
- cortesoft 3y ago> That people's self-estimation increases with their actual ability, but only by a surprisingly small degree (actual results show a slight upwards trend, simulated random results are flat) -- another significant difference If everyone thinks they are slightly above average, isn't this inevitable? If everyone thinks they are slightly above average, people who are slightly above average are going to be the most accurate at predicting where they land?
- zuminator 3y agoEven if "people tend to slightly overrate their own ability," was the only takeaway, it would still refute the author's conclusion that DK has nothing to do with human psychology.
- IanCal 3y agoYes but then you'd see a flat line for people's estimates, which wasn't the result.
- cycomanic 3y agoHave you not just summarized the Dunning-Kruger effect in other words? That essentially follows from everyone assume they are slightly above average. That's also the crux of the refutation and why the whole autocorrelation is a red hering, even if we all would just self assess completely randomly, that actually confirms the Dunning-Kruger effect is real (because if we self assess randomly worse performance are more likely to overestimate). We could argue that this is not surprising, but the "surprising" bit is that the curves show that better performers are actually more skilled at assessing their performance, which incidentally was also confirmed by the followup studies.
- quetzthecoatl 3y agoIs it though? Everyone overestimating their ability a bit isn't DK effect. It's when people with less knowledge and ability vastly over estimate their ability (because they don't know how little they know - while others do), and the opposite for those who are truly more able and knowledgeable (again because they understand how vast the topic is and though they know more and are capable more than the average person, they also understand how little they truly know compared to what they don't know)
- what-the-grump 3y agoI'll give it a stab. There are those that don't know, and don't know that they don't know. They evaluate themselves the highest. There are those that know, and don't know that they don't know. They evaluate themselves a bit better than those before. There are those that know, and know that they don't know. They evaluate themselves worst than those before them. This is the d-k valley, imposter syndrome, confidence issues. There are those that know, and know that they know. They are much better at evaluating themselves than those before them. They have experience to know what they know, and what they dont know, and they still continue to underrate themselves vs the first bunch, but they are more accurate and closer to the truth. Anyway thats how I always understood d-k.
- somenameforme 3y ago> "On average, people estimate their ability around the 65th percentile (actual results) rather than the 50th (simulated random results) -- a significant difference" This is a different issue than D-K. The D-K hypothesis is that self assessment and actual performance are less correlated for weaker than higher performing individuals. People think they're better than average is a different (and much less controversial) bias. --- [DK-Effect] : I totally know I scored at least a 30% on that test, and that's certainly way better than average (it's not). [Actually scored 10%] [No DK-Effect] : I totally know I scored at least a 30% on that test, and that's certainly way better than average (it's not). [Actually scored 30%] ---
- kstenerud 3y ago> The D-K hypothesis is that self assessment and actual performance are less correlated for weaker than higher performing individuals. Isn't that what the graph shows? The bottom quartile group is guessing almost 50 percentile points higher than their actual performance, whereas the top quartile is at most 15 points off. They're all guessing somewhere between the 60th and 75th percentiles (i.e. "I'm a bit better than average") - with some upwards trend since the high performers seem to at least know they have some skill, although not very accurately. It's just that for the poor performers, a guess of the 60th percentile wayyy off the mark.
- somenameforme 3y agoEDIT: Something important for the rest of this post. In case it's not clear, the graph is showing your percentile ranking within the group - not your actual score. Nope, because there's an interesting statistical trick in play. Imagine you take 100 highly skilled physicists and give them some lengthy series of otherwise relatively basic physics questions. Everybody is going to rate their predicted performance as high. But some people will miss some questions simply due to silly mistakes or whatever. And those people would end up on the bottom 10% of this group, even if the difference between #1 and #100 was e.g. 0.5 points. Graph it as D-K did, and you'd show a huge Dunning Kruger effect, even when there is obviously nothing of the sort. In fact the fewer differences in ability within a group, and the greater the relative ease of a task, the bigger the Dunning-Kruger effect you'd show. Because everybody will rate themselves relatively high, but you will always have a bottom 10%, even if they are practically identical to the top 10%. You can see this most clearly in the original paper. They carried out 4 experiments. The one that was most objective and least subject to confounding variables was #2, where they asked people a series of LSAT based logic questions, and assessed their predicted vs actual results. And there was very little difference. Quoting the paper, "Participants did not, however, overestimate how many questions they answered correctly, M = 13.3 (perceived) vs. 12.9 (actual), t < 1. As in Study 1, perceptions of ability were positively related to actual ability, although in this case, not to a significant degree." Yet look at the graph for it, and again it shows some seemingly large D-K effect. And there's even more issues with D-K, and especially experiment #1 (which is the one with the prettiest graph by far), but that's outside the scope of this post. I'm happy to get into it, if you are though. I find this all just kind of shocking and exceptionally interesting! I've referenced the D-K effect countless times in the past, never again after today! [1] - https://sci-hub.se/10.1037/0022-3514.77.6.1121 https://sci-hub.se/10.1037/0022-3514.77.6.1121
- singingfish 3y ago> assuming the sample sizes are statistically significant Nitpick: should read "assuming the sample sizes provide sufficient statistical power"