3 ms·

I found two very interesting things in the original D-K paper [1] that challenge your otherwise reasonable point. The first is that the graph everybody associat

I found two very interesting things in the original D-K paper [1] that challenge your otherwise reasonable point. The first is that the graph everybody associates with D-K, the one showing the beautifully perfect linear result, is one of 4. The other 3 graphs are far messier, and indeed the paper discusses the fact that the correlations tend to be weaker and in some cases nonexistent. The second thing is that that beautiful perfectly linear graph everybody references, was measuring 'humor'!!! Humor is going to be something that's all but guaranteed to create near complete noise between self evaluation and 'expert' (professional comedians in this case) evaluation. And if everybody is essentially randomly guessing on their performance, then it will always show an extremely strong D-K effect with the top performers underestimating themselves, and the bottom performers overestimating themselves. The experiment that most simply and directly measured 'intelligence', without complicating matters in a potentially confounding fashion, is #2. It was based on logic problems from the LSAT. And the resultant graph is just all over the place. Quoting the paper's evaluation of this study: --- "Participants did not, however, overestimate how many questions they answered correctly, M = 13.3 (perceived) vs. 12.9 (actual), t < 1. As in Study 1, perceptions of ability were positively related to actual ability, although in this case, not to a significant degree." --- This is really looking like another Zimbardo. [1] - https://sci-hub.se/10.1037/0022-3514.77.6.1121 https://sci-hub.se/10.1037/0022-3514.77.6.1121

mike_hearn 3y ago
Yes, D-K is another one of those "classic" psychology studies that everyone knows about but is actually rubbish and shouldn't be cited for anything. You're not the first to notice this, I pointed it out on HN last year: https://news.ycombinator.com/item?id=31119836 https://news.ycombinator.com/item?id=31119836 At some point I should write up a proper blog post on the D-K paper in the hope that it eventually surfaces in search results, because it's past time for this paper to be put to bed. The problems you cite aren't even the full set. The whole thing was (of course) a study on a handful of psych undergrads, their selection method for expert comedians has circular logic in it and it all goes downhill from there.
tempestn 3y ago
But again isn't the fact that, "perceptions of ability were positively related to actual ability, although in this case, not to a significant degree" an interesting result? Not the fact that they were related, but the fact that they mostly weren't! That does demonstrate the core result as I understand it, that people are little better than random at evaluating their own performance, which was a surprising finding.
- somenameforme 3y ago
  Nope, because I think D-K played a neat little trick. Whether it was intentional or not is another topic. They were using a largely homogenous group of people - Cornell undergrads taking psychology classes, and querying them on things where all performance would fall close to a similar mean. Imagine you take 100 literal clones of somebody and query them on something, and then ask them to predict their performance. Assuming these clones are smart, they'd all estimate their performance as being at 50%, which is what would be expected. But due to natural variance, not everybody will score identically (in the same way even identical twins do not perform identically). And so you'd end up seeing some huge D-K effect with literal clones! Those at the bottom would be greatly overestimating their performance, while those at the top would be greatly underestimating it. Now step away from clones into the regular world of students, where everybody is going to think they're a bit better than average, and you get people predicting a score of about 60%. Now suddenly you see the same thing, except the lower performers would be overestimating their performance by a greater degree than the top performers were underestimating theirs. To truly measure D-K, you'd need an extremely heterogenous group of people, and you'd also need questions with perceived and real domain expertise. Would a farmer with a 5th grade education evaluate his performance on a differential equations test as above average? Would a professor of diff eq evaluate his performance on a test of optimal growth strategies for buckwheat and corn, as above-average? Of course not, but then you can't make a shocking claim, don't get published, and don't become famous.

mike_hearn 3y ago

Yes, D-K is another one of those "classic" psychology studies that everyone knows about but is actually rubbish and shouldn't be cited for anything. You're not the first to notice this, I pointed it out on HN last year: https://news.ycombinator.com/item?id=31119836 https://news.ycombinator.com/item?id=31119836 At some point I should write up a proper blog post on the D-K paper in the hope that it eventually surfaces in search results, because it's past time for this paper to be put to bed. The problems you cite aren't even the full set. The whole thing was (of course) a study on a handful of psych undergrads, their selection method for expert comedians has circular logic in it and it all goes downhill from there.

tempestn 3y ago

But again isn't the fact that, "perceptions of ability were positively related to actual ability, although in this case, not to a significant degree" an interesting result? Not the fact that they were related, but the fact that they mostly weren't! That does demonstrate the core result as I understand it, that people are little better than random at evaluating their own performance, which was a surprising finding.

somenameforme 3y ago
Nope, because I think D-K played a neat little trick. Whether it was intentional or not is another topic. They were using a largely homogenous group of people - Cornell undergrads taking psychology classes, and querying them on things where all performance would fall close to a similar mean. Imagine you take 100 literal clones of somebody and query them on something, and then ask them to predict their performance. Assuming these clones are smart, they'd all estimate their performance as being at 50%, which is what would be expected. But due to natural variance, not everybody will score identically (in the same way even identical twins do not perform identically). And so you'd end up seeing some huge D-K effect with literal clones! Those at the bottom would be greatly overestimating their performance, while those at the top would be greatly underestimating it. Now step away from clones into the regular world of students, where everybody is going to think they're a bit better than average, and you get people predicting a score of about 60%. Now suddenly you see the same thing, except the lower performers would be overestimating their performance by a greater degree than the top performers were underestimating theirs. To truly measure D-K, you'd need an extremely heterogenous group of people, and you'd also need questions with perceived and real domain expertise. Would a farmer with a 5th grade education evaluate his performance on a differential equations test as above average? Would a professor of diff eq evaluate his performance on a test of optimal growth strategies for buckwheat and corn, as above-average? Of course not, but then you can't make a shocking claim, don't get published, and don't become famous.