3 ms·

The participants in the study didn’t self-evaluate directly, they self-ranked, against the other participants (all Cornell undergrads volunteering for extra cre

by dahart 6y ago

The participants in the study didn’t self-evaluate directly, they self-ranked, against the other participants (all Cornell undergrads volunteering for extra credit). There’s a very important difference there, because self-ranking requires knowledge of other people’s skills. Do note that the DK paper found a positive correlation between confidence and skill. Most people here and elsewhere seem to believe the DK effect shows a negative correlation. Also note that the skills people were self-ranking were all easy. Ability to get a joke was one of the 4 tasks. How would you rank your own ability to get a joke against those around you? The question barely makes sense. I think it’s a big mistake to take any lesson from the DK paper other than sometimes people see what they want to.

maxerickson 6y ago
(I went and skimmed back over it) In most of the studies, they had the participants estimate how they'd do relative to others and also how they would score. Not the first one, but the rest of them. For confidence, the positive correlation doesn't help with the problem where someone with low end confidence estimates similarly to someone with higher confidence but is much more wrong.
- dahart 6y ago
  Agreed, the positive correlation makes the results harder to judge, and specifically it means that one cannot reliably use confidence to predict skill, other than higher confidence is generally associated with higher skill. It's unfortunate how many people cite DK to erroneously justify linking confidence with lack of skill. Based on the paper, the only thing we can say about skill by looking at confidence is that more confident people are statistically likely to be more skilled. You are right, they did mention self-evaluation of score. "Lest one think these results reflect erroneous peer assessment rather then erroneous self-assessment, participants in the bottom quartile also overestimated the number of test items they had gotten right by nearly 50%." But the results and conclusions of all the studies are focusing on the rankings and not the scores. It's difficult to even understand what the absolute range of scores is, and of course we don't see the actual questions they asked, so we don't know what the scores mean. Figures 2 and 3 say to me that zero people in the study knew how well they ranked or what their score was, the survey data is basically flat with noise. This could be explained by questions that are hard to understand, or by participants guessing at the answers. They didn't propose those explanations though, they wax on and on about how people who underperform overestimate their abilities. How do we know the participants even tried to answer correctly? They were all undergrads getting extra credit (which suggests they needed extra credit and were within the range of grades where it might help to get extra credit... consider what that means relative to skill: it's a narrow range of students not getting an F nor getting an A.) And they were getting extra credit just for being there, not for answering correctly. What if they were lazy? What if they were told their answers didn't matter? Why should we assume they didn't answer randomly? Even if they did try, the participants were all Ivy League undergrads who have been told they're among the elite and made it through a highly selective process of admission. Of course they rate & rank themselves highly. 100% of the participants were in a single college psych class, there were no uneducated people, no morons, and nobody who actually had meaningful "deficits in metacognitive skill" relative to the general population. Why is the set of people used in this study not considered an invalidating confounding factor for the results?

by dahart 6y ago

maxerickson 6y ago

(I went and skimmed back over it) In most of the studies, they had the participants estimate how they'd do relative to others and also how they would score. Not the first one, but the rest of them. For confidence, the positive correlation doesn't help with the problem where someone with low end confidence estimates similarly to someone with higher confidence but is much more wrong.

dahart 6y ago
Agreed, the positive correlation makes the results harder to judge, and specifically it means that one cannot reliably use confidence to predict skill, other than higher confidence is generally associated with higher skill. It's unfortunate how many people cite DK to erroneously justify linking confidence with lack of skill. Based on the paper, the only thing we can say about skill by looking at confidence is that more confident people are statistically likely to be more skilled. You are right, they did mention self-evaluation of score. "Lest one think these results reflect erroneous peer assessment rather then erroneous self-assessment, participants in the bottom quartile also overestimated the number of test items they had gotten right by nearly 50%." But the results and conclusions of all the studies are focusing on the rankings and not the scores. It's difficult to even understand what the absolute range of scores is, and of course we don't see the actual questions they asked, so we don't know what the scores mean. Figures 2 and 3 say to me that zero people in the study knew how well they ranked or what their score was, the survey data is basically flat with noise. This could be explained by questions that are hard to understand, or by participants guessing at the answers. They didn't propose those explanations though, they wax on and on about how people who underperform overestimate their abilities. How do we know the participants even tried to answer correctly? They were all undergrads getting extra credit (which suggests they needed extra credit and were within the range of grades where it might help to get extra credit... consider what that means relative to skill: it's a narrow range of students not getting an F nor getting an A.) And they were getting extra credit just for being there, not for answering correctly. What if they were lazy? What if they were told their answers didn't matter? Why should we assume they didn't answer randomly? Even if they did try, the participants were all Ivy League undergrads who have been told they're among the elite and made it through a highly selective process of admission. Of course they rate & rank themselves highly. 100% of the participants were in a single college psych class, there were no uneducated people, no morons, and nobody who actually had meaningful "deficits in metacognitive skill" relative to the general population. Why is the set of people used in this study not considered an invalidating confounding factor for the results?