5 ms·

The fact that the statistical artifact is seen in completely uncorrelated data is only shown as a demonstration that it is not itself evidence of the claimed ef

The fact that the statistical artifact is seen in completely uncorrelated data is only shown as a demonstration that it is not itself evidence of the claimed effect. To gather evidence that the effect doesn't actually exist you need a new experiment, not just a new analysis of the same data, because the the different levels of actual skill need to be established separately from establishing the error in skill self-assessment. In the original experiment the test results were used to establish both, which is not sufficient. But the article presents the results from just such a new experiment. In this one they used university education level (sophomore through to professor) and measured skill self-assessment level within those groups. Higher education level (a good proxy for skill on the test used, which was about science literacy) was found to be associated with more accurate skill self-assessment but the bias of lower-skilled people overestimating their skills was not observed. That's just one study of course, but it sounds like a much better designed one than the original and does constitute actual evidence that the Dunning-Kruger effect doesn't exist.

andersource 5y ago
> The fact that the statistical artifact is seen in completely uncorrelated data is only shown as a demonstration that it is not itself evidence of the claimed effect I don't understand this part. "Completely uncorrelated data" is usually taken to represent the null hypothesis, but that's not the case here. In the DK paper, the implicit null hypothesis is "people of all skill levels are good at estimating their performance". In this case the "completely uncorrelated data" matches an alternative hypothesis, "people's skills have nothing to do with their ability to estimate their performance in tasks testing that skill". This hypothesis doesn't outright contradict the DK proposed hypothesis (and is certainly not the DK null hypothesis), so getting similar results is unsurprising to me, and I'm not sure that we learn from it anything about the DK results. As for the other study cited, the figure shown in the article doesn't give a lot of information on density, and looking at the paper itself, figure 4 does actually seem to show that self-assessment gradually shifts left with increasing level of education. (Edited for accuracy).
- omnicognate 5y ago
  The null hypothesis for Dunning-Kruger isn't "people of all skill levels are good at estimating their performance" it's "people of all skill levels have equal bias" in estimating their performance" (remember that Dunning-Kruger isn't that lower skilled people are bad at estimating their skill, it's that they systematically overestimate their skill). The randomly generated data used is one example of that, albeit an unrealistic one: a world in which all people are completely incapable of estimating their performance at all. In this case all of them have no bias at all in their (totally random) estimates. The fact that the artifact is seen in such data is a powerful demonstration that it is not evidence of the Dunning-Kruger effect. Re the other experiment, as I say it's just one study and I haven't looked deeply into it or others (and nor do I have a position on whether there is a real effect of this nature, or any great interest in it). The point is the article isn't claiming their randomly generated data example is evidence against the Dunning-Kruger effect itself. That needs further experiments such as the one they showed. The random data example is a demonstration that the original paper's analysis is flawed and doesn't support its conclusions.
  - geysersam 5y ago
    Maybe another way of putting it is that the "Dunner-Kruger effect" is simply a tautology. Some formulation of it can still be true - albeit in a rather uninteresting way.
    - alecbz 5y ago
      It’s not a complete tautology though, if people’s estimates of their skill were accurate in an unbiased way, we wouldn’t see a DK effect (or we’d only see a slight one, since you can’t really be unbiased at the low and high ends of the spectrum as the other comment pointed out). This isn’t true in the case of uniform random data, or in the real data we see, but it could be true of some data.
      - geysersam 5y ago
        Yes, you are right. It's not a tautology.

andersource 5y ago

> The fact that the statistical artifact is seen in completely uncorrelated data is only shown as a demonstration that it is not itself evidence of the claimed effect I don't understand this part. "Completely uncorrelated data" is usually taken to represent the null hypothesis, but that's not the case here. In the DK paper, the implicit null hypothesis is "people of all skill levels are good at estimating their performance". In this case the "completely uncorrelated data" matches an alternative hypothesis, "people's skills have nothing to do with their ability to estimate their performance in tasks testing that skill". This hypothesis doesn't outright contradict the DK proposed hypothesis (and is certainly not the DK null hypothesis), so getting similar results is unsurprising to me, and I'm not sure that we learn from it anything about the DK results. As for the other study cited, the figure shown in the article doesn't give a lot of information on density, and looking at the paper itself, figure 4 does actually seem to show that self-assessment gradually shifts left with increasing level of education. (Edited for accuracy).

omnicognate 5y ago
The null hypothesis for Dunning-Kruger isn't "people of all skill levels are good at estimating their performance" it's "people of all skill levels have equal bias" in estimating their performance" (remember that Dunning-Kruger isn't that lower skilled people are bad at estimating their skill, it's that they systematically overestimate their skill). The randomly generated data used is one example of that, albeit an unrealistic one: a world in which all people are completely incapable of estimating their performance at all. In this case all of them have no bias at all in their (totally random) estimates. The fact that the artifact is seen in such data is a powerful demonstration that it is not evidence of the Dunning-Kruger effect. Re the other experiment, as I say it's just one study and I haven't looked deeply into it or others (and nor do I have a position on whether there is a real effect of this nature, or any great interest in it). The point is the article isn't claiming their randomly generated data example is evidence against the Dunning-Kruger effect itself. That needs further experiments such as the one they showed. The random data example is a demonstration that the original paper's analysis is flawed and doesn't support its conclusions.
- geysersam 5y ago
  Maybe another way of putting it is that the "Dunner-Kruger effect" is simply a tautology. Some formulation of it can still be true - albeit in a rather uninteresting way.
  - alecbz 5y ago
    It’s not a complete tautology though, if people’s estimates of their skill were accurate in an unbiased way, we wouldn’t see a DK effect (or we’d only see a slight one, since you can’t really be unbiased at the low and high ends of the spectrum as the other comment pointed out). This isn’t true in the case of uniform random data, or in the real data we see, but it could be true of some data.
    - geysersam 5y ago
      Yes, you are right. It's not a tautology.