4 ms·

The null hypothesis for Dunning-Kruger isn't "people of all skill levels are good at estimating their performance" it's "people of all skill levels have equal b

The null hypothesis for Dunning-Kruger isn't "people of all skill levels are good at estimating their performance" it's "people of all skill levels have equal bias" in estimating their performance" (remember that Dunning-Kruger isn't that lower skilled people are bad at estimating their skill, it's that they systematically overestimate their skill). The randomly generated data used is one example of that, albeit an unrealistic one: a world in which all people are completely incapable of estimating their performance at all. In this case all of them have no bias at all in their (totally random) estimates. The fact that the artifact is seen in such data is a powerful demonstration that it is not evidence of the Dunning-Kruger effect. Re the other experiment, as I say it's just one study and I haven't looked deeply into it or others (and nor do I have a position on whether there is a real effect of this nature, or any great interest in it). The point is the article isn't claiming their randomly generated data example is evidence against the Dunning-Kruger effect itself. That needs further experiments such as the one they showed. The random data example is a demonstration that the original paper's analysis is flawed and doesn't support its conclusions.

geysersam 5y ago
Maybe another way of putting it is that the "Dunner-Kruger effect" is simply a tautology. Some formulation of it can still be true - albeit in a rather uninteresting way.
- alecbz 5y ago
  It’s not a complete tautology though, if people’s estimates of their skill were accurate in an unbiased way, we wouldn’t see a DK effect (or we’d only see a slight one, since you can’t really be unbiased at the low and high ends of the spectrum as the other comment pointed out). This isn’t true in the case of uniform random data, or in the real data we see, but it could be true of some data.
  - geysersam 5y ago
    Yes, you are right. It's not a tautology.
andersource 5y ago
> The null hypothesis for Dunning-Kruger isn't "people of all skill levels are good at estimating their performance" it's "people of all skill levels have equal bias" in estimating their performance" OK, > remember that Dunning-Kruger isn't that lower skilled people are bad at estimating their skill, it's that they systematically overestimate their skill The way I see it these aren't very different - conditioning on low skill and randomly sampling will tend to give way more overestimates than underestimates. What is the distinction between being bad at the skill and bad at estimation, and being bad at the skill and systematic overestimation? > The fact that the artifact is seen in such data is a powerful demonstration that it is not evidence of the Dunning-Kruger effect "Such data" refers to a world where everyone have absolutely no idea how good or bad they are. To me that is a much stronger argument than that made by DK. So perhaps our differences all come down to our priors. My prior belief (before looking at any data) is that people would know how good they are at a certain skill. If your prior is to expect that people don't know how good they are, then your arguments make sense to me. If however your prior is that people do know how good they are, but are also biased (all in the same direction), then I don't understand how the random data experiment reveals anything relevant to your beliefs.
- omnicognate 5y ago
  > What is the distinction between being bad at the skill and bad at estimation, and being bad at the skill and systematic overestimation? The difference between bias and variance. > conditioning on low skill and randomly sampling will tend to give way more overestimates than underestimates That's the hypothesis that's being tested.
  - andersource 5y ago
    > The difference between bias and variance. But when you're bad at the skill and can't underestimate, they look the same. > That's the hypothesis that's being tested And evidence from DK supports it.
    - omnicognate 5y ago

geysersam 5y ago

Maybe another way of putting it is that the "Dunner-Kruger effect" is simply a tautology. Some formulation of it can still be true - albeit in a rather uninteresting way.

alecbz 5y ago
It’s not a complete tautology though, if people’s estimates of their skill were accurate in an unbiased way, we wouldn’t see a DK effect (or we’d only see a slight one, since you can’t really be unbiased at the low and high ends of the spectrum as the other comment pointed out). This isn’t true in the case of uniform random data, or in the real data we see, but it could be true of some data.
- geysersam 5y ago
  Yes, you are right. It's not a tautology.

andersource 5y ago

> The null hypothesis for Dunning-Kruger isn't "people of all skill levels are good at estimating their performance" it's "people of all skill levels have equal bias" in estimating their performance" OK, > remember that Dunning-Kruger isn't that lower skilled people are bad at estimating their skill, it's that they systematically overestimate their skill The way I see it these aren't very different - conditioning on low skill and randomly sampling will tend to give way more overestimates than underestimates. What is the distinction between being bad at the skill and bad at estimation, and being bad at the skill and systematic overestimation? > The fact that the artifact is seen in such data is a powerful demonstration that it is not evidence of the Dunning-Kruger effect "Such data" refers to a world where everyone have absolutely no idea how good or bad they are. To me that is a much stronger argument than that made by DK. So perhaps our differences all come down to our priors. My prior belief (before looking at any data) is that people would know how good they are at a certain skill. If your prior is to expect that people don't know how good they are, then your arguments make sense to me. If however your prior is that people do know how good they are, but are also biased (all in the same direction), then I don't understand how the random data experiment reveals anything relevant to your beliefs.

omnicognate 5y ago
> What is the distinction between being bad at the skill and bad at estimation, and being bad at the skill and systematic overestimation? The difference between bias and variance. > conditioning on low skill and randomly sampling will tend to give way more overestimates than underestimates That's the hypothesis that's being tested.
- andersource 5y ago
  > The difference between bias and variance. But when you're bad at the skill and can't underestimate, they look the same. > That's the hypothesis that's being tested And evidence from DK supports it.
  - omnicognate 5y ago