3 ms·

I'm watching this thread with interest and will try to restate my understanding of GP's argument, by means of an example. If a person's true skill is 5 on a 1-

I'm watching this thread with interest and will try to restate my understanding of GP's argument, by means of an example. If a person's true skill is 5 on a 1-100 point scale, but the person is completely unaware of their true skill and will guess randomly, then their estimate will bias heavily in the direction of overestimating their skill, even if they were not intrinsically motivated to overestimate their skill, simply because far more of the available guesses are higher than their true ability. In other words, the available probability space itself biases in the direction of overestimating their ability, for those people. Is that right andersource? I don't know the statistical right answer here, but curious to know.

omnicognate 5y ago
It's worth reading the discussion between author and Nicolas Bonneel that starts with the first comment below the article. The author's explanation is very helpful regarding this point. The main point is that in the paper's randomly generated numbers example, the DK effect disappears if you measure the actual "skill" and the "prediction error" in separate, independent experiments. In the example if you take a "person" and conduct the test you get a totally random result, and you get another, independent totally random result if you test them again. If you perform your "actual skill" measurement using one of those test runs and your "skill estimation error" measurement using another, the DK effect disappears completely. So, to the extent the result of your skills test has any "noisiness" to it, if you analyse it the way Dunning & Kruger did, the autocorrelation resulting from using the same sample of that noise in the two things you're trying to assess the relationship between will show up as a powerful DK effect, and can easily swamp any actual correlations in the underlying distribution. Edit: Also worth mentioning footnote 3 on the article, which points out that the use of quantiles introduces a separate bias for the same reason you mention (about there being a minimum and maximum score).

omnicognate 5y ago

It's worth reading the discussion between author and Nicolas Bonneel that starts with the first comment below the article. The author's explanation is very helpful regarding this point. The main point is that in the paper's randomly generated numbers example, the DK effect disappears if you measure the actual "skill" and the "prediction error" in separate, independent experiments. In the example if you take a "person" and conduct the test you get a totally random result, and you get another, independent totally random result if you test them again. If you perform your "actual skill" measurement using one of those test runs and your "skill estimation error" measurement using another, the DK effect disappears completely. So, to the extent the result of your skills test has any "noisiness" to it, if you analyse it the way Dunning & Kruger did, the autocorrelation resulting from using the same sample of that noise in the two things you're trying to assess the relationship between will show up as a powerful DK effect, and can easily swamp any actual correlations in the underlying distribution. Edit: Also worth mentioning footnote 3 on the article, which points out that the use of quantiles introduces a separate bias for the same reason you mention (about there being a minimum and maximum score).