5 ms·

> DK doesn't mean no correlation, it means inverse correlation. No, it means that people's self-assessment, their prediction of what their test scores will be,

by pdonis 5y ago

> DK doesn't mean no correlation, it means inverse correlation. No, it means that people's self-assessment, their prediction of what their test scores will be, is uncorrelated (or more precisely weakly correlated--that's what the original D-K data showed) with their actual test scores. Which is not what we would expect: we would expect that their predictions of their test scores would be strongly (or at least more strongly) correlated with their actual test scores. The question the D-K effect raises is why that is not the case, and it's a valid question--one which this article does not even attempt to answer.

bumby 5y ago
>their prediction of what their test scores will be, is uncorrelated I couldn't access the original 1999 full article because it's behind a paywall, but is this accurate? The abstract states that people "grossly overestimated their test performance and ability". If it were truly uncorrelated, wouldn't we expect there to be roughly equal number of people who "grossly underestimate" their performance? In any event, I wouldn't expect a weak correlation to be described as "grossly [anything]". Or maybe this is just the D-K effect in myself trying to understand the abstract :-)
- pdonis 5y ago
  > The abstract states that people "grossly overestimated their test performance and ability". I understand that the words D-K used were along those lines; but the actual data shown in their graphs says what I said. Just look at the lines on the graphs. The "actual" lines are 45 degree lines up and to the right--as they must be. But the "perceived" lines are horizontal, or roughly so. That means the two lines are uncorrelated, or nearly so. Saying that underperformers grossly overestimated their performance is only describing part of that--the part where the "perceived" lines on the left sides of the graphs are above the "actual" lines. It does not describe the other part--the part where the "perceived" lines on the right sides of the graphs are below the "actual" lines. D-K don't talk about that at all in their paper, which means their paper itself misdescribes the actual data they found. And if this article had said that, it would have been a valid criticism. But this article is not talking about what D-K said in words; it's talking about D-K's actual graph. And the article's claimed criticism of D-K's graph is not valid: the article is in fact just describing the same graph in different words. It's not offering a different "explanation" of why the graph looks that way; in fact, as I said, it offers no explanation of that at all; it doesn't even recognize that as a question.
  - bumby 5y ago
    Thanks for taking the time to help me understand the data. Looking at the graphs, isn't the implication that the distance between the two lines is the metric of interest? In other words, that the distance between real/perceived scores in Q1 & Q2 is "grossly" wider than the distance between Q3 & Q4, with the noted exception that only Q4 underestimates?
    - pdonis 5y ago
      > isn't the implication that the distance between the two lines is the metric of interest? That's more or less what D-K are saying, yes. That doesn't necessarily mean it's the best metric to use for understanding what the data is saying. Also, "distance between the two lines" is misleading because the data is bucketed--on the x axis by quartiles, on the y axis by percentiles. In other words, the actual data is the circles/triangles/squares, not the lines. When put that way, "distance between the lines" looks a lot less like an actual metric and a lot more like an artificial one.
      - bumby 5y ago
        Right, bad wording on my part. Thanks for correcting.
      - saurik 5y ago
        You seem to be first correcting the statement made by D-K in a way so compatible with this article that it is literally their example and then claiming that this corrected statement is correct so the article is wrong... the actual statement--as you even admit--was wrong, and that their data was fine was not being questioned by the article: it was their interpretation, which is what the D-K effect actually is... an inverse correlation between skill and perceived skill. Yes: if you choose to read their graph in a way which D-K didn't consider you can see the correct statement, but D-K didn't do that, and that isn't what anyone--but I guess you--means when they reference the conclusions of the paper and the resulting "effect", which purports to explain "why people tend to hold overly optimistic and miscalibrated views about themselves".
- pdonis 5y ago
  > I couldn't access the original 1999 full article because it's behind a paywall The article this thread is about links to a PDF of it, I was able to download it just fine: https://www.avaresearch.com/files/UnskilledAndUnawareOfIt.pdf https://www.avaresearch.com/files/UnskilledAndUnawareOfIt.pd...
- ianferrel 5y ago
  Overestimating vs underestimating their ability has less to do with whether their estimates are correlated and more to do with the average estimation. If estimates of ability were a truly random sample with mean of 75th percentile ability, then the net effect will be vast overestimation. If they're a truly random sample with mean at 50th percentile, it will be equal. At 25th, underestimation. It's like how 90% of people estimate that they're in the top 50% of drivers by ability.
- ACow_Adonis 5y ago
  having also not read the original paper recently, I'm in the camp that says the question is interesting and valid, but you've got to be careful about interpretation. how much the scores correlate (or autocorrelate) with self assessment IS an interesting question, and it's not apparent a priori what the answer will be, or whether the real effect we're interested in will differ for different levels of expertise. but the use of percentile to percentile (or quartiles, but those are just grouped percentiles) to give the impression of a particular kind of effect (lower groups overestimating and higher groups underestimating) is a flaw i think. it's a common one, in my experience, when dealing with percentages. if you think about it, percentages/percentiles have to be bounded at 0 or 100. for a dunning kruger effect not to appear, participants at both ends of the ability spectrum would have to be eerily accurate in their self-assessments. if they aren't, there's just more space on one side of the measurement scale for each group to make an error (if you score 1 in ability, there's ~99 percentiles available for you to make an overestimate and only 1 to be accurate, ditto for those with high ability. those in the middle of the ability group have equal chance on either side and so appear statistically more accurate even with a purely random distribution of guesses). so if there is any measure of central tendancy towards the middle percentile in the estimations of peoples abilities at all (and i would argue there is a priori reason to believe there would be, as the alternative would require those at both ends of the distribution to be getting increasingly accurate, which would be really weird), then practically any real world graph of percentile performance to percentile estimation will show a dunning kruger effect (with lower ends overestimating and higher ends underestimating). the article does a good job of showing this by plotting just random estimations and observing one appears.
pklausler 5y ago
I think of the DK effect in kind of a recursive way: gauging one's skill level is itself a skill. And if you are bad at that skill, you can't know it.
joe_the_user 5y ago
>> DK doesn't mean no correlation, it means inverse correlation. > No, it means that people's self-assessment, their prediction of what their test scores will be, is uncorrelated (or more precisely weakly correlated--that's what the original D-K data showed) with their actual test scores. The problem is that in common Internet discussions, there is no fixed meaning of D-K. When things heat up, D-K becomes "his belief in his competent is an indication of his incompetence,ha" but in more calm debate is become "less competent people have a somewhat less accurate understanding of their competence". In this form, D-K is a classic Motte and Baily device [1]. It's plausible, unsurprising and uninteresting that less competent people in a field self-access somewhat less accurately (the motte). It's completely unsupported but very "juicy" that someone's claim of competence indicates an incompetence (the bailey). [1] To one more pieces of rhetorical which I'm mention also proves nothing but is illuminating - https://en.wikipedia.org/wiki/Motte-and-bailey_fallacy https://en.wikipedia.org/wiki/Motte-and-bailey_fallacy
- pdonis 5y ago
  > in common Internet discussions, there is no fixed meaning of D-K That's why I focused explicitly on the actual data in the D-K paper and what it actually says.

by pdonis 5y ago

bumby 5y ago

>their prediction of what their test scores will be, is uncorrelated I couldn't access the original 1999 full article because it's behind a paywall, but is this accurate? The abstract states that people "grossly overestimated their test performance and ability". If it were truly uncorrelated, wouldn't we expect there to be roughly equal number of people who "grossly underestimate" their performance? In any event, I wouldn't expect a weak correlation to be described as "grossly [anything]". Or maybe this is just the D-K effect in myself trying to understand the abstract :-)

pdonis 5y ago
> The abstract states that people "grossly overestimated their test performance and ability". I understand that the words D-K used were along those lines; but the actual data shown in their graphs says what I said. Just look at the lines on the graphs. The "actual" lines are 45 degree lines up and to the right--as they must be. But the "perceived" lines are horizontal, or roughly so. That means the two lines are uncorrelated, or nearly so. Saying that underperformers grossly overestimated their performance is only describing part of that--the part where the "perceived" lines on the left sides of the graphs are above the "actual" lines. It does not describe the other part--the part where the "perceived" lines on the right sides of the graphs are below the "actual" lines. D-K don't talk about that at all in their paper, which means their paper itself misdescribes the actual data they found. And if this article had said that, it would have been a valid criticism. But this article is not talking about what D-K said in words; it's talking about D-K's actual graph. And the article's claimed criticism of D-K's graph is not valid: the article is in fact just describing the same graph in different words. It's not offering a different "explanation" of why the graph looks that way; in fact, as I said, it offers no explanation of that at all; it doesn't even recognize that as a question.
- bumby 5y ago
  Thanks for taking the time to help me understand the data. Looking at the graphs, isn't the implication that the distance between the two lines is the metric of interest? In other words, that the distance between real/perceived scores in Q1 & Q2 is "grossly" wider than the distance between Q3 & Q4, with the noted exception that only Q4 underestimates?
  - pdonis 5y ago
    > isn't the implication that the distance between the two lines is the metric of interest? That's more or less what D-K are saying, yes. That doesn't necessarily mean it's the best metric to use for understanding what the data is saying. Also, "distance between the two lines" is misleading because the data is bucketed--on the x axis by quartiles, on the y axis by percentiles. In other words, the actual data is the circles/triangles/squares, not the lines. When put that way, "distance between the lines" looks a lot less like an actual metric and a lot more like an artificial one.
    - bumby 5y ago
      Right, bad wording on my part. Thanks for correcting.
    - saurik 5y ago
      You seem to be first correcting the statement made by D-K in a way so compatible with this article that it is literally their example and then claiming that this corrected statement is correct so the article is wrong... the actual statement--as you even admit--was wrong, and that their data was fine was not being questioned by the article: it was their interpretation, which is what the D-K effect actually is... an inverse correlation between skill and perceived skill. Yes: if you choose to read their graph in a way which D-K didn't consider you can see the correct statement, but D-K didn't do that, and that isn't what anyone--but I guess you--means when they reference the conclusions of the paper and the resulting "effect", which purports to explain "why people tend to hold overly optimistic and miscalibrated views about themselves".
pdonis 5y ago
> I couldn't access the original 1999 full article because it's behind a paywall The article this thread is about links to a PDF of it, I was able to download it just fine: https://www.avaresearch.com/files/UnskilledAndUnawareOfIt.pdf https://www.avaresearch.com/files/UnskilledAndUnawareOfIt.pd...
ianferrel 5y ago
Overestimating vs underestimating their ability has less to do with whether their estimates are correlated and more to do with the average estimation. If estimates of ability were a truly random sample with mean of 75th percentile ability, then the net effect will be vast overestimation. If they're a truly random sample with mean at 50th percentile, it will be equal. At 25th, underestimation. It's like how 90% of people estimate that they're in the top 50% of drivers by ability.
ACow_Adonis 5y ago
having also not read the original paper recently, I'm in the camp that says the question is interesting and valid, but you've got to be careful about interpretation. how much the scores correlate (or autocorrelate) with self assessment IS an interesting question, and it's not apparent a priori what the answer will be, or whether the real effect we're interested in will differ for different levels of expertise. but the use of percentile to percentile (or quartiles, but those are just grouped percentiles) to give the impression of a particular kind of effect (lower groups overestimating and higher groups underestimating) is a flaw i think. it's a common one, in my experience, when dealing with percentages. if you think about it, percentages/percentiles have to be bounded at 0 or 100. for a dunning kruger effect not to appear, participants at both ends of the ability spectrum would have to be eerily accurate in their self-assessments. if they aren't, there's just more space on one side of the measurement scale for each group to make an error (if you score 1 in ability, there's ~99 percentiles available for you to make an overestimate and only 1 to be accurate, ditto for those with high ability. those in the middle of the ability group have equal chance on either side and so appear statistically more accurate even with a purely random distribution of guesses). so if there is any measure of central tendancy towards the middle percentile in the estimations of peoples abilities at all (and i would argue there is a priori reason to believe there would be, as the alternative would require those at both ends of the distribution to be getting increasingly accurate, which would be really weird), then practically any real world graph of percentile performance to percentile estimation will show a dunning kruger effect (with lower ends overestimating and higher ends underestimating). the article does a good job of showing this by plotting just random estimations and observing one appears.

pklausler 5y ago

I think of the DK effect in kind of a recursive way: gauging one's skill level is itself a skill. And if you are bad at that skill, you can't know it.

joe_the_user 5y ago

>> DK doesn't mean no correlation, it means inverse correlation. > No, it means that people's self-assessment, their prediction of what their test scores will be, is uncorrelated (or more precisely weakly correlated--that's what the original D-K data showed) with their actual test scores. The problem is that in common Internet discussions, there is no fixed meaning of D-K. When things heat up, D-K becomes "his belief in his competent is an indication of his incompetence,ha" but in more calm debate is become "less competent people have a somewhat less accurate understanding of their competence". In this form, D-K is a classic Motte and Baily device [1]. It's plausible, unsurprising and uninteresting that less competent people in a field self-access somewhat less accurately (the motte). It's completely unsupported but very "juicy" that someone's claim of competence indicates an incompetence (the bailey). [1] To one more pieces of rhetorical which I'm mention also proves nothing but is illuminating - https://en.wikipedia.org/wiki/Motte-and-bailey_fallacy https://en.wikipedia.org/wiki/Motte-and-bailey_fallacy

pdonis 5y ago
> in common Internet discussions, there is no fixed meaning of D-K That's why I focused explicitly on the actual data in the D-K paper and what it actually says.