3 ms·

>their prediction of what their test scores will be, is uncorrelated I couldn't access the original 1999 full article because it's behind a paywall, but is thi

by bumby 5y ago

>their prediction of what their test scores will be, is uncorrelated I couldn't access the original 1999 full article because it's behind a paywall, but is this accurate? The abstract states that people "grossly overestimated their test performance and ability". If it were truly uncorrelated, wouldn't we expect there to be roughly equal number of people who "grossly underestimate" their performance? In any event, I wouldn't expect a weak correlation to be described as "grossly [anything]". Or maybe this is just the D-K effect in myself trying to understand the abstract :-)

pdonis 5y ago
> The abstract states that people "grossly overestimated their test performance and ability". I understand that the words D-K used were along those lines; but the actual data shown in their graphs says what I said. Just look at the lines on the graphs. The "actual" lines are 45 degree lines up and to the right--as they must be. But the "perceived" lines are horizontal, or roughly so. That means the two lines are uncorrelated, or nearly so. Saying that underperformers grossly overestimated their performance is only describing part of that--the part where the "perceived" lines on the left sides of the graphs are above the "actual" lines. It does not describe the other part--the part where the "perceived" lines on the right sides of the graphs are below the "actual" lines. D-K don't talk about that at all in their paper, which means their paper itself misdescribes the actual data they found. And if this article had said that, it would have been a valid criticism. But this article is not talking about what D-K said in words; it's talking about D-K's actual graph. And the article's claimed criticism of D-K's graph is not valid: the article is in fact just describing the same graph in different words. It's not offering a different "explanation" of why the graph looks that way; in fact, as I said, it offers no explanation of that at all; it doesn't even recognize that as a question.
- bumby 5y ago
  Thanks for taking the time to help me understand the data. Looking at the graphs, isn't the implication that the distance between the two lines is the metric of interest? In other words, that the distance between real/perceived scores in Q1 & Q2 is "grossly" wider than the distance between Q3 & Q4, with the noted exception that only Q4 underestimates?
  - pdonis 5y ago
    > isn't the implication that the distance between the two lines is the metric of interest? That's more or less what D-K are saying, yes. That doesn't necessarily mean it's the best metric to use for understanding what the data is saying. Also, "distance between the two lines" is misleading because the data is bucketed--on the x axis by quartiles, on the y axis by percentiles. In other words, the actual data is the circles/triangles/squares, not the lines. When put that way, "distance between the lines" looks a lot less like an actual metric and a lot more like an artificial one.
    - bumby 5y ago
      Right, bad wording on my part. Thanks for correcting.
    - saurik 5y ago
      You seem to be first correcting the statement made by D-K in a way so compatible with this article that it is literally their example and then claiming that this corrected statement is correct so the article is wrong... the actual statement--as you even admit--was wrong, and that their data was fine was not being questioned by the article: it was their interpretation, which is what the D-K effect actually is... an inverse correlation between skill and perceived skill. Yes: if you choose to read their graph in a way which D-K didn't consider you can see the correct statement, but D-K didn't do that, and that isn't what anyone--but I guess you--means when they reference the conclusions of the paper and the resulting "effect", which purports to explain "why people tend to hold overly optimistic and miscalibrated views about themselves".
      - pdonis 5y ago
        > it is literally their example No, it isn't. The article is claiming that the D-K effect doesn't actually exist; actually all that is going on is "autocorrelation". That claim is wrong. The D-K effect does exist, but it isn't what D-K said it was: it's not that there is an inverse correlation between perceived skill and actual skill; it is that there is no correlation (or at best a weak positive correlation) between perceived skill and actual skill (when we would intuitively expect a fairly strong positive correlation between them). That is what D-K's actual data shows; but neither D-K themselves in their paper, nor this article, correctly describe that.
pdonis 5y ago
> I couldn't access the original 1999 full article because it's behind a paywall The article this thread is about links to a PDF of it, I was able to download it just fine: https://www.avaresearch.com/files/UnskilledAndUnawareOfIt.pdf https://www.avaresearch.com/files/UnskilledAndUnawareOfIt.pd...
ianferrel 5y ago
Overestimating vs underestimating their ability has less to do with whether their estimates are correlated and more to do with the average estimation. If estimates of ability were a truly random sample with mean of 75th percentile ability, then the net effect will be vast overestimation. If they're a truly random sample with mean at 50th percentile, it will be equal. At 25th, underestimation. It's like how 90% of people estimate that they're in the top 50% of drivers by ability.
ACow_Adonis 5y ago
having also not read the original paper recently, I'm in the camp that says the question is interesting and valid, but you've got to be careful about interpretation. how much the scores correlate (or autocorrelate) with self assessment IS an interesting question, and it's not apparent a priori what the answer will be, or whether the real effect we're interested in will differ for different levels of expertise. but the use of percentile to percentile (or quartiles, but those are just grouped percentiles) to give the impression of a particular kind of effect (lower groups overestimating and higher groups underestimating) is a flaw i think. it's a common one, in my experience, when dealing with percentages. if you think about it, percentages/percentiles have to be bounded at 0 or 100. for a dunning kruger effect not to appear, participants at both ends of the ability spectrum would have to be eerily accurate in their self-assessments. if they aren't, there's just more space on one side of the measurement scale for each group to make an error (if you score 1 in ability, there's ~99 percentiles available for you to make an overestimate and only 1 to be accurate, ditto for those with high ability. those in the middle of the ability group have equal chance on either side and so appear statistically more accurate even with a purely random distribution of guesses). so if there is any measure of central tendancy towards the middle percentile in the estimations of peoples abilities at all (and i would argue there is a priori reason to believe there would be, as the alternative would require those at both ends of the distribution to be getting increasingly accurate, which would be really weird), then practically any real world graph of percentile performance to percentile estimation will show a dunning kruger effect (with lower ends overestimating and higher ends underestimating). the article does a good job of showing this by plotting just random estimations and observing one appears.

by bumby 5y ago

pdonis 5y ago

> The abstract states that people "grossly overestimated their test performance and ability". I understand that the words D-K used were along those lines; but the actual data shown in their graphs says what I said. Just look at the lines on the graphs. The "actual" lines are 45 degree lines up and to the right--as they must be. But the "perceived" lines are horizontal, or roughly so. That means the two lines are uncorrelated, or nearly so. Saying that underperformers grossly overestimated their performance is only describing part of that--the part where the "perceived" lines on the left sides of the graphs are above the "actual" lines. It does not describe the other part--the part where the "perceived" lines on the right sides of the graphs are below the "actual" lines. D-K don't talk about that at all in their paper, which means their paper itself misdescribes the actual data they found. And if this article had said that, it would have been a valid criticism. But this article is not talking about what D-K said in words; it's talking about D-K's actual graph. And the article's claimed criticism of D-K's graph is not valid: the article is in fact just describing the same graph in different words. It's not offering a different "explanation" of why the graph looks that way; in fact, as I said, it offers no explanation of that at all; it doesn't even recognize that as a question.

bumby 5y ago
Thanks for taking the time to help me understand the data. Looking at the graphs, isn't the implication that the distance between the two lines is the metric of interest? In other words, that the distance between real/perceived scores in Q1 & Q2 is "grossly" wider than the distance between Q3 & Q4, with the noted exception that only Q4 underestimates?
- pdonis 5y ago
  > isn't the implication that the distance between the two lines is the metric of interest? That's more or less what D-K are saying, yes. That doesn't necessarily mean it's the best metric to use for understanding what the data is saying. Also, "distance between the two lines" is misleading because the data is bucketed--on the x axis by quartiles, on the y axis by percentiles. In other words, the actual data is the circles/triangles/squares, not the lines. When put that way, "distance between the lines" looks a lot less like an actual metric and a lot more like an artificial one.
  - bumby 5y ago
    Right, bad wording on my part. Thanks for correcting.
  - saurik 5y ago
    You seem to be first correcting the statement made by D-K in a way so compatible with this article that it is literally their example and then claiming that this corrected statement is correct so the article is wrong... the actual statement--as you even admit--was wrong, and that their data was fine was not being questioned by the article: it was their interpretation, which is what the D-K effect actually is... an inverse correlation between skill and perceived skill. Yes: if you choose to read their graph in a way which D-K didn't consider you can see the correct statement, but D-K didn't do that, and that isn't what anyone--but I guess you--means when they reference the conclusions of the paper and the resulting "effect", which purports to explain "why people tend to hold overly optimistic and miscalibrated views about themselves".
    - pdonis 5y ago
      > it is literally their example No, it isn't. The article is claiming that the D-K effect doesn't actually exist; actually all that is going on is "autocorrelation". That claim is wrong. The D-K effect does exist, but it isn't what D-K said it was: it's not that there is an inverse correlation between perceived skill and actual skill; it is that there is no correlation (or at best a weak positive correlation) between perceived skill and actual skill (when we would intuitively expect a fairly strong positive correlation between them). That is what D-K's actual data shows; but neither D-K themselves in their paper, nor this article, correctly describe that.

pdonis 5y ago

> I couldn't access the original 1999 full article because it's behind a paywall The article this thread is about links to a PDF of it, I was able to download it just fine: https://www.avaresearch.com/files/UnskilledAndUnawareOfIt.pdf https://www.avaresearch.com/files/UnskilledAndUnawareOfIt.pd...

ianferrel 5y ago

Overestimating vs underestimating their ability has less to do with whether their estimates are correlated and more to do with the average estimation. If estimates of ability were a truly random sample with mean of 75th percentile ability, then the net effect will be vast overestimation. If they're a truly random sample with mean at 50th percentile, it will be equal. At 25th, underestimation. It's like how 90% of people estimate that they're in the top 50% of drivers by ability.

ACow_Adonis 5y ago

having also not read the original paper recently, I'm in the camp that says the question is interesting and valid, but you've got to be careful about interpretation. how much the scores correlate (or autocorrelate) with self assessment IS an interesting question, and it's not apparent a priori what the answer will be, or whether the real effect we're interested in will differ for different levels of expertise. but the use of percentile to percentile (or quartiles, but those are just grouped percentiles) to give the impression of a particular kind of effect (lower groups overestimating and higher groups underestimating) is a flaw i think. it's a common one, in my experience, when dealing with percentages. if you think about it, percentages/percentiles have to be bounded at 0 or 100. for a dunning kruger effect not to appear, participants at both ends of the ability spectrum would have to be eerily accurate in their self-assessments. if they aren't, there's just more space on one side of the measurement scale for each group to make an error (if you score 1 in ability, there's ~99 percentiles available for you to make an overestimate and only 1 to be accurate, ditto for those with high ability. those in the middle of the ability group have equal chance on either side and so appear statistically more accurate even with a purely random distribution of guesses). so if there is any measure of central tendancy towards the middle percentile in the estimations of peoples abilities at all (and i would argue there is a priori reason to believe there would be, as the alternative would require those at both ends of the distribution to be getting increasingly accurate, which would be really weird), then practically any real world graph of percentile performance to percentile estimation will show a dunning kruger effect (with lower ends overestimating and higher ends underestimating). the article does a good job of showing this by plotting just random estimations and observing one appears.