5 ms·

> therefore there would be strong a correlation between their performance and self-evaluation of said performance. If the data weren't related, then this hypoth

> therefore there would be strong a correlation between their performance and self-evaluation of said performance. If the data weren't related, then this hypothesis isn't likely, which is exactly what DK means. DK doesn't mean no correlation, it means inverse correlation. It's the correct analysis at the bottom that shows what no correlation actually looks like (at least no correlation in tend, there is heteroskedasticity). > a world in which people are very bad at estimating their own skill, therefore, statistically, people with lower skills tend to overestimate their skills, and experts tend to underestimate it. Be careful here, the conclusion you drew doesn't actually follow. > y - x ~ x, this is called the residual plot You're giving x and y meaning that they don't have. In the article these are uncorrelated random variables - the plot of y-x ~ x will always look that way. That's however not the case if you're plotting y_hat - y ~ y_hat for a y_hat taken out of a model. That won't be a random variable in your setup. Edit: note on heteroskedasticity

andersource 5y ago
> DK doesn't mean no correlation, it means inverse correlation. It's the correct analysis at the bottom that shows what no correlation actually looks like (at least no correlation in tend, there is heteroskedasticity). Not sure I follow, inverse correlation between what? The analysis at the bottom (assuming you mean fig. 11) is too dense to show if there's a correlation or not (between skill and self-assessment bias), and looking at the relevant figure from the paper itself gives me the impression that there is a correlation. >> a world in which people are very bad at estimating their own skill, therefore, statistically, people with lower skills tend to overestimate their skills, and experts tend to underestimate it. > Be careful here, the conclusion you drew doesn't actually follow. Can you elaborate? If X and Y are two independent random variables, X representing skill and Y representing self-assessment of skill, X and Y will be negatively correlated - this is exactly what the first part of the article is about, although from my perspective it's the author who's drawing the wrong conclusion. >> y - x ~ x, this is called the residual plot > You're giving x and y meaning that they don't have. In the article these are uncorrelated random variables - the plot of y-x ~ x will always look that way. That's however not the case if you're plotting y_hat - y ~ y_hat for a y_hat taken out of a model. That won't be a random variable in your setup. Not following again. Other than calling x y_hat, and having y_hat be your own estimate vs. x be the subjects' estimate, what is the distinction? What do you mean by "the plot of y-x ~ x will always look that way" - what way? The shape of the plot will necessarily depend on the relationship between x and y.
- pedrosorio 5y ago
  > If X and Y are two independent random variables, X representing skill and Y representing self-assessment of skill, X and Y will be negatively correlated “If two variables are independent, then their correlation will be 0” https://web.stanford.edu/class/archive/cs/cs109/cs109.1178/lectureHandouts/150-covariance.pdf https://web.stanford.edu/class/archive/cs/cs109/cs109.1178/l... “ If the variables are independent, Pearson's correlation coefficient is 0” https://en.wikipedia.org/wiki/Correlation#Correlation_and_independence https://en.wikipedia.org/wiki/Correlation#Correlation_and_in... Proof: https://www.themathcitadel.com/uncorrelated-and-independent-related-but-not-equivalent/ https://www.themathcitadel.com/uncorrelated-and-independent-...
  - andersource 5y ago
    Right, sorry, my bad - X and Y - X will be negatively correlated
- leto_ii 5y ago
  > If X and Y are two independent random variables, X representing skill and Y representing self-assessment of skill, X and Y will be negatively correlated 1. As mentioned in another comment, X and Y can't be independent and correlated at the same time 2. The point of the article is to show that you can replicate the results of the DK paper starting from purely random data. In the article X and Y don't mean anything, they're just random variables that the author draws samples from. The fact that you can get DK results from this very strongly suggests that DK is just an artifact of statistics, not an actual result. > What do you mean by "the plot of y-x ~ x will always look that way" - what way? See Figure 8 in the article, also panel B in Figure 10. > The shape of the plot will necessarily depend on the relationship between x and y. What I meant to say is that since X and Y are simulated data, not actual observations, the shape in Figure 8 will not actually depend on any possible relationship between ability and self-assessment. It's just a statistical artifact. Figure 9 is based on this simulated data as well, and since it closely replicates Figure 2 there's good reason to believe that Figure 2 itself is actually just a statistical artifact and that the DK data don't actually show the purported correlation. This point is further strengthened by referring to a few papers and by showing some corrected results in Figure 11.
tacitusarc 5y ago
> > a world in which people are very bad at estimating their own skill, therefore, statistically, people with lower skills tend to overestimate their skills, and experts tend to underestimate it. > Be careful here, the conclusion you drew doesn't actually follow. How does that not follow? It's just regression to the mean.
- dangerlibrary 5y ago
  I believe the god-emperor was implying that it is possible to imagine a world where people are bad at estimating their own skill, but in the other direction - people who are good at something drastically overestimate how good they are, and vice-versa.
- jkqwzsoo 5y ago
  > a world in which people are very bad at estimating their own skill, therefore, statistically, *people with lower skills tend to overestimate their skills, and experts tend to underestimate it*. I think the correct conclusion is that if a cohort is not good at estimating their own skill, you can conclude that the variance in their predicted self-assessment will be high (since they’re concluding things without strong evidence), but you can’t assume that their estimates will be biased without additional evidence. That is, indeed, what was shown in the last panel of the article: freshman (and undergraduates in general) are much worse at assessing their own ability than professors, but no group has a strong bias towards over- or under-assessing their own ability (recalling the “my guesses are much better than your guesses” from “The Death of Expertise”).
pdonis 5y ago
> DK doesn't mean no correlation, it means inverse correlation. No, it means that people's self-assessment, their prediction of what their test scores will be, is uncorrelated (or more precisely weakly correlated--that's what the original D-K data showed) with their actual test scores. Which is not what we would expect: we would expect that their predictions of their test scores would be strongly (or at least more strongly) correlated with their actual test scores. The question the D-K effect raises is why that is not the case, and it's a valid question--one which this article does not even attempt to answer.
- bumby 5y ago
  >their prediction of what their test scores will be, is uncorrelated I couldn't access the original 1999 full article because it's behind a paywall, but is this accurate? The abstract states that people "grossly overestimated their test performance and ability". If it were truly uncorrelated, wouldn't we expect there to be roughly equal number of people who "grossly underestimate" their performance? In any event, I wouldn't expect a weak correlation to be described as "grossly [anything]". Or maybe this is just the D-K effect in myself trying to understand the abstract :-)
  - pdonis 5y ago
    > The abstract states that people "grossly overestimated their test performance and ability". I understand that the words D-K used were along those lines; but the actual data shown in their graphs says what I said. Just look at the lines on the graphs. The "actual" lines are 45 degree lines up and to the right--as they must be. But the "perceived" lines are horizontal, or roughly so. That means the two lines are uncorrelated, or nearly so. Saying that underperformers grossly overestimated their performance is only describing part of that--the part where the "perceived" lines on the left sides of the graphs are above the "actual" lines. It does not describe the other part--the part where the "perceived" lines on the right sides of the graphs are below the "actual" lines. D-K don't talk about that at all in their paper, which means their paper itself misdescribes the actual data they found. And if this article had said that, it would have been a valid criticism. But this article is not talking about what D-K said in words; it's talking about D-K's actual graph. And the article's claimed criticism of D-K's graph is not valid: the article is in fact just describing the same graph in different words. It's not offering a different "explanation" of why the graph looks that way; in fact, as I said, it offers no explanation of that at all; it doesn't even recognize that as a question.
uncomputation 5y ago
> DK doesn't mean no correlation, it means inverse correlation Both are wrong. DK effect describes a weak positive correlation, but weaker than we intuitively expect. The top quartile still estimates their ability better than the bottom - positive correlation. But this is still below their actual ability. The bottom quartile correctly predicts they score worse than the top, but they overestimate their own ability ie they underestimate how much better the top quartile actually performs.

andersource 5y ago

> DK doesn't mean no correlation, it means inverse correlation. It's the correct analysis at the bottom that shows what no correlation actually looks like (at least no correlation in tend, there is heteroskedasticity). Not sure I follow, inverse correlation between what? The analysis at the bottom (assuming you mean fig. 11) is too dense to show if there's a correlation or not (between skill and self-assessment bias), and looking at the relevant figure from the paper itself gives me the impression that there is a correlation. >> a world in which people are very bad at estimating their own skill, therefore, statistically, people with lower skills tend to overestimate their skills, and experts tend to underestimate it. > Be careful here, the conclusion you drew doesn't actually follow. Can you elaborate? If X and Y are two independent random variables, X representing skill and Y representing self-assessment of skill, X and Y will be negatively correlated - this is exactly what the first part of the article is about, although from my perspective it's the author who's drawing the wrong conclusion. >> y - x ~ x, this is called the residual plot > You're giving x and y meaning that they don't have. In the article these are uncorrelated random variables - the plot of y-x ~ x will always look that way. That's however not the case if you're plotting y_hat - y ~ y_hat for a y_hat taken out of a model. That won't be a random variable in your setup. Not following again. Other than calling x y_hat, and having y_hat be your own estimate vs. x be the subjects' estimate, what is the distinction? What do you mean by "the plot of y-x ~ x will always look that way" - what way? The shape of the plot will necessarily depend on the relationship between x and y.

pedrosorio 5y ago
> If X and Y are two independent random variables, X representing skill and Y representing self-assessment of skill, X and Y will be negatively correlated “If two variables are independent, then their correlation will be 0” https://web.stanford.edu/class/archive/cs/cs109/cs109.1178/lectureHandouts/150-covariance.pdf https://web.stanford.edu/class/archive/cs/cs109/cs109.1178/l... “ If the variables are independent, Pearson's correlation coefficient is 0” https://en.wikipedia.org/wiki/Correlation#Correlation_and_independence https://en.wikipedia.org/wiki/Correlation#Correlation_and_in... Proof: https://www.themathcitadel.com/uncorrelated-and-independent-related-but-not-equivalent/ https://www.themathcitadel.com/uncorrelated-and-independent-...
- andersource 5y ago
  Right, sorry, my bad - X and Y - X will be negatively correlated
leto_ii 5y ago
> If X and Y are two independent random variables, X representing skill and Y representing self-assessment of skill, X and Y will be negatively correlated 1. As mentioned in another comment, X and Y can't be independent and correlated at the same time 2. The point of the article is to show that you can replicate the results of the DK paper starting from purely random data. In the article X and Y don't mean anything, they're just random variables that the author draws samples from. The fact that you can get DK results from this very strongly suggests that DK is just an artifact of statistics, not an actual result. > What do you mean by "the plot of y-x ~ x will always look that way" - what way? See Figure 8 in the article, also panel B in Figure 10. > The shape of the plot will necessarily depend on the relationship between x and y. What I meant to say is that since X and Y are simulated data, not actual observations, the shape in Figure 8 will not actually depend on any possible relationship between ability and self-assessment. It's just a statistical artifact. Figure 9 is based on this simulated data as well, and since it closely replicates Figure 2 there's good reason to believe that Figure 2 itself is actually just a statistical artifact and that the DK data don't actually show the purported correlation. This point is further strengthened by referring to a few papers and by showing some corrected results in Figure 11.

tacitusarc 5y ago

> > a world in which people are very bad at estimating their own skill, therefore, statistically, people with lower skills tend to overestimate their skills, and experts tend to underestimate it. > Be careful here, the conclusion you drew doesn't actually follow. How does that not follow? It's just regression to the mean.

dangerlibrary 5y ago
I believe the god-emperor was implying that it is possible to imagine a world where people are bad at estimating their own skill, but in the other direction - people who are good at something drastically overestimate how good they are, and vice-versa.
jkqwzsoo 5y ago
> a world in which people are very bad at estimating their own skill, therefore, statistically, *people with lower skills tend to overestimate their skills, and experts tend to underestimate it*. I think the correct conclusion is that if a cohort is not good at estimating their own skill, you can conclude that the variance in their predicted self-assessment will be high (since they’re concluding things without strong evidence), but you can’t assume that their estimates will be biased without additional evidence. That is, indeed, what was shown in the last panel of the article: freshman (and undergraduates in general) are much worse at assessing their own ability than professors, but no group has a strong bias towards over- or under-assessing their own ability (recalling the “my guesses are much better than your guesses” from “The Death of Expertise”).

pdonis 5y ago

> DK doesn't mean no correlation, it means inverse correlation. No, it means that people's self-assessment, their prediction of what their test scores will be, is uncorrelated (or more precisely weakly correlated--that's what the original D-K data showed) with their actual test scores. Which is not what we would expect: we would expect that their predictions of their test scores would be strongly (or at least more strongly) correlated with their actual test scores. The question the D-K effect raises is why that is not the case, and it's a valid question--one which this article does not even attempt to answer.

bumby 5y ago
>their prediction of what their test scores will be, is uncorrelated I couldn't access the original 1999 full article because it's behind a paywall, but is this accurate? The abstract states that people "grossly overestimated their test performance and ability". If it were truly uncorrelated, wouldn't we expect there to be roughly equal number of people who "grossly underestimate" their performance? In any event, I wouldn't expect a weak correlation to be described as "grossly [anything]". Or maybe this is just the D-K effect in myself trying to understand the abstract :-)
- pdonis 5y ago
  > The abstract states that people "grossly overestimated their test performance and ability". I understand that the words D-K used were along those lines; but the actual data shown in their graphs says what I said. Just look at the lines on the graphs. The "actual" lines are 45 degree lines up and to the right--as they must be. But the "perceived" lines are horizontal, or roughly so. That means the two lines are uncorrelated, or nearly so. Saying that underperformers grossly overestimated their performance is only describing part of that--the part where the "perceived" lines on the left sides of the graphs are above the "actual" lines. It does not describe the other part--the part where the "perceived" lines on the right sides of the graphs are below the "actual" lines. D-K don't talk about that at all in their paper, which means their paper itself misdescribes the actual data they found. And if this article had said that, it would have been a valid criticism. But this article is not talking about what D-K said in words; it's talking about D-K's actual graph. And the article's claimed criticism of D-K's graph is not valid: the article is in fact just describing the same graph in different words. It's not offering a different "explanation" of why the graph looks that way; in fact, as I said, it offers no explanation of that at all; it doesn't even recognize that as a question.

uncomputation 5y ago

> DK doesn't mean no correlation, it means inverse correlation Both are wrong. DK effect describes a weak positive correlation, but weaker than we intuitively expect. The top quartile still estimates their ability better than the bottom - positive correlation. But this is still below their actual ability. The bottom quartile correctly predicts they score worse than the top, but they overestimate their own ability ie they underestimate how much better the top quartile actually performs.