12 ms·

The Dunning-Kruger effect is autocorrelation

• hn_throwaway_99 3y ago
  Previous discussion: https://news.ycombinator.com/item?id=31036800 https://news.ycombinator.com/item?id=31036800
• eterevsky 3y ago
  I think this article would've made more sense if it had a title "The Dunning-Kruger effect is regression toward the mean", because that's what the author is actually showing.
  • tgv 3y ago
    I think your description is the most apt. OP's own analysis shows that using random data (two variables uniformly distributed over the same range!) for both skill and self-assessment results in a different graph. The original comparison therefor implies another effect on the second dimension, which could be interpreted as: people don't estimate their skills correctly, but drift towards the mean. But then the question becomes: what did they really ask their subjects? To pick the percentile or a true test score?
• Jensson 3y ago
  Psychologists using their pet theories to explain results and then people taking that explanation as the truth when they should really just look at the data is probably an as large problem as the replication crisis.
• glitchc 3y ago
  Geez, this is eye-opening. Thank you for sharing this.
• tempestn 3y ago
  I don't buy this take, and this rebuttal does a better job than I could of explaining why: https://andersource.dev/2022/04/19/dk-autocorrelation.html https://andersource.dev/2022/04/19/dk-autocorrelation.html Basically, this autocorrelation take shows that if performance and evaluation of performance were random and independent, you would get a graph like the D-K one, and therefore it states that the effect is just autocorrelation. But in reality, it would be very surprising if performance and evaluation of performance were independent. We expect people to be able to accurately rate their own ability. And D-K did indeed show a correlation between the two, just not as strong of one as we would expect. Rather, they showed a consistent bias. That's the interesting result. They then posit reasons for this. One could certainly debate those reasons. But to say the whole effect is just a statistical artifact because random, independent variables would act in a similar way ignores the fact that these variables aren't expected to be independent.
  • Jensson 3y ago
    The effect that the worst overestimate their skill is known since before, that wasn't the main result of Dunning-Kruger. The effect that the best underestimate their skill can be chalked up to auto-correlation.
    • tempestn 3y ago
      The best don't tend to overestimate their skill; they underestimate it. The D-K results show a consistent bias in estimates toward (somewhere near) the mean. Hence an overestimate at the bottom and an underestimate at the top.
      • Jensson 3y ago
        > The best don't tend to overestimate their skill; they underestimate I wrote the wrong word, I fixed it. The best can't overestimate their rank, so of course that wasn't what I meant.
      • anonymouskimmer 3y ago
        Dunning-Kruger posits this as a psychological effect, yes? On the top half psychological effects such as imposter syndrome could come in to play. Have sociological factors such as being kind or big fish little pond been considered as likely causes of the misestimates?
        • chiefalchemist 3y ago
          I have the same question...why do some get it so wrong? Was there a nudge in the process of the study that caused some to answer what they did? Heck, I'm wondering if "Honestly, I can't say" was an allowed response. Or were they forced to pick a number? If so, then I'd want to know what happens when you ask 100 ppl to pick a number between 0 and 100. I bet it's not evenly distributed. Maybe the beginners give a "discounted" version of the distribution? Even if the autocorrection explanation is off, there does now seem to be flaws in DK, at least from the perspective of pure and proper science
  • 3y ago
• dmbche 3y ago
  Isn't it ironic that they fooled themselves?
  • ulizzle 3y ago
    It was actually hilarious but I don’t think many people here got the irony
    • DangitBobby 3y ago
      Literally the closing paragraph of TFA is about that exact irony.
      • robwwilliams 3y ago
        And here it is from OP (which made me laugh—right or wrong). And leave your hubris at home unless you rate yourself a damn fine statistician ;-) “However, there is a delightful irony to the circumstances of their blunder. Here are two Ivy League professors7 arguing that unskilled people have a ‘dual burden’: not only are unskilled people ‘incompetent’ … they are unaware of their own incompetence. “The irony is that the situation is actually reversed. In their seminal paper, Dunning and Kruger are the ones broadcasting their (statistical) incompetence by conflating autocorrelation for a psychological effect. In this light, the paper’s title may still be appropriate. It’s just that it was the authors (not the test subjects) who were ‘unskilled and unaware of it’.
• lencastre 3y ago
  Wasn’t this DK effect already debunked?
  • jahewson 3y ago
    I don’t know much about it but I’m sure you’re right.
  • hasch 3y ago
    Article mentions 2016 somewhere. They explain a bit on top of that, with more depth ... at least my rough take on this
  • xbar 3y ago
    Yes. This article highlights the 2016, 2017 and 2020 debunkings of DK. But it hangs on as an oft repeated scientific fallacy. The fact that anyone has to ask if it has debunked shows how desirable some people find the DK myth. Even in the comments here, people are not willing to be skeptical of DK. That's interesting psychology.
  • deleted 3y ago
    [deleted]
  • mrkeen 3y ago
    Yes but some claim to have debunked the debunking also. [1] This paper (2023) claims "the magnitude of the effect was minimal; bringing its meaningfulness into question." [2] [1] https://andersource.dev/2022/04/19/dk-autocorrelation.html https://andersource.dev/2022/04/19/dk-autocorrelation.html [2] https://www.sciencedirect.com/science/article/abs/pii/S0160289622000988 https://www.sciencedirect.com/science/article/abs/pii/S01602...
• pie_flavor 3y ago
  This take is a perfect example of Dunning-Kruger itself, ironically. https://andersource.dev/2022/04/19/dk-autocorrelation.html https://andersource.dev/2022/04/19/dk-autocorrelation.html
  • dahart 3y ago
    How so? DK shows a positive correlation between confidence and competence.
• mewpmewp2 3y ago
  My take on Dunning Kruger: 1. People really like the idea of smart people being humble and arrogance meaning stupidity, so they like to believe that DK is true, and they like to repeat this. 2. Some smart/skilled people are humble, some are arrogant. 3. Some smart/skilled people underestimate their skills, some overestimate. 4. Some stupid people are humble, some are arrogant. 5. Some stupid people underestimate their skills, some overestimate. Overall, even if there is a correlation, you can't tell by just arrogance of a person whether we are dealing with DK or whether it's an effect at all. People's personalities, skills and everything are a bit more complex than that. Overall bringing DK up seems like some sort of social justice/fairness effort rather than something that is actually true given any situation where someone is arrogant.
  • spacebacon 3y ago
    Maybe this shows how effective dumb people are at keeping smart people hammered down with thought stopping arguments.
• greenthrow 3y ago
  Lmao this article is an example of Dunning-Kruger at work. The author thinks they have found and are revealing something but they are just failing to fully understand the subject. Amazing.
  • flappyeagle 3y ago
    Try reading the article again and understanding the argument.
    • greenthrow 3y ago
      Oh I did. Completely.
      • mattxxx 3y ago
        Wait... but what if this is DK? What if my comment is DK??
• joefourier 3y ago
  So from my understanding, the Dunning-Kruger Effect paper doesn’t show the distribution of the perceived test scores nor the standard deviation, only an average, which rises with actual test score level. If they showed the spread bar in each bin, you could form very different conclusions. Do low skilled people consistently estimate their score at around 60, or do they give effectively random results centred around 60? Assuming the latter, it could mean that low skilled individuals are completely unable to evaluate their performance while higher skilled people are slightly better at it but still not very good, giving a slightly positive correlation which… is very distinct from what the DK effect implied.
• xanderlewis 3y ago
  Naïve take: I’ve always felt like Dunning-Kruger is just the result of the fact that when guessing the value of anything people tend towards some common mean, and so if the true value is low your guess tends to be high, and vice versa. This assumes nothing about what is being guessed, but does assume (perhaps wrongly) that there is a commonly believed mean value and that people tend to imagine they are close to it.
  • wavemode 3y ago
    That's essentially the plain-language interpretation of what the author of this article is pointing out - when you plot (actual score) against (difference between test score and actual score), you will always find a trend that underperformers overestimate and overperformers underestimate - for the exact reason you state.
• r0uv3n 3y ago
  The discussion between Nicolas Boneel and the author in the comments of the article is interesting and Nicolas expresses the doubts I had when reading this. The whole point of the DK effect is that people are bad at estimating their skill, so if you assume that they randomly guess their skill level then of course you will replicate the results. The correct model for a world without DK should be something like (estimated test scores)=(actual test scores)+noise, and then the only form of spurious DK you'd expect is caused by the fact that there's a minimum and maximum test score. But this effect would be proportional to the variance of the noise, and I assume the variance on the additional dataset is too low to fully understand the effect seen there. Also, in this model on average everyone should still guess correctly in which half of the distribution they are, but even the bottom quartile seemed to estimate their abilities as above the 50th percentile
  • svnt 3y ago
    Just because the data appear random doesn’t mean you’ve gotten at the cause though. From those charts it could equally be low skill throughout, or something nuanced like lack of skill at estimating at the bottom, improving skill in estimating through the middle, and high skill and learned modesty at the top.
  • Jensson 3y ago
    > Also, in this model on average everyone should still guess correctly in which half of the distribution they are, but even the bottom quartile seemed to estimate their abilities as above the 50th percentile Depends on the noise applied. If the noise is -10% to +100% for everyone then you get roughly the graph Dunning-Kruger got. So there is no reason to believe that the best are better at estimating their abilities, just that you can't estimate your own rank as better than the best.
    • tempestn 3y ago
      That's a great observation. For what it's worth though, it does seem logical to me that the best would also be best at estimating their skill. Not necessarily because they're better at it per se (though there's likely some of that too, for the reasons originally posited by D-K), but also because they have an easier problem to solve. When you know something well, it's fairly obvious that that's the case. (Think of the experience of acing a math test. It's entirely possible you'd know you answered everything correctly.) When you struggle somewhat though, it's much more difficult to estimate how much you're struggling compared to how others would fare.
• bitshiftfaced 3y ago
  The authors did "X - Y vs X," but that's not even the biggest problem. The authors subtracted two measures that had been transformed and bounded from 0 to 1 (think percentiles). What happens at the extremes of those bounds? How much can your top performers overestimate their performance? They're almost at 1 already, so not much. If they were to overestimate and underestimate at the same rate and by the same magnitude in terms of raw values, the ceiling effect on the transformed values means that the graph will make it look like they underestimate more often. The opposite problem happens for the worst performers. See "Random Number Simulations Reveal How Random Noise Affects the Measurements and Graphical Portrayals of Self-Assessed Competency." Numeracy 9, Iss. 1 (2016), particularly figures 7, 8, and 9.
  • anonymouskimmer 3y ago
    This can be dealt with to an extent by truncating the extreme ends. Even the middle quartiles in the graphs in the linked article show the same trends.
    • bitshiftfaced 3y ago
      Not that simple. This article demonstrates why enforcing bounds results in the changes in slope that you see in the expected grades (figure 2 and 4): https://www.frontiersin.org/articles/10.3389/fpsyg.2022.840180/full https://www.frontiersin.org/articles/10.3389/fpsyg.2022.8401...
  • ImaCake 3y ago
    Thanks for stating just how much of a statistical minefield this is. The reference does a great job showing just how wrong the DK studies are. Unfortunately, most people have already made up their minds and are happy to link conflicting blog posts as evidence.
    • Probiotic6081 3y ago
      Probably in another year or two they'll find another statistic that will render the old one moot like again and again.
• chiefalchemist 3y ago
  DK for me is simply: "You don't know what you don't know." When that happens, it's easy - surprise, surprise! - to misjudge your skill level. In a way, it almost feels cruel to ask someone with too few points of reference to say how much they know. The fact is whether high, low, or in the middle...they are guessing. On the other hand, with enough experience the depth and breadth of your context improves, as it should. At that point, mis-self-assessment is the result of arrogance, bravado, etc. That's a different problem than simply not knowing. If nothing else, DK has a case of apple v oranges.
• thewanderer1983 3y ago
  The Dunning-Kruger effect isn't as the article first quotes. It's an effect that everyone experiences. We as humans tend to over simplify things we don't understand well or at all. Therefore we over estimate our expertise on these subjects. We also tend to under estimate how much an expert on subjects we do know well. Everyone does this. It's not just dumb people.
  • Jensson 3y ago
    > We also tend to under estimate how much an expert on subjects we do know well Any evidence for this, except Dunning-Kruger? To me it looks like everyone overestimates themselves. There are a lot of professionals who think they are undervalued and that people worse than them gets all the rewards and fame.
• vismwasm 3y ago
  The author measures the Dunning Kruger effect on his random data exactly because he assumes it when generating his random data. By modelling skill and perceived skill as uniform draws between 0 and 100, the unskilled (e.g. skill=0) will over-estimate their skills (estimated skill = 50, the mean on the uniform random variable) and the skilled (e.g. skill=100) will underestimate it (as 50 as well, again the mean of the same random variable). The only ones who will be correct (on average) are the average skilled ones (skill=50).
• beltsazar 3y ago
  I don't know if I agree that it's an autocorrelation, but one way to explain The Dunning-Krugger Effect is by acknowledging this simple fact: Most people think that they are an average person, but they can't be all average—there must be some people substantially below the median. Therefore, those people must overestimate their abilities. This also applies to other aspects, such as attractiveness. Less attractive people would overestimate their attractiveness.
  • anonymouskimmer 3y ago
    For all of the tests and rebuttals of the Dunning-Kruger effect the people tested are not drawing from the totality of other people, but trying to compare themselves solely to those who also took the same test. Anyone in a position to take such a test is almost guaranteed to be above average compared to the general population (which includes babies for intellectual tests, or the extremely old for attractiveness tests). I think this complicates personal evaluation.
  • bigbacaloa 3y ago
    [dead]
• salty_biscuits 3y ago
  It's just correlation, why do they keep calling it autocorrelation.
  • stubish 3y ago
    auto correlation, or self correlation. A correlation between different things may indicate an actual relation (smoking is correlated with early mortality). A self correlation is a tautology.
• snarkconjecture 3y ago
  Nonstandard terminology warning: the author is using "autocorrelation" in a way I've never seen before. There is a much more common usage of "autocorrelation" to refer to the correlation of a timeseries with itself (shifted by some amount). If you use autocorrelation to refer to the thing in OP, you'll probably confuse people who know statistics, and vice versa.
  • ketozhang 3y ago
    The more common experience with autocorrelations are with time series, but what the author said is correct even in that context. A time series autocorrelation relates the same time series function at different times. At the simplest you plot the arrays X vs X where X[i] = f(t[i]). You then may complicate it further by some transformation g(X) vs X (e.g., moving average).
  • epigramx 3y ago
    you might say the article author might have some ..dunning-kruger on what autocorrelation is.
    • nothrowaways 3y ago
      L2 of dk
  • xpe 3y ago
    > Nonstandard terminology warning: the author is using "autocorrelation" in a way I've never seen before. That's a nice way of putting it. A more accurate description would be: the author is butchering the key essence of autocorrelation, since they don't clearly mention that it is a temporal relationship! > What is autocorrelation? > Autocorrelation occurs when you correlate a variable with itself. Groan. A standard definition is: > Autocorrelation refers to the degree of correlation of the same variables between two successive time intervals. It measures how the lagged version of the value of a variable is related to the original version of it in a time series. Autocorrelation, as a statistical concept, is also known as serial correlation.
  • gnicholas 3y ago
• anonymouskimmer 3y ago
  > If the Dunning-Kruger effect were present, it would show up in Figure 11 as a downward trend in the data (similar to the trend in Figure 7). Such a trend would indicate that unskilled people overestimate their ability, and that this overestimate decreases with skill. Looking at Figure 11, there is no hint of a trend. There certainly is a hint of a trend. Why do people, when visualizing data with a distinct trend, say that because the "error bars" from a particular statistical test overlap zero that no trend exists!? Freshman trend to over-confidence. Grad students trend to under-confidence. Undergrads in general trend to over-confidence (though this trend decreases as year in school increases), and post-graduates, whether grad students or professors, trend to under-confidence. These "trends" are not statistically significant, but they certainly are a trend! Also, the random data distribution in figure 9 doesn't show the same trends as Dunning-Kruger's curve in figure 2. Perhaps there is at least one psycho-social mechanism here worth investigating?
  • mrkeen 3y ago
    > These "trends" are not statistically significant, but they certainly are a trend! This is an oxymoron.
    • Dylan16807 3y ago
      Oxymorons only sound contradictory on a surface level. Something "certainly" being a "trend" is the definition of statistical significance, so this is a straight up contradiction.
      • anonymouskimmer 3y ago
        See here: https://news.ycombinator.com/item?id=38416858 https://news.ycombinator.com/item?id=38416858 "Trend" has multiple meanings. Statistics doesn't get to claim all of the meaning.
    • anonymouskimmer 3y ago
      Show how. I place mechanistic theory prior to statistics in science. Mechanistic theory can be tested, statistics are a kind of test. If a statistically-insignificant result shows consistent, though non-significant deviations, such as the kind seen in Figure 11, then it tells me it's worth investigating whether mechanism(s) are explaining a very small portion of the variation that will not, in itself, show up as statistically significant, as it's being swamped by variation in other parameters.
• abnry 3y ago
  If there is a linear relationship between test score (X, ability) and test score self-assessment (Y, self-perception), then the random variables are modeled as: $$ Y \sim aX+b+N $$ Where N is some statistically independent noise, mean zero. This means the covariance between them is $$ Cov(Y-X,X) = E[ ((a-1)X+b+N -(a-1)E[X]-b) (X - E[X]) ] $$ Which is $$ Cov(Y-X,X) = E[(a-1)(X-E[X])(X-E[X])] + E[N(X-E[X])]= (a-1) Var[X] $$ To get a "DK effect" we need (a-1) < 0, or a < 1. If a=0, in the case of the blog post, then this is absolutely true. If a=1 (which, along with b=0, is the ideal scenario), then this is barely not true. If a > 1, then we'd have a whole new effect about arrogant experts. So the only thing that matters from this "auto-correlation perspective" is the rate at which an individual's self-assessment increases with their ability. As long as they underestimate the increase, a "DK effect" will occur. However, in the above analysis, we ignored the variable b. If a = 0.8 and b=0, we'd never have the so-called "DK effect" even though it matches the "auto-correlation perspective" because everyone would underestimate their ability. This tells me that the value of b matters. It is sort of like the prior ability everyone assumes they have. What the DK papers shows is that b > .5, which I think is in line with the spirit of the popular interpretation of the "DK effect". People should not be assuming they have, at a minimum, a capacity higher than the average. At the same time, the value b isn't insanely higher than .5, which also makes me want to cut those unskilled and unaware some slack. It "seems reasonable" to assume your baseline is average. That can't be the case, but it feels intuitive.
• im3w1l 3y ago
  It's fascinating how great Elo and similar ranking systems are at curbing DK. You just get a number, and that's how good (bad) you are. It's incredibly precise too, there's just no arguing with it. Also since the topic is D-K I'm a bit scared that I'm the fool here, but isn't he misusing the term autocorrelation? What he describes sounds like just normal correlation?
• concordDance 3y ago
  The author fails to make his point quite badly. Of course if everyone's self assessment was random the bottom quartile would overrate themselves! And that would be half of the Dunning-Kruger effect and we could truthfully say "the bottom quartile of people overrate themselves"! The other part where those at the top have a better idea or where they rank noticeably does not come out in his toy example. Honestly, he comes across as not having the slightest understanding of how people interpet those graphs...
• toasted-subs 3y ago
  Idk I genuinely feel like after having to deal with 10+ doctors who all had different opinions. The last doctor finally made the same conclusion as me and he was the last person I had to see. There's always exceptions. And sometimes reading publications pertaining to a very specific thing should give you more say on a subject. I just feel bad American tax payer money and the best years of my life was spent on telling medical professionals they don't know what they are talking about.
• dclowd9901 3y ago
  I think what this article is missing is “the chart DK should have used.” Instead we get a spurious explanation that doesn’t make a lot of sense based on completely fabricated data. It’s entirely natural for something that looks like DK to emerge from randomized data, especially when the Y axis is represented by some number of the mean (actually 50ish in this case).
• a-dub 3y ago
  i think of acf as a measure of repeating temporal structure and how "strong" and "long" it is, if it exists. that is, it gives you a notion of if and what order of an ar model should fit any repeating structure in the data.
• randomizedalgs 3y ago
  Consider the imaginary world that the author describes, in which people's estimate of their score is independent of their actual score. Wouldn't it be fair to say that, in this imaginary world, the DK effect is real? The point of the effect is that people who score low tend to overestimate their score and people who score high tend to underestimate. Of course there are lots of rational reasons why this could occur (including the toy example the author gave, where nobody has any good sense of what their score will be), but the phenomenon appears to me to be correct.
  • mrkeen 3y ago
    If it's a statistical illusion, the correlation is still true, it just has no business being studied by psychologists. If I roll a die, and then roll a second die, I might study the behaviour of the second die and wonder why it wants to add up to 7 with the first die. Since they're dice, I can dismiss that as a stupid idea, but if they were people, I could certainly be led astray by psychological theories about them.
  • skrebbel 3y ago
    Woa of course, this is the point. The author's example with random points is bad because you might reasonably expect people to behave differently than uniform random points. It'd be reasonable to expect that people who are good at a thing estimate that they are good at it, and that people who are bad at a thing, estimate that they're bad at it. I mean, my kids love math and always estimate themselves to do well on math tests (and they usually do). They have classmates loudly detest math, estimate they'll do badly, and often do (at least somewhat). Similarly I'm a bad cook and I have no doubt that if I join a cooking contest, I'll get few jury points. The expected data is correlated. So if a study finds that, well actually, the data is not at all that correlated! Lots of people who estimate that they'll do fine actually don't, and equally many people who estimate that they'll do badly, actually do fine (ie it looks like uniform random data), then that's surprising, and that's the D-K effect. Right? I'm no statistician at all so I might be missing something.
• ezekiel68 3y ago
  > However, there is a delightful irony to the circumstances of their blunder. Indeed. And I find the tendency of people in this comment section to defend the flawed theory is further confirmation of another scientific finding: that we decide based on emotion and then justify our decision using rationality.
  • stubish 3y ago
    Even when the article cites the 3 papers it is based on, no refutations of the published science by people who grok it.
• notShabu 3y ago
  every domain of expertise has two "elo" systems, the niche one and the broader one. e.g. you can learn basic juggling in 30 minutes that you are top 10% of your friends/colleagues etc... however within the juggling community itself this is known as the "3 ball cascade" a really simple trick relative to the ones that requires years to master. an outsider may not be able to tell the difference between the 1 year expert and the 10 year master. a lot dunning-kruger can be explained by people in one or the other not understanding the other system
• lopatin 3y ago
  Oh I read about the about the DK effect a while ago. I'm pretty much an expert in Psychology now, AMA.
• eagerpace 3y ago
  Is this the opposite of imposter syndrome?
• markhahn 3y ago
  the numeric experiment does not produce a line identical to what DK report. if DK's line where horizontal at 50%, it would indeed be nothing but autocorrelation.
• dahart 3y ago
  Most people, even here on HN, do not know what the DK effect actually claimed to show. It does not show that confident people are more likely to be incompetent. Their primary result shows a positive correlation between confidence and supposed skill. (What skill, you ask?*) This article suggests DK is even simpler than autocorrelation, that it’s just regression toward the mean. https://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-kruger-effect-is-and-isnt/ https://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-... I don’t know which statistical artifact it is, but I am quite convinced that the so-called DK effect is not demonstrating something interesting about human psychology, I don’t buy that this is a real cognitive bias. I’ve read the paper several times, and the methodology seems to be lacking rigor. They tested a small handful of Cornell undergrads volunteering for extra credit, not a large sample, not the general population, and tested nobody who actually fits the description of ‘incompetent’ in a meaningful way. They primarily measured how people rank each other, not what their absolute skill was - and ranking each other requires speculating on the skills of others. There are obvious bias problems with asking a group of pampered Ivy League kids how well they think they rank. * One of the four “skills” they measured was ability to get a joke - “appreciation of humor” - Huh? This is subjective! The jokes used aren’t given in the paper, either. Another was ‘grammar’ tests.
• TrackerFF 3y ago
  The DK effect has gotten WAY more cred than it should. Today, it is just anoter feel-good piece that people use to justify their feeling that they're (ironically) surrounded by loud idiots.
• austin-cheney 3y ago
  The best way to differentiate DK from autocorrection is motive. Low performance people will focus on motives that reinforce the perception of their competence, for example preferring code style over code delivery because while both may be arguably important one requires less effort and risk to attain. There is research to qualify this out of Stanford. People will shift motives to attain complements and the types of compliments received will dictate the challenges they are willing to accept. When a compliment is specific to an action and measurable people will strive for continuously more challenging tasks to continually receive specific compliments. When compliments are generic and directed to the person they will tend to preference progressively less challenging tasks so that they continue to shine relative to the attempted effort. The differences in behavior produces a natural Dunning-Kruger effect wherein people seeking less qualified activities are more likely to over estimate their potential and degree of success. This also statistically verified in research that correlates predictions to confidence. The more confidence a person is in their predictions, such as political talk radio hosts, the less accurate their predictions tend to be.
• James_K 3y ago
  I think the issue here is a confusion about what "bias" means. If they are self-assessing at random, then the high performers will all underestimate themselves, but this is not a bias towards underestimation as they are choosing randomly. That said, the chart from D-K seems to show a different bias and line up roughly with what you would expect. Someone with no knowledge assumes they are average skill and hence inflates their position, someone who is very good doesn't want to rate themselves the best because they assume others know as much as they do. The assumption underlying both groups is that you are normal and others are similar to you. I hypothesise that most people think they're average, which is something you could easily test by asking them to rate how well they think the average person would do on a test and comparing it to that individual's test score. I'm almost certain that high performers will overestimate the average, and low performers underestimate it.
• riazrizvi 3y ago
  A general problem with Dunning Kruger is the assumption that if you score low on a test then you are bad at the subject it is evaluating. I’ve taken enough bad quizzes that purportedly evaluate skills that I am an expert in, to know that that is a leap.
• nitwit005 3y ago
  If self evaluations are random, and you group a bunch of them together, then you'll see values around the 50th percentile. That's why their self evaluation line is nearly flat. In the actual data though, the line clearly trends upward. The people who did well appear to be scoring themselves non-randomly.
• resource0x 3y ago
  Can someone explain the difference between Dunning-Kruger effect and "illusory superiority" effect (https://en.wikipedia.org/wiki/Illusory_superiority https://en.wikipedia.org/wiki/Illusory_superiority)?
  • zeroonetwothree 3y ago
    DK says that skilled people tend to underestimate their skill while unskilled people tend to overestimate their skill. This is likely a statistical artifact. IS says that people tend to overestimate their own skill compared to how other people estimate their skill. This seems likely true on average but not necessarily in all cases.
    • deleted 3y ago
      [deleted]
• fnord77 3y ago
  wikipedia's article intro on this doesn't state it is invalid :/
• jongjong 3y ago
  This makes sense. IMO, the reason why Dunning-Kruger effect is so popular among the upper classes (along with Impostor Syndrome) is that it helps to provide justification for social inequalities as it corrects inner monologues. "How come I have so much given that I'm not as skilled as these other people? I must suffer from impostor syndrome." "Look at all these people complaining instead of taking responsibility for their own failures, they probably suffer from Dunning-Kruger effect. Their work must not be good enough." But of course this requires a certain detachment from reality (hence why many upper class people have blind spots). If they actually took a look at the evidence, they may find that some of these 'Dunning-Kruger people' are actually far more skilled than they imagine. I think it explains why people like Jürgen Schmidhuber who made significant contributions to AI tend to be ignored. Then because people are ignoring them, they are compelled to promote themselves harder to try to get their fair share of attention but they are then put in the 'Dunning-Kruger basket' until someone with a very good reputation like Elon Musk comes along and gives them credit. I think the same could be said about the mathematician Srinivasa Ramanujan; many mathematicians ignored his work or assumed he was a fraud because he seemed too sure of himself for someone who was completely unknown at the time. If such gross injustice can happen in a perfectly-quantifiable field like math, you can be sure it can happen in any field.
• deleted 3y ago
  [deleted]
• _8j50 3y ago
  In my experience, people abuse flattery too much so it is hard to tell if their positive opinions of me are genuine and with merit. Generally speaking, I try to see the big picture and realize no matter how well I do, in a more global sense at best I am too 50th percentile, slightly above average. It is chance,relationships and supply/demand economics that ultimately decide our ability to apply our talents effectively. When it comes to others, I wish more people experienced the D&R effect. It gets frustrating sometimes dealing with smart and talented people who think they are revolutionary rockstars. You know the kind, they see other people's work and they are shocked how bad everything is, but never fear, they, our heroes are here to refactor everything until they leave and another hero looks at their work and rescues metropolis from it again. Patience and humility are a rare virtue for all of us.
• golol 3y ago
  I disagree. Dunning Kruger is not a statement about predicted score correlating with actual score in some way. It states that predicted score does not correlate well with actual score. This can be rephrased as the prediction error having a negative correlation with the actual score. The article then claims that this negative correlation is autocorrelation. That is true but the correlation still exist. The thing is that ideally we EXPECT there to be no correlation of the prediction error with the actual score, but we find autocorrelation. Going back to variables where this autocorrelation is not there, we EXPECTED to find a 1:1 positive correlation between predicted score and actual score but find no correlation, or a weak correlation. So finding autocorrelation when you expected to find no correlation is pretty much the Dunning-Kruger effect here. In fact their example with the random data totally makes sense: Suppose people uniformly randomly estimate their performance. Then the people who are low skilled will consistently over-estimate and the people who are high-skilled will consistently underestimate. Of course there is no causation here, as the people choose randomly, but there is an undeniable correlation. I guess the question is if you view the Dunning-Kruger effect as a claim to low skill CAUSING positive prediction error, or just correlating with it.
• lifeisstillgood 3y ago
  The Dunning Kruger effect is simply the same reason expensive projects are undertaken and never hit budget - not because we cannot estimate costs but because if we did we would never do anything.
• CalChris 3y ago
  The article's definition of autocorrelation: Autocorrelation occurs when you correlate a variable with itself. Wikipedia's definition of autocorrelation: Autocorrelation, sometimes known as serial correlation in the discrete time case, is the correlation of a signal with a delayed copy of itself as a function of delay. Of course, 0 delay is the trivial case of time delay but really, the article's definition is at best inaccurate. D-K has nothing to do with time delay and calling it autocorrelation seems like a weird pun that doesn't quite land.
  • walleeee 3y ago
    To be fair, there is such a thing as spatial autocorrelation (from geostatistics), the term autocorrelation does not necessarily imply the varying dimension is time
• hyperthesis 3y ago
  If unskilled and skilled self-assessed themselves the same on average, then unskilled overestimate, and skilled underestimate. That would be a significant result alone - that no one had any idea. (but as https://news.ycombinator.com/item?id=38416100 https://news.ycombinator.com/item?id=38416100 notes, there is a correlation).
• chmod600 3y ago
  A related effect that I've wondered about is: perhaps lower-skilled people compare themselves to the general public, while perhaps skilled people compare themselves to a smaller group of skilled peers. In other words, if you asked me if I'm good at riding a bicycle, I'd compare myself to others in the general population and say "yes". But if you ask a weekend bicyclist, they'd be better than me but perhaps compare themselves to weekend bicyclists, and rate themselves lower. And the effect might repeat for competitive bicyclists. If true, this could explain why we intuitively believe the DK effect.
• RevEng 3y ago
  What Blair Fix's article gets wrong is that there are two stark differences between what Fix generated with random data and what Dunning and Kruger observed in theirs. Fix has each person guess randomly between 0 and 99 where they will lie in the percentiles. They simulate every person having no idea and giving equal probability to being the best or the worst. If we then sort them by how well they really did into quartiles and then evaluate the average of how well they thought they would do, we get what we would expect: each quartile has an equal chance of predicting that they will do well or do poorly, with an average expected percentile of 50, which is what you would expect by a random guess. Note two key things about this: - All quartiles guessed the same - there was no correlation between what they guessed and how well they actually did - All quartiles guessed the expected average percentile - 50%. This means they were unbiased in how well they thought they would do. If people were unbiased but also unaware, this is the null hypothesis we would expect: on average people predict themselves to be average and there's no correlation between how well they predicted they would do and how well they actually did. Now compare that to what Dunning and Kruger observed: - The quartiles did NOT guess the same. There was a bit of an upwards trend, which suggests that people at least somewhat were able to determine their actual percentiles, even if only weakly on average. - The predictions were biased. All groups estimated they would do better than the expected average. That is to say, on average, they thought they were above average. This is an important bias. - The differentials between quartiles are not equal. The first and second quartile typically predicted the same, over-estimated value, implying that neither group had any idea they were better or worse than each other. However, the upper quartile consistently estimates a higher average. That is to say, people who perform well, on average, believe they are performing even better than those who don't perform well. And perhaps most surprisingly, there was often a statistically significant dip at the third quantile. Comparing their beliefs, people who did well believed they had done worse than the people who actually did worse. Fix also fails to go beyond the first figure of the paper. After seeing this inconsistent behaviour between the quartiles, Dunning and Kruger then test what happens if the respondents are given an opportunity to grade each other - therefore getting an idea of what the percentiles actually look like - and to have their skills improved - thereby possibly making them better able to judge their own and each other's abilities. Again, if Fix's premise that this is all just a result of manipulating the autocorrelation of an otherwise unbiased random sequence, then these interventions should have no discernable effect. Yet, Dunning and Kruger find markedly significant changes after these interventions, and those changes are different within the different quantiles. It is precisely this difference between quantiles which is the Dunning-Kruger effect. Fix effectively makes their point for them by building a null model and showing what would happen if there were no Dunning-Kruger effect - if people were fully unaware and unbiased. Instead, it is the way in which Dunning and Kruger's observations deviate from this model that is the very effect that bears their name. Instead, all that Fix manages to do is point out how confusing the plot is that Dunning and Kruger produced. The plot can easily be misinterpreted to suggest that it's the difference between y and y-x that is important. Instead, in their writing, Dunning and Kruger actually focus on the differences in how y-x changes when the situation changes, demonstrating that it's actually dependent on knowledge and how different people respond to that knowledge. What they actually show is that delta(y-x) vs x has a nonzero relationship and this is particularly interesting. Perhaps if Dunning and Kruger had not included the example of perfect knowledge as a comparison, but instead included the example of unbiased and unknowledgeable that Fix produced as the thing to compare against, the Dunning-Kruger effect would be much better understood. Further, both could benefit greatly from plotting and tabulating not just an average, but the overall distribution within each group. Fix should know that variance is just as important as bias. Even if all groups are biased in their prediction, differences in variance between each group indicates their confidence in their belief. Knowledge should help to reduce both bias and variance. A guess with high variance tells us little, while a guess with low variance tells us quite a bit. Even if all quartiles predicted the same average, we wouldn't fault those with little ability for guessing a high number if they did so with low confidence. On the contrary, we would expect people with high ability to be more confident (and correct) in the assessment of their ability.
• hyperthesis 3y ago
  It's Dunning-Krugers all the way down - including this self-referential smugness.
• epigramx 3y ago
  "Autocorrelation is the statistical equivalent of stating that 5 = 5." no sure if the author has some ..dunning-kruger there.
• 6510 3y ago
  I was curious if the self assessment is done before or after the test. Bing chat gave me this wild answer: > The effect is usually measured by comparing self-assessment with objective performance. For example, participants may take a quiz and estimate their performance afterward, which is then compared to their actual results 1. Therefore, people estimate their ability before the test by Dunning-Kruger. In the case estimation is done before: If you've had training, like a soup of ingredients, that matches the priorities and biases of the test it would be strange if no measurable effect remained. If it's done after: You can create trick questions specifically designed to test if someone learned a specific thing. A good test would test for that. If someone didn't learn the specific thing they could give/guess the wrong answer with some confidence. The design of the test has great influence on how poorly you'll think you've done. I would argue that the superior test is the one designed to fool you. Hans Rosling famously created a multiple choice test with 4 answers per question with average results below 25%. On a more fascinating note, unskilled means all areas of expertise outside your own. People who are universally unskilled in all areas are of course more likely to think they are unskilled. In reality these people know little bits about many things. This in contrast with people who spend all day, every day, for their entire lives pondering topics inside their area of expertise. If you are doing one thing you aren't doing all of the other things. Wikipedia had hilarious instances of experts contributing to countless articles accidentally ending up on the wrong page. Suddenly they have no patience, think they know everything and act like children. It's funny because you cant just ban valuable contributors. I would love to see this DK test done with professors furthest removed from the area of expertise.
• civilized 3y ago
  We discussed this in a previous thread. The author is basically hypothesizing that perhaps people are so universally terrible at predicting their ability, their self-rating is like an unconditional random variable - just a random draw that is not influenced by their actual ability level at all. If this is true, then when your actual ability is high, your self-rating is likely to be lower than your ability simply by random chance. For example, if ability ranges from 0-100, your actual ability is 99, and your self-rating is a uniform random number from 0-100, your self-rating is 99% likely to be lower than your actual ability. Conversely, if your actual ability is low, your self-rating is likely to exceed your actual ability level. When it's explained clearly and simply, the criticism raises a lot of questions. Are people actually that bad at rating their own ability? I doubt it.
• zw123456 3y ago
  I know I'm not smart enough on statistics or psychology to evaluate the article but it always struck me that D&K seemed to say something similar to what my grandpa said when I was a wee lad, "The more you know, the more you realize how much you don't know", I know he wasn't the first person to say that, but he was the first person to say it to me. I don't know if D&K is autocorrelation or not, but I know that an awful lot of people seem to think they know more than maybe they actually do, probably me included. Hmmm, maybe the author of that article as well? I wonder if that occurred to him, seems like a glaring oversight not to at least recognize that possible irony.
  • Arch485 3y ago
    In the article, a real study was used as a counterexample to the DK effect. Part of the results was a correlation that people who were "less capable" were also worse at predicting their own skill, and people who were "more capable" were better at predicting their own skill. While similar to the DK effect, this is different, as the DK effect states that "less capable" individuals specifically _overestimate_ their skill, as opposed to simply being wrong (both over and under -estimating). With relation to some people "seeming to think they know more than they actually know", this is likely confirmation bias in the sense that there are an equal number of people who don't know much, and know that they don't know much.
• dimask 3y ago
  I would call this type of argument a case of regression to the mean rather than "autocorrelation". That, of course, in principle requires independence between performance and assessment of performance. In many cases, it would make little sense to assume that the performance and assessment of performance are independent. But even then, one can simulate random data with some correlation, and still get a DK effect merely as statistical artifact. An overview of similar critiques, and a similar argument in https://www.frontiersin.org/articles/10.3389/fpsyg.2022.840180/full https://www.frontiersin.org/articles/10.3389/fpsyg.2022.8401... .
• 19f191ty 3y ago
  That is not an autocorrelation. The OP is equating linear dependence with autocorrelation, which not how we use that term. Autocorrelation is when a random process is correlated with time lagged version of itself.
• dilawar 3y ago
  David Dunning response (2022): https://www.bps.org.uk/psychologist/dunning-kruger-effect-and-its-discontents https://www.bps.org.uk/psychologist/dunning-kruger-effect-an...
• BrenBarn 3y ago
  Yeah I don't buy this either. I do think the original Dunning-Kruger plot is a bit of an odd presentation. The way I look at it is just to say that people's self-estimates of their ability fall into a relatively narrow range (e.g., 55-75th percentile on the graph), whereas their actual abilities of course cover the whole range from 0-100th percentile. You don't really need the plot of "x versus x" (average score in each quartile). You just need to say "people's self-assessments seem to start unrealistically high and only go up a little, even as their ability goes up a lot".
• PeterStuer 3y ago
  You can take out the x from both sides, and the y would still not be a horizontal line. In their eagerness to 'deconstruct' the narrative, do the authors merely provide another example of Dunning-Kuger by overestimating their own cleverness?
• mattbit 3y ago
  This is not ‘autocorrelation’, it is regression to the mean. I find the article unclear and imprecise. For those interested in a better overview of the Dunning–Kruger effect, I recommend this short article by McIntosh & Della Sala instead: https://www.bps.org.uk/psychologist/persistent-irony-dunning-kruger-effect https://www.bps.org.uk/psychologist/persistent-irony-dunning...
  • mattbit 3y ago
    This is how McIntosh & Della Sala put it: > in the academic literature, it has been suggested that the signature pattern of the DKE (Figure 1A) might be nothing more than a statistical artefact. In a typical study, people’s tendencies to under- or overestimation are analysed as a function of their ability for the task. This involves a ‘double dipping’ into the data because the task performance score is used once to rank people for ability, and then again to determine whether the self-estimate is an under- or over-estimate. This dubious double-dipping makes the analysis prone to a slippery statistical phenomenon called ‘regression to the mean’.
• pmavrodiev 3y ago
  Noone seems to have read OP's post in its entirety. A crucial point was made by referencing this paper: https://digitalcommons.usf.edu/cgi/viewcontent.cgi?article=1215&context=numeracy https://digitalcommons.usf.edu/cgi/viewcontent.cgi?article=1.... Figure 2 in this paper shows the result of an experiment where skill and perception of one's skill are measured independently. To eliminate any statistical artifact of auto-correlation. And lo and behold - on average skill is uncorrelated to the accuracy one's own assessment. No DK effect at all. What does show up actually is that more qualified people are more consistent in estimating their skill (i.e. their assessments are less variable), but the mean accuracy is still 0. So indeed, on average actual and perceived skills are uncorrelated. That's exactly what the numerical proof with random numbers shows and why in many cases we apply Occam's razor.
• psychoslave 3y ago
  I went through the whole article, and I am not only very skeptical about the claimed debunk but wonder what kind of psychological trope you might label as corelative to such an article. I mean "bad science built only on rhetoric" is a double edged sword, you know. To start with, the graph presented at the end does not look like the one from the original article, where the self assessment does grow significantly, though it starts higher than average and grows less quickly than external assessment. Also the article focus on "random" data set which, but we know that there are different classes of apparent noisy plots. Noisy distribution of self assessment would actually be an informative figure too. So the biggest issue here is its kind of pretending that whatever the way the ordinate value is coupled to, if it includes the abscissa in its definition you'll get the same kind of plot as a result, which is obviously false. You could easily come with arbitrary values coupled to "x" that would look radically different.
• rom1v 3y ago
  If Y = X + estimation_error, then substracting X (in Y-X) removes the correlation rather than adding it.
• Spiwux 3y ago
  At the risk of sounding like a complete idiot, isn't the hypothesis of the original paper still true? Let's assume self assessment score is perfectly random between 0% and 100%, so on average every group will always estimate themselves to be 50% correct Then by definition that means people who are unskilled and often incorrect will overestimate themselves, while people who are often correct will underestimate themselves. Take a complete idiot for example. You always get 0% test score. Yet your self-assessment is random between 0% and 100%. Hence you overestimate yourself much more often than people who always get 100% test score. In fact, if the two are uncorrelated, then that still means that 1) Idiots don't recognize they're idiots 2) Skilled people don't recognize they're skilled
• bsza 3y ago
  Article claims Dunning-Kruger is present in a population where everyone estimates their own skills based on dice rolls. Someone who estimates their own skills based on a dice roll is objectively crap at estimating their own skills. Dunning-Kruger claims people are objectively crap at estimating their own skills. Where is the contradiction?
• dudeinjapan 3y ago
  So you're saying that the Dunning-Kruger effect applies to Dunning & Kruger.
• falserum 3y ago
  Article feels like a personal attack towards D and K.
• powera 3y ago
  Nope. I must object to this paragraph: "To be honest, I’m not particularly convinced by the analytic arguments above. It’s only by using real data that I can understand the problem with the Dunning-Kruger effect. So let’s have a look at some real numbers." He then goes on to use synthetic data. Beyond that dishonest slight of hand, this is in the category of "one thought experiment didn't prove the phenomenon exists, therefore it must not exist" logical errors.
• zephrx1111 3y ago
  A more generalizable explanation is regression towards the mean: everybody thinks they are an average person.
• beinvisible 3y ago
  How does the author miss the fact that in his graph the self-estimation hovers around the 50th percentile (as expected with random data), whereas in the DK graph it averages around the 60th to 75th percentile? That is a significant bias. As a control, the author should have plotted Fig.8, but based on DK's data (or at least estimate how Fig. 8 would look like based on the result in Fig. 3). Then it would have been obvious that the self-estimation error tends more towards overestimation in the lower quartiles than ist does towards underestimation in the higher quartiles - which is exactly what DK's conclusion was. In essence: this article actually confirms that DK is not just a random statistical artifact.
• gsuuon 3y ago
  I think the take-away from Dunning-Kruger is that most people think they're closer to average than they are, on both sides.