4 ms·
I'm definitely not even close to a statistician, but I'm also having a hard time accepting this analysis. I'll admit that part of it also comes from personal e
by IceDane 4y ago
I'm definitely not even close to a statistician, but I'm also having a hard time accepting this analysis.
I'll admit that part of it also comes from personal experience, at work and elsewhere. I've met some catastrophically incompetent people were completely oblivious to their own incompetence, and this has very often felt like that the more incompetent they were, the more likely they were to be try to do stuff that was waaaay out of their comfort zone, which would make even experienced, competent people tread carefully.
But even ignoring personal experiences, I'm not convinced by the arguments either. I understand what they are saying, but I don't see how this disproves the DK effect.
Even if everyone is equally bad at estimating their own skill, so that their estimate is essentially a completely random variable, then we would expect the self-assessment score average to be around 50. If I understand it correctly, this is essentially what figure 9 is demonstrating.
But that figure still says that worse performers are then likely to overestimate their own ability, just as much as it says that better performers are bad at it.
If we look at the original DK figure and contrast it with figure 9 with random data, then I think one way of interpreting the differences is that, yes, worse performers are indeed bad at self-assessment, but they're just kind of bad at it as if their self-assessment is a completely random variable. It then seems to keep being essentially random but as people's skills improve, the distance between their score and their self-assessment becomes a bit tighter.. so in conclusion: most people are pretty bad at self-assessment, but skilled people are a bit less so.
The end result is still that people in the bottom quartiles are going to over-estimate their own ability.
I don't know, maybe this is way out in the weeds. Please school me.
- js8 4y agoYou might be biased towards avoiding catastrophic risk. Somebody who doesn't know what they are doing without knowing it is more dangerous than somebody who knows what they are doing yet taking precautions as if they don't.
- andersource 4y agoThis is my understanding as well.
- bouncycastle 4y agoThe problem is that they calculated each person’s ‘self-assessment error’ with the actual test score. This error is the difference between a person’s self assessment and their test score, This is like comparing x - y to x, and if you do this, you will get a correlation no matter what.
- andersource 4y agoNot necessarily. If for example y =~ x, comparing x - y to x would yield zero correlation, which to me is what DK is all about - people _aren't_ as good as we would expect at estimating their own skill.
- bouncycastle 4y agoThey are not saying that the hypothesis is wrong, just that the proof of the hypothesis is wrong. You can frame it however you like, but it won't work if you have x on both sides of the equation.
- motoboi 4y agoThe feeling your got from looking at the graph come from the idea that the red line represent absolute values. It’s actually an average of the real value as that is a really big difference and of the reasons why it’s very easy to lie with statistics. To correct correlate the two variables (assessment and actual-score) you need to correlate the actual data, not measures of its caracteristics (average being one of them). The actual data is shown. Even by eye it’s possible to see no strong (or significant) correlation exists.
- denton-scratch 4y ago/me not a statistician either. If people are all pretty-much crap at estimating their own skill, then you'd expect all estimates to be roughly their actual skill, plus-or-minus some random error-margin with some kind of probabilistic distribution (Gaussian?). If that were the case, then high-skilled people would be more likely to underestimate their skill (because their actual skill is greater than the mean). That (I think) is an example of "reversion to the mean". If that reasoning is right, then the DK claim may be true, but it says nothing about the comparative estimating propensities of high-skilled and low-skilled people. It just says that low-skilled people tend to overestimate their skill, and vice-versa. But that's exactly what you'd expect, amirite?
- BlueTemplar 4y agoYou seem to be forgetting about under-estimation, so your conclusions don't follow from your premises ? > most people are pretty bad at self-assessment, but skilled people are a bit less so This is pretty much the conclusion of the article, except that it isn't the tautologic DK or figure 9 that shows it, but Nuhfer's figure 11.
- cestith 4y agoI think part of the issue here is you're talking about people estimating future accomplishment where they have no skill, and all variations of these experiments are people assessing how they performed on a test they just took. There's very likely a difference between someone doing open-ended talking wildly in a field where they don't even know what's possible and an introspective assessment of themselves on a concrete task they just completed.
- roguecoder 4y agoAnd where they got feedback. What the original paper presented turns out to be a tautology: in the absence of feedback, people don't know if they performed well or poorly.