4 ms·

Does this article actually contains explanation of what was wrong with the original study? Maybe I don't know how to read with understanding, but to me this a

Does this article actually contains explanation of what was wrong with the original study? Maybe I don't know how to read with understanding, but to me this article makes only two points: 1. The effect was about all people, not just dumb people. 2. The same graph can be obtained "by random". Point 1. makes sense, but 2. is left unexplained. There is a lot of words, but no content. Almost as if GPT-3 was writing it :) .

Will_Do 6y ago
Yea I agree. It all relies on (2) but they do a very poor job explaining what exactly the simulation is. If there is a paper by McNight, it isn't cited and I can't find it. Can't find it on his webpage[0][1] or google scholar. This is a non-trivial simulation so the details are really important! Shame they are nowhere to be found. [0]: https://ibi.gmu.edu/faculty-directory/patrick-mcknight/ https://ibi.gmu.edu/faculty-directory/patrick-mcknight/ [1]:https://scholar.google.com/citations?hl=en&user=sH44LC4AAAAJ&view_op=list_works&sortby=pubdate https://scholar.google.com/citations?hl=en&user=sH44LC4AAAAJ...
SoSoRoCoCo 6y ago
Exactly! Why does the author not tell us how the McKnight model works? This is the only substantial claim the author makes, yet the result is delivered in two measly sentences. I feel snookered.
- thethirdone 6y ago
  The other paper by Ackermann also has a simulated graph, but I have yet to find the simulation model.
lrossi 6y ago
On top of that, the graphs are only barely similar, and I would not call them the same. There is a clear difference in magnitude.
deleted 6y ago
[deleted]
jonnycomputer 6y ago
The article gives us a few clues: 1. the DK-effect supposedly increases the noisier subjects estimates of their abilities are, 2. the graph relies on breaking the data into quartiles 3. experts are less noisy in their estimates of their abilities than are non-experts. There is also the paper: https://scholarcommons.usf.edu/cgi/viewcontent.cgi?article=1215&context=numeracy https://scholarcommons.usf.edu/cgi/viewcontent.cgi?article=1... Still I think the article does a disservice in not laying out the argument in plain language. I don't particularly feel like puzzling it out
glangdale 6y ago
Point 2 also had me scratching my head. The supposed evidence for it (that graph) is statistical mystery meat. If the two metrics (confidence and ability) were independent in a simulation, you'd expect to see a more-or-less flat line on confidence if you sorted by ability. So the model clearly insists on some sort of association between them, no? If the association is weak, you get a cross-over. Big deal. Also, the graphs don't really look all that similar - especially when you consider the fact that the line on which the associated variable have been sorted by quantile is always going to be ascending (duh). Tufte says "to clarify, add data" - one thing that neither the chart from the original study nor the simulated one respects. Surely we'd learn more from also being able to eyeball a scatterplot.
masswerk 6y ago
As I understand it, the original study suggested a bias, which causes us to assume that our abilities are closer to the mean distribution than they actually are. The example about the language test misses out on this entirely. (Regarding that example, a person may even withstand the confrontation with the test score, like, "I guess, even a native speaker doesn't get more than 65% on average", or, "99% is surely not that unusual, everybody with a decent understanding should get a score like this".)

Will_Do 6y ago

Yea I agree. It all relies on (2) but they do a very poor job explaining what exactly the simulation is. If there is a paper by McNight, it isn't cited and I can't find it. Can't find it on his webpage[0][1] or google scholar. This is a non-trivial simulation so the details are really important! Shame they are nowhere to be found. [0]: https://ibi.gmu.edu/faculty-directory/patrick-mcknight/ https://ibi.gmu.edu/faculty-directory/patrick-mcknight/ [1]:https://scholar.google.com/citations?hl=en&user=sH44LC4AAAAJ&view_op=list_works&sortby=pubdate https://scholar.google.com/citations?hl=en&user=sH44LC4AAAAJ...

SoSoRoCoCo 6y ago

Exactly! Why does the author not tell us how the McKnight model works? This is the only substantial claim the author makes, yet the result is delivered in two measly sentences. I feel snookered.

thethirdone 6y ago
The other paper by Ackermann also has a simulated graph, but I have yet to find the simulation model.

lrossi 6y ago

On top of that, the graphs are only barely similar, and I would not call them the same. There is a clear difference in magnitude.

deleted 6y ago

[deleted]

jonnycomputer 6y ago

The article gives us a few clues: 1. the DK-effect supposedly increases the noisier subjects estimates of their abilities are, 2. the graph relies on breaking the data into quartiles 3. experts are less noisy in their estimates of their abilities than are non-experts. There is also the paper: https://scholarcommons.usf.edu/cgi/viewcontent.cgi?article=1215&context=numeracy https://scholarcommons.usf.edu/cgi/viewcontent.cgi?article=1... Still I think the article does a disservice in not laying out the argument in plain language. I don't particularly feel like puzzling it out

glangdale 6y ago

Point 2 also had me scratching my head. The supposed evidence for it (that graph) is statistical mystery meat. If the two metrics (confidence and ability) were independent in a simulation, you'd expect to see a more-or-less flat line on confidence if you sorted by ability. So the model clearly insists on some sort of association between them, no? If the association is weak, you get a cross-over. Big deal. Also, the graphs don't really look all that similar - especially when you consider the fact that the line on which the associated variable have been sorted by quantile is always going to be ascending (duh). Tufte says "to clarify, add data" - one thing that neither the chart from the original study nor the simulated one respects. Surely we'd learn more from also being able to eyeball a scatterplot.

masswerk 6y ago

As I understand it, the original study suggested a bias, which causes us to assume that our abilities are closer to the mean distribution than they actually are. The example about the language test misses out on this entirely. (Regarding that example, a person may even withstand the confrontation with the test score, like, "I guess, even a native speaker doesn't get more than 65% on average", or, "99% is surely not that unusual, everybody with a decent understanding should get a score like this".)