4 ms·

You've been corrected about "what DK means" in the other comments, but this is not quite the point of the post. This is not about if DK (as expressed in English

by krick 5y ago

You've been corrected about "what DK means" in the other comments, but this is not quite the point of the post. This is not about if DK (as expressed in English words) is true or not — in fact, author points out in the beginning that it's one of these "everybody knows it's like that" ideas (as it often is with social psychology). The point is, that the original DK paper is bullshit. At least, this plot is. And people tend to miss it, until they start to carefully read the labels and think about the caveats. In fact, as presented here it looks like it shouldn't even be accepted as a valid study, this is outright deceptive, maliciously so. If there is assumed to be a correlation between x & y, how about we start by plotting x against y then? I know, it may be messy. It almost certainly will be. Because of that, I personally won't even be offended (but some people might) by you removing the outliers and producing the unnaturally clean version of the plot in the end to highlight the main idea. Then some statistical tests to make the results quantified. But here we see nothing, it really is just comparing x to x. IMO, this is pretty much the invariant of most of the problems of academic research in the last God-knows-how-many decades (maybe always was, I don't know). Computer science papers without the code. Data science papers without the data. Yeah-yeah, I've heard hundreds of excuses why researchers do it like that. But it's pointless, such "research" shouldn't be accepted by anybody. Either you make your findings actually public by providing everything to replicate every single step of your study (which is supposed to be the point), or you just don't publish anything and keep the research proprietary (I mean, obviously it's never black and white, there always will be concerns about test-subject anonymity, etc. — but it's ridiculous to discuss that when the accepted standard even in "proper" sciences are 20 pages of dense text which might never even get to the point of the study, i.e., actually showing the data to any extent.)

andersource 5y ago
I strongly disagree, not necessarily with everything (e.g. I don't have access to the raw data from the DK experiment, don't know how well they performed all the analysis leading to the plot). But the plot itself is not inherently deceptive, and, unlike implied in the article, is not equivalent to "just comparing x to x". The plot essentially shows the actual performance vs. self-assessment of performance, compared to what we would expect if there were perfect correlation.
- krick 5y ago
  Well, I wouldn't say "equivalent" (and I wouldn't say it's implied in the article), but it isn't very far from that. And meaning of "very far" is relative. I mean, f(x,y) ~ x obviously isn't equivalent to x ~ x, but the importance of pointing this out depends on how likely people are to mistake it for x ~ y, and how likely they are to skip that f() is DEFINED by authors and not found in the nature. Same with "misleading": your perspective on if something is misleading obviously depends on where it leads you. But my evaluation of 2 questions above are "very likely" and "it does lead majority of people to the wrong conclusions", so the whole sentiments stands true. I'll get back to it in a moment. Anyway, my (lesser) point is that a study like that (even in 1999) must contain at least one plot of [actual score] vs [self-assessment score], or it is worthless, and I stand by it. (The original paper doesn't contain such plot.) My more general point is that what is accepted as "sufficient" data provided in overwhelming majority of research is laughable, and I really want people to stop tolerating that. (I don't want to delve into discussing the particular paper, I'm purposefully trying to keep all my point as general as possible. Some problems are actually addressed in DK-1999, but the whole study makes you smirk at a circus social psychology experiments are. Here's a nice quote for you (being tested is "ability to recognize what's funny"): "To assess joke quality, we contacted several professional comedians … and asked them to rate each joke on a scale ranging from 1 (not at all funny) to 11 (very funny). Eight comedians responded to our request …, an analysis of interrater correlations found that one (and only one) comedian's ratings failed to correlate positively with the others (mean r = -.09). We thus excluded this comedian's ratings in our calculation of the humor value of each joke.") Now, back to the f(x, y) ~ x. Let's imagine f(x, y) = C. E.g., every respondent evaluated himself as "average". Is it true that less knowledgeable people turned out to overestimate themselves, and more knowledgeable people turned out to underestimate themselves? Well, yeah, I guess. Does x correlate with y? No. So, what's more valuable here, to plot [x ~ y] or [f(x, y) ~ x]? Moreover, this is obviously different from the case where y (the self-assessment score) is uniformly distributed and doesn't correlate with x, yet it will yield the same line-plot as the above, if we construct them like D&K did. And that's skipping the part that there are several viable ways to define "self-assessment of performance" as f(x,y). I think, if I'll try hard enough I would be able to "prove" almost any hypothesis about self-assessment using this method. Of course, I won't really prove anything, but the plots will be convincing enough, and the "results" will be much more demonstrative than the original data is, same as probably is with DK-1999 (but that's impossible to tell without seeing actual source data of DK-1999). TL;DR: It all boils down to the old adage about "3 kinds of lies". Is the result being discussed an example of auto-correlation? Yes, it is. Is auto-correlation necessarily bad? Not really. Is it important and can it obstruct interpretation in the particular example of DK-1999 paper? It's subjective and hasn't been measured, but my evaluation is — absolutely.

by krick 5y ago

andersource 5y ago

I strongly disagree, not necessarily with everything (e.g. I don't have access to the raw data from the DK experiment, don't know how well they performed all the analysis leading to the plot). But the plot itself is not inherently deceptive, and, unlike implied in the article, is not equivalent to "just comparing x to x". The plot essentially shows the actual performance vs. self-assessment of performance, compared to what we would expect if there were perfect correlation.

krick 5y ago
Well, I wouldn't say "equivalent" (and I wouldn't say it's implied in the article), but it isn't very far from that. And meaning of "very far" is relative. I mean, f(x,y) ~ x obviously isn't equivalent to x ~ x, but the importance of pointing this out depends on how likely people are to mistake it for x ~ y, and how likely they are to skip that f() is DEFINED by authors and not found in the nature. Same with "misleading": your perspective on if something is misleading obviously depends on where it leads you. But my evaluation of 2 questions above are "very likely" and "it does lead majority of people to the wrong conclusions", so the whole sentiments stands true. I'll get back to it in a moment. Anyway, my (lesser) point is that a study like that (even in 1999) must contain at least one plot of [actual score] vs [self-assessment score], or it is worthless, and I stand by it. (The original paper doesn't contain such plot.) My more general point is that what is accepted as "sufficient" data provided in overwhelming majority of research is laughable, and I really want people to stop tolerating that. (I don't want to delve into discussing the particular paper, I'm purposefully trying to keep all my point as general as possible. Some problems are actually addressed in DK-1999, but the whole study makes you smirk at a circus social psychology experiments are. Here's a nice quote for you (being tested is "ability to recognize what's funny"): "To assess joke quality, we contacted several professional comedians … and asked them to rate each joke on a scale ranging from 1 (not at all funny) to 11 (very funny). Eight comedians responded to our request …, an analysis of interrater correlations found that one (and only one) comedian's ratings failed to correlate positively with the others (mean r = -.09). We thus excluded this comedian's ratings in our calculation of the humor value of each joke.") Now, back to the f(x, y) ~ x. Let's imagine f(x, y) = C. E.g., every respondent evaluated himself as "average". Is it true that less knowledgeable people turned out to overestimate themselves, and more knowledgeable people turned out to underestimate themselves? Well, yeah, I guess. Does x correlate with y? No. So, what's more valuable here, to plot [x ~ y] or [f(x, y) ~ x]? Moreover, this is obviously different from the case where y (the self-assessment score) is uniformly distributed and doesn't correlate with x, yet it will yield the same line-plot as the above, if we construct them like D&K did. And that's skipping the part that there are several viable ways to define "self-assessment of performance" as f(x,y). I think, if I'll try hard enough I would be able to "prove" almost any hypothesis about self-assessment using this method. Of course, I won't really prove anything, but the plots will be convincing enough, and the "results" will be much more demonstrative than the original data is, same as probably is with DK-1999 (but that's impossible to tell without seeing actual source data of DK-1999). TL;DR: It all boils down to the old adage about "3 kinds of lies". Is the result being discussed an example of auto-correlation? Yes, it is. Is auto-correlation necessarily bad? Not really. Is it important and can it obstruct interpretation in the particular example of DK-1999 paper? It's subjective and hasn't been measured, but my evaluation is — absolutely.