having also not read the original paper recently, I'm in the camp that says the question is interesting and valid, but you've got to be careful about interpreta

having also not read the original paper recently, I'm in the camp that says the question is interesting and valid, but you've got to be careful about interpretation. how much the scores correlate (or autocorrelate) with self assessment IS an interesting question, and it's not apparent a priori what the answer will be, or whether the real effect we're interested in will differ for different levels of expertise. but the use of percentile to percentile (or quartiles, but those are just grouped percentiles) to give the impression of a particular kind of effect (lower groups overestimating and higher groups underestimating) is a flaw i think. it's a common one, in my experience, when dealing with percentages. if you think about it, percentages/percentiles have to be bounded at 0 or 100. for a dunning kruger effect not to appear, participants at both ends of the ability spectrum would have to be eerily accurate in their self-assessments. if they aren't, there's just more space on one side of the measurement scale for each group to make an error (if you score 1 in ability, there's ~99 percentiles available for you to make an overestimate and only 1 to be accurate, ditto for those with high ability. those in the middle of the ability group have equal chance on either side and so appear statistically more accurate even with a purely random distribution of guesses). so if there is any measure of central tendancy towards the middle percentile in the estimations of peoples abilities at all (and i would argue there is a priori reason to believe there would be, as the alternative would require those at both ends of the distribution to be getting increasingly accurate, which would be really weird), then practically any real world graph of percentile performance to percentile estimation will show a dunning kruger effect (with lower ends overestimating and higher ends underestimating). the article does a good job of showing this by plotting just random estimations and observing one appears.