>> Side-notes: - I looked for the 0.17% number in the paper, and what the paper actually says is that the Pearson correlation coefficient between inter-coder ag

>> Side-notes: - I looked for the 0.17% number in the paper, and what the paper actually says is that the Pearson correlation coefficient between inter-coder agreement and classification accuracy is 0.17 (=positive but weak). This isn't a number comparing the correlation between different coders. (Frankly, I think comparing ChatGPT's self-consistency to consistency between different people is unenlightening) - Section 4 sub-section "Crowd-Workers Annotation" says that each tweet was classified by two different workers, and no single worker classified more than 20% of the dataset. You're right, I got confused. Quoting from the paper: >> The relationship between intercoder agreement and accuracy is positive, but weak (Pearson’s correlation coefficient: 0.17). I thought they used agreement between Mechanical Turks to evaluate their annotations. Thanks for the kind correction. Regarding grad students, well, most grad students at my (UK) university (Imperial College) are mainly not native English speakers. But that's besides the point I think. My problem is that they took two grad students and compared them to a number of Mechanical Turks. That's just asking for a difference in disagreement between the two groups of annotators, especially if the grad students could communicate with each other and the Mechanical Turks not (which I suspect was the case). Anyway that's a different criticism than the one in my post above, but I think also useful to keep in mind.