3 ms·

The comparison in the paper is who approximates trained annotators better, MTurk or ChatGPT. Trained annotators are the gold standard.

YeGoblynQueenne 4y ago
The "gold standard" in the article is two graduate students. The Mechanical Turks are probably more- I can't find the number in the paper. Given the "inter-coder agreement" is low for the Mechanical Turks (0.17% Pearson cor. coeff) it is no surprise that the "gold standard" and the Mechanical Turks' decisions diverge. So the comparison is pointless.
- shenberg 4y ago
  It's the gold standard of text labeling to use grad students to label data as a gold standard, and it really depends on the difficulty of the task whether this makes sense. For example, if your labeling task is to, say, transcribe a bunch of recordings of spoken English in American English, maybe a grad student is a nice short-hand for "native American English speaker who is reasonably literate" which is a good baseline, compared to MTurk where many English speakers come from countries which speak English with different accents and idioms, maybe as a second language as well, and so I would expect them to have a harder time, and their computer setup is likely worse, and their pay isn't as high, etc. It seems to me that, A, at least for some tasks, grad-students-as-gold-standard is not wacky, and B, grad students won't necessarily have the same output as MTurk workers. Given these two points, it's perfectly reasonable to ask how consistent-with-grad-students MTurk workers are, seeing as the reason people use MTurk in similar scenarios is as a cheaper alternative to grad students. Re: inter-coder agreement, I think that the low inter-coder agreement for MTurk is in itself potentially a surprising aspect. Perhaps the explanations that worked well for grad students (and ChatGPT) didn't explain the task properly to MTurks. That's pretty much a point in favor of ChatGPT, though maybe it can be viewed as a point against the actual tasks they use (ie if the explanation couldn't get people of a likely more dissimilar background to output the same results as grad students then maybe the task is less objective than deemed by the task's authors). Side-notes: - I looked for the 0.17% number in the paper, and what the paper actually says is that the Pearson correlation coefficient between inter-coder agreement and classification accuracy is 0.17 (=positive but weak). This isn't a number comparing the correlation between different coders. (Frankly, I think comparing ChatGPT's self-consistency to consistency between different people is unenlightening) - Section 4 sub-section "Crowd-Workers Annotation" says that each tweet was classified by two different workers, and no single worker classified more than 20% of the dataset.
  - YeGoblynQueenne 4y ago
    >> Side-notes: - I looked for the 0.17% number in the paper, and what the paper actually says is that the Pearson correlation coefficient between inter-coder agreement and classification accuracy is 0.17 (=positive but weak). This isn't a number comparing the correlation between different coders. (Frankly, I think comparing ChatGPT's self-consistency to consistency between different people is unenlightening) - Section 4 sub-section "Crowd-Workers Annotation" says that each tweet was classified by two different workers, and no single worker classified more than 20% of the dataset. You're right, I got confused. Quoting from the paper: >> The relationship between intercoder agreement and accuracy is positive, but weak (Pearson’s correlation coefficient: 0.17). I thought they used agreement between Mechanical Turks to evaluate their annotations. Thanks for the kind correction. Regarding grad students, well, most grad students at my (UK) university (Imperial College) are mainly not native English speakers. But that's besides the point I think. My problem is that they took two grad students and compared them to a number of Mechanical Turks. That's just asking for a difference in disagreement between the two groups of annotators, especially if the grad students could communicate with each other and the Mechanical Turks not (which I suspect was the case). Anyway that's a different criticism than the one in my post above, but I think also useful to keep in mind.
cowl 4y ago
The comparison is useless because it is not considering Motivation. MTurk economy values volume at the expense of accuracy. The economic claim is nothing new and nothing unexpected. The computers, AI or not are faster/cheaper than humans at any well-defined task. Emphasis on the well-defined.
- shenberg 4y ago
  They showed that, at least for their tasks, their definition of the task was well-defined-enough for ChatGPT. That's exactly why the comparison is useful. MTurk is often used in these tasks in place of more expensive human annotators (e.g. grad students), and this paper says that for their case, at least, ChatGPT worked better, in the sense that, given the exact same instructions, it gave answers closer to the more-expensive annotators. Using MTurk seriously often entails extra steps intended to verify motivation, e.g. adding a question that says "select option 7" to make sure the person isn't just making random choices, or gathering more answers for questions where there was disagreement between annotators. What these extra steps have in common is that they take both more time when designing the labeling process, and cost more money.

YeGoblynQueenne 4y ago

The "gold standard" in the article is two graduate students. The Mechanical Turks are probably more- I can't find the number in the paper. Given the "inter-coder agreement" is low for the Mechanical Turks (0.17% Pearson cor. coeff) it is no surprise that the "gold standard" and the Mechanical Turks' decisions diverge. So the comparison is pointless.

shenberg 4y ago
It's the gold standard of text labeling to use grad students to label data as a gold standard, and it really depends on the difficulty of the task whether this makes sense. For example, if your labeling task is to, say, transcribe a bunch of recordings of spoken English in American English, maybe a grad student is a nice short-hand for "native American English speaker who is reasonably literate" which is a good baseline, compared to MTurk where many English speakers come from countries which speak English with different accents and idioms, maybe as a second language as well, and so I would expect them to have a harder time, and their computer setup is likely worse, and their pay isn't as high, etc. It seems to me that, A, at least for some tasks, grad-students-as-gold-standard is not wacky, and B, grad students won't necessarily have the same output as MTurk workers. Given these two points, it's perfectly reasonable to ask how consistent-with-grad-students MTurk workers are, seeing as the reason people use MTurk in similar scenarios is as a cheaper alternative to grad students. Re: inter-coder agreement, I think that the low inter-coder agreement for MTurk is in itself potentially a surprising aspect. Perhaps the explanations that worked well for grad students (and ChatGPT) didn't explain the task properly to MTurks. That's pretty much a point in favor of ChatGPT, though maybe it can be viewed as a point against the actual tasks they use (ie if the explanation couldn't get people of a likely more dissimilar background to output the same results as grad students then maybe the task is less objective than deemed by the task's authors). Side-notes: - I looked for the 0.17% number in the paper, and what the paper actually says is that the Pearson correlation coefficient between inter-coder agreement and classification accuracy is 0.17 (=positive but weak). This isn't a number comparing the correlation between different coders. (Frankly, I think comparing ChatGPT's self-consistency to consistency between different people is unenlightening) - Section 4 sub-section "Crowd-Workers Annotation" says that each tweet was classified by two different workers, and no single worker classified more than 20% of the dataset.
- YeGoblynQueenne 4y ago
  >> Side-notes: - I looked for the 0.17% number in the paper, and what the paper actually says is that the Pearson correlation coefficient between inter-coder agreement and classification accuracy is 0.17 (=positive but weak). This isn't a number comparing the correlation between different coders. (Frankly, I think comparing ChatGPT's self-consistency to consistency between different people is unenlightening) - Section 4 sub-section "Crowd-Workers Annotation" says that each tweet was classified by two different workers, and no single worker classified more than 20% of the dataset. You're right, I got confused. Quoting from the paper: >> The relationship between intercoder agreement and accuracy is positive, but weak (Pearson’s correlation coefficient: 0.17). I thought they used agreement between Mechanical Turks to evaluate their annotations. Thanks for the kind correction. Regarding grad students, well, most grad students at my (UK) university (Imperial College) are mainly not native English speakers. But that's besides the point I think. My problem is that they took two grad students and compared them to a number of Mechanical Turks. That's just asking for a difference in disagreement between the two groups of annotators, especially if the grad students could communicate with each other and the Mechanical Turks not (which I suspect was the case). Anyway that's a different criticism than the one in my post above, but I think also useful to keep in mind.

cowl 4y ago

The comparison is useless because it is not considering Motivation. MTurk economy values volume at the expense of accuracy. The economic claim is nothing new and nothing unexpected. The computers, AI or not are faster/cheaper than humans at any well-defined task. Emphasis on the well-defined.

shenberg 4y ago
They showed that, at least for their tasks, their definition of the task was well-defined-enough for ChatGPT. That's exactly why the comparison is useful. MTurk is often used in these tasks in place of more expensive human annotators (e.g. grad students), and this paper says that for their case, at least, ChatGPT worked better, in the sense that, given the exact same instructions, it gave answers closer to the more-expensive annotators. Using MTurk seriously often entails extra steps intended to verify motivation, e.g. adding a question that says "select option 7" to make sure the person isn't just making random choices, or gathering more answers for questions where there was disagreement between annotators. What these extra steps have in common is that they take both more time when designing the labeling process, and cost more money.