5 ms·

Suppose you have two classifiers, A and B, and some un-annotated data, D. You want to know how good is classifier B at annotating the data, compared to classifi

Suppose you have two classifiers, A and B, and some un-annotated data, D. You want to know how good is classifier B at annotating the data, compared to classifier A. One problem is that you don't have the ground truth for D. So you start by annotating D with the labels assinged by a third classifier, C: C(D) → D₁ Having thus established a modicum of "ground truth", ish, you proceed to annotate D with the two classifiers you are comparing, A and B: A(D) → D₂ B(D) → D₃ Then you compare the classifications D₂ and D₃ to D₁, and find that D₃ better approximates D₁ than D₂. What is the result of the experiment? I summarise it as follows: Classifier B better approximates the labelling of D by classifier C than classifier A. Now we can name the three classifiers as they were used in the experiments in the linked article: A: Human annotators. B: ChatGPT C: Human annotators. So the result of the paper is that, plugging in the names: ChatGPT better approximates the labelling of D by Human annotators than Human annotators. And that, is the finding of the paper. Which is clearly absurd and a cause to re-think methodology.

nmca 4y ago
Ah yes, ye olde "all humans are equally capable of all tasks" axiom (the paper is correct and the parent comment is wrong, perhaps obviously)
- YeGoblynQueenne 4y ago
  To be clear, that's a different criticism of the article's methodology than mine, yes? If you assume that the two sets of human annotators are different, then what are you comparing, exactly? The ability of one group to second-guess the other? There are other issues if you choose to assume that the two groups are fundamentally dissimilar: one group was two grad students, the other a number of Mechanical Turks. You can expect there to be more disagreement between (more than two? I'm not sure) Mechanical Turks than between two grad students (both in political science). Ultimately, my problem with the study is that the labelling they took as ground truth (i.e. that they compared ChatGPT and Mechanical Turks against) is too uncertain, or even subjective, to know for sure what exactly they found out. Edit: Oh, wait, I didn't put that in the comment above. Damn. I thought I had.
  - nmca 4y ago
    In the framing of your original comment: A: Mturkers B: ChatGPT C: Experts => ChatGPT better approximates the labelling of D by experts than mTurkers Which is a coherent and interesting conclusion. Edit: also, please forgive the snark in my first response
    - YeGoblynQueenne 4y ago
      >> Edit: also, please forgive the snark in my first response No need to apologise! Your snark wasn't overboard, I thought. Anyway, big girl, can take it :) >> ChatGPT better approximates the labelling of D by experts than mTurkers That could be a "coherent and interesting conclusion" but it's not what the article really claims. The article's title is I think hedging its bets, by being very precise about who, exactly, was outperformed by ChatGPT, although it still manages to be vague about how ChatGPT outperformed the Mechanical Turks. I'm also really doubtful that two political science graduate students can be considered as "experts" in the annotation tasks they were called to perform, which were, again if I got that right, about content moderation. "Experts" in this setting would be people with experience in moderating discussion boards etc. I don't see that this was the case with the two students that provided the initial annotation.
malshe 4y ago
Your comment is based on a very strong assumption that all human annotators are alike in their motivations and abilities to do the tasks. As someone who has used mTurk in the past quite a lot, I think this assumption is wrong. That was the reason I stopped using mTurk. On a separate note, how do you type those arrows and subscripts in an HN comment?
- Minor49er 4y ago
  This is true. I set up a data validation project with mTurk in the past to validate scraped data by presenting a simply survey of the results. Basically "go to this webpage. The title is X (True or False). The description is Y (True or False)" etc. There were several users who would speedrun the surveys which created a lot of false positives mTurk has some ways to make results more accurate, such as being able to programmatically tank users who have given bad results and making it easy to give bonuses to users who give good results. But those have to be carefully designed within the application
  - malshe 4y ago
    Yes, that's a lot of work. I remember in one image annotation task we got back what looked like a uniform distribution of responses. This was a pilot to determine the quality of responses so we did not waste a lot of money on it. The more limits we use, the smaller is the pool of mTurkers making it impractical for large tasks.
- YeGoblynQueenne 4y ago
  >> On a separate note, how do you type those arrows and subscripts in an HN comment? I type in gvim. It lets you type "digraphs" with Ctrl + K and another couple of keys. For the right-arrow it's (in insert mode and without typing spaces): Ctrl K - > In gvim, you can see all the digraphs with :digraph (in command mode). The digraph you want to type has to be supported by the font you use of course. gvim shows you the available digraphs in a scratch buffer that's not terribly easy to search (there's lots of them) but you can yank it to a register and paste it into a new buffer. I don't remember how I do that, I'd have to look up my vim notes :) >> Your comment is based on a very strong assumption that all human annotators are alike in their motivations and abilities to do the tasks. Yes and no. There's a few more threads here that discuss this as if ChatGPT beats all humans in annotating data, more or less. I think that's the general assumption anyway. Also, the way the study was carried out it seems to me that the authors themselves made this assumption, that one group of human annotators is like the other, and you can measure the difference between them as if one was the ground truth and the other just trying to approximate it, where the reality is that they are probably both trying to approximate some completely subjective measure of goodness.
sebzim4500 4y ago
I think your assumption that all humans are equally capable should be reconsidered.
- YeGoblynQueenne 4y ago
  That seems to be the authors' assumption.
  - amluto 4y ago
    I didn’t check the paper’s methodology, but if humans and ChatGPT classify the same on average but ChatGPT is more consistent, then humans will match ChatGPT better than they match other humans.
  - shenberg 4y ago
    The comparison in the paper is who approximates trained annotators better, MTurk or ChatGPT. Trained annotators are the gold standard.
    - YeGoblynQueenne 4y ago
      The "gold standard" in the article is two graduate students. The Mechanical Turks are probably more- I can't find the number in the paper. Given the "inter-coder agreement" is low for the Mechanical Turks (0.17% Pearson cor. coeff) it is no surprise that the "gold standard" and the Mechanical Turks' decisions diverge. So the comparison is pointless.
      - shenberg 4y ago
        It's the gold standard of text labeling to use grad students to label data as a gold standard, and it really depends on the difficulty of the task whether this makes sense. For example, if your labeling task is to, say, transcribe a bunch of recordings of spoken English in American English, maybe a grad student is a nice short-hand for "native American English speaker who is reasonably literate" which is a good baseline, compared to MTurk where many English speakers come from countries which speak English with different accents and idioms, maybe as a second language as well, and so I would expect them to have a harder time, and their computer setup is likely worse, and their pay isn't as high, etc. It seems to me that, A, at least for some tasks, grad-students-as-gold-standard is not wacky, and B, grad students won't necessarily have the same output as MTurk workers. Given these two points, it's perfectly reasonable to ask how consistent-with-grad-students MTurk workers are, seeing as the reason people use MTurk in similar scenarios is as a cheaper alternative to grad students. Re: inter-coder agreement, I think that the low inter-coder agreement for MTurk is in itself potentially a surprising aspect. Perhaps the explanations that worked well for grad students (and ChatGPT) didn't explain the task properly to MTurks. That's pretty much a point in favor of ChatGPT, though maybe it can be viewed as a point against the actual tasks they use (ie if the explanation couldn't get people of a likely more dissimilar background to output the same results as grad students then maybe the task is less objective than deemed by the task's authors). Side-notes: - I looked for the 0.17% number in the paper, and what the paper actually says is that the Pearson correlation coefficient between inter-coder agreement and classification accuracy is 0.17 (=positive but weak). This isn't a number comparing the correlation between different coders. (Frankly, I think comparing ChatGPT's self-consistency to consistency between different people is unenlightening) - Section 4 sub-section "Crowd-Workers Annotation" says that each tweet was classified by two different workers, and no single worker classified more than 20% of the dataset.

nmca 4y ago

Ah yes, ye olde "all humans are equally capable of all tasks" axiom (the paper is correct and the parent comment is wrong, perhaps obviously)

YeGoblynQueenne 4y ago
To be clear, that's a different criticism of the article's methodology than mine, yes? If you assume that the two sets of human annotators are different, then what are you comparing, exactly? The ability of one group to second-guess the other? There are other issues if you choose to assume that the two groups are fundamentally dissimilar: one group was two grad students, the other a number of Mechanical Turks. You can expect there to be more disagreement between (more than two? I'm not sure) Mechanical Turks than between two grad students (both in political science). Ultimately, my problem with the study is that the labelling they took as ground truth (i.e. that they compared ChatGPT and Mechanical Turks against) is too uncertain, or even subjective, to know for sure what exactly they found out. Edit: Oh, wait, I didn't put that in the comment above. Damn. I thought I had.
- nmca 4y ago
  In the framing of your original comment: A: Mturkers B: ChatGPT C: Experts => ChatGPT better approximates the labelling of D by experts than mTurkers Which is a coherent and interesting conclusion. Edit: also, please forgive the snark in my first response
  - YeGoblynQueenne 4y ago
    >> Edit: also, please forgive the snark in my first response No need to apologise! Your snark wasn't overboard, I thought. Anyway, big girl, can take it :) >> ChatGPT better approximates the labelling of D by experts than mTurkers That could be a "coherent and interesting conclusion" but it's not what the article really claims. The article's title is I think hedging its bets, by being very precise about who, exactly, was outperformed by ChatGPT, although it still manages to be vague about how ChatGPT outperformed the Mechanical Turks. I'm also really doubtful that two political science graduate students can be considered as "experts" in the annotation tasks they were called to perform, which were, again if I got that right, about content moderation. "Experts" in this setting would be people with experience in moderating discussion boards etc. I don't see that this was the case with the two students that provided the initial annotation.

malshe 4y ago

Your comment is based on a very strong assumption that all human annotators are alike in their motivations and abilities to do the tasks. As someone who has used mTurk in the past quite a lot, I think this assumption is wrong. That was the reason I stopped using mTurk. On a separate note, how do you type those arrows and subscripts in an HN comment?

Minor49er 4y ago
This is true. I set up a data validation project with mTurk in the past to validate scraped data by presenting a simply survey of the results. Basically "go to this webpage. The title is X (True or False). The description is Y (True or False)" etc. There were several users who would speedrun the surveys which created a lot of false positives mTurk has some ways to make results more accurate, such as being able to programmatically tank users who have given bad results and making it easy to give bonuses to users who give good results. But those have to be carefully designed within the application
- malshe 4y ago
  Yes, that's a lot of work. I remember in one image annotation task we got back what looked like a uniform distribution of responses. This was a pilot to determine the quality of responses so we did not waste a lot of money on it. The more limits we use, the smaller is the pool of mTurkers making it impractical for large tasks.
YeGoblynQueenne 4y ago
>> On a separate note, how do you type those arrows and subscripts in an HN comment? I type in gvim. It lets you type "digraphs" with Ctrl + K and another couple of keys. For the right-arrow it's (in insert mode and without typing spaces): Ctrl K - > In gvim, you can see all the digraphs with :digraph (in command mode). The digraph you want to type has to be supported by the font you use of course. gvim shows you the available digraphs in a scratch buffer that's not terribly easy to search (there's lots of them) but you can yank it to a register and paste it into a new buffer. I don't remember how I do that, I'd have to look up my vim notes :) >> Your comment is based on a very strong assumption that all human annotators are alike in their motivations and abilities to do the tasks. Yes and no. There's a few more threads here that discuss this as if ChatGPT beats all humans in annotating data, more or less. I think that's the general assumption anyway. Also, the way the study was carried out it seems to me that the authors themselves made this assumption, that one group of human annotators is like the other, and you can measure the difference between them as if one was the ground truth and the other just trying to approximate it, where the reality is that they are probably both trying to approximate some completely subjective measure of goodness.

sebzim4500 4y ago

I think your assumption that all humans are equally capable should be reconsidered.

YeGoblynQueenne 4y ago
That seems to be the authors' assumption.
- amluto 4y ago
  I didn’t check the paper’s methodology, but if humans and ChatGPT classify the same on average but ChatGPT is more consistent, then humans will match ChatGPT better than they match other humans.
- shenberg 4y ago
  The comparison in the paper is who approximates trained annotators better, MTurk or ChatGPT. Trained annotators are the gold standard.
  - YeGoblynQueenne 4y ago
    The "gold standard" in the article is two graduate students. The Mechanical Turks are probably more- I can't find the number in the paper. Given the "inter-coder agreement" is low for the Mechanical Turks (0.17% Pearson cor. coeff) it is no surprise that the "gold standard" and the Mechanical Turks' decisions diverge. So the comparison is pointless.
    - shenberg 4y ago
      It's the gold standard of text labeling to use grad students to label data as a gold standard, and it really depends on the difficulty of the task whether this makes sense. For example, if your labeling task is to, say, transcribe a bunch of recordings of spoken English in American English, maybe a grad student is a nice short-hand for "native American English speaker who is reasonably literate" which is a good baseline, compared to MTurk where many English speakers come from countries which speak English with different accents and idioms, maybe as a second language as well, and so I would expect them to have a harder time, and their computer setup is likely worse, and their pay isn't as high, etc. It seems to me that, A, at least for some tasks, grad-students-as-gold-standard is not wacky, and B, grad students won't necessarily have the same output as MTurk workers. Given these two points, it's perfectly reasonable to ask how consistent-with-grad-students MTurk workers are, seeing as the reason people use MTurk in similar scenarios is as a cheaper alternative to grad students. Re: inter-coder agreement, I think that the low inter-coder agreement for MTurk is in itself potentially a surprising aspect. Perhaps the explanations that worked well for grad students (and ChatGPT) didn't explain the task properly to MTurks. That's pretty much a point in favor of ChatGPT, though maybe it can be viewed as a point against the actual tasks they use (ie if the explanation couldn't get people of a likely more dissimilar background to output the same results as grad students then maybe the task is less objective than deemed by the task's authors). Side-notes: - I looked for the 0.17% number in the paper, and what the paper actually says is that the Pearson correlation coefficient between inter-coder agreement and classification accuracy is 0.17 (=positive but weak). This isn't a number comparing the correlation between different coders. (Frankly, I think comparing ChatGPT's self-consistency to consistency between different people is unenlightening) - Section 4 sub-section "Crowd-Workers Annotation" says that each tweet was classified by two different workers, and no single worker classified more than 20% of the dataset.