4 ms·

Your comment is based on a very strong assumption that all human annotators are alike in their motivations and abilities to do the tasks. As someone who has use

by malshe 4y ago

Your comment is based on a very strong assumption that all human annotators are alike in their motivations and abilities to do the tasks. As someone who has used mTurk in the past quite a lot, I think this assumption is wrong. That was the reason I stopped using mTurk. On a separate note, how do you type those arrows and subscripts in an HN comment?

Minor49er 4y ago
This is true. I set up a data validation project with mTurk in the past to validate scraped data by presenting a simply survey of the results. Basically "go to this webpage. The title is X (True or False). The description is Y (True or False)" etc. There were several users who would speedrun the surveys which created a lot of false positives mTurk has some ways to make results more accurate, such as being able to programmatically tank users who have given bad results and making it easy to give bonuses to users who give good results. But those have to be carefully designed within the application
- malshe 4y ago
  Yes, that's a lot of work. I remember in one image annotation task we got back what looked like a uniform distribution of responses. This was a pilot to determine the quality of responses so we did not waste a lot of money on it. The more limits we use, the smaller is the pool of mTurkers making it impractical for large tasks.
YeGoblynQueenne 4y ago
>> On a separate note, how do you type those arrows and subscripts in an HN comment? I type in gvim. It lets you type "digraphs" with Ctrl + K and another couple of keys. For the right-arrow it's (in insert mode and without typing spaces): Ctrl K - > In gvim, you can see all the digraphs with :digraph (in command mode). The digraph you want to type has to be supported by the font you use of course. gvim shows you the available digraphs in a scratch buffer that's not terribly easy to search (there's lots of them) but you can yank it to a register and paste it into a new buffer. I don't remember how I do that, I'd have to look up my vim notes :) >> Your comment is based on a very strong assumption that all human annotators are alike in their motivations and abilities to do the tasks. Yes and no. There's a few more threads here that discuss this as if ChatGPT beats all humans in annotating data, more or less. I think that's the general assumption anyway. Also, the way the study was carried out it seems to me that the authors themselves made this assumption, that one group of human annotators is like the other, and you can measure the difference between them as if one was the ground truth and the other just trying to approximate it, where the reality is that they are probably both trying to approximate some completely subjective measure of goodness.
- malshe 4y ago
  Thanks for explaining how you used gvim for writing the comment! Btw, the authors may make that assumption but it's still a very strong assumption. On mTurk there is no reliable filter for the expertise the authors are looking for. So they used generic filters.

by malshe 4y ago

Minor49er 4y ago

This is true. I set up a data validation project with mTurk in the past to validate scraped data by presenting a simply survey of the results. Basically "go to this webpage. The title is X (True or False). The description is Y (True or False)" etc. There were several users who would speedrun the surveys which created a lot of false positives mTurk has some ways to make results more accurate, such as being able to programmatically tank users who have given bad results and making it easy to give bonuses to users who give good results. But those have to be carefully designed within the application

malshe 4y ago
Yes, that's a lot of work. I remember in one image annotation task we got back what looked like a uniform distribution of responses. This was a pilot to determine the quality of responses so we did not waste a lot of money on it. The more limits we use, the smaller is the pool of mTurkers making it impractical for large tasks.

YeGoblynQueenne 4y ago

>> On a separate note, how do you type those arrows and subscripts in an HN comment? I type in gvim. It lets you type "digraphs" with Ctrl + K and another couple of keys. For the right-arrow it's (in insert mode and without typing spaces): Ctrl K - > In gvim, you can see all the digraphs with :digraph (in command mode). The digraph you want to type has to be supported by the font you use of course. gvim shows you the available digraphs in a scratch buffer that's not terribly easy to search (there's lots of them) but you can yank it to a register and paste it into a new buffer. I don't remember how I do that, I'd have to look up my vim notes :) >> Your comment is based on a very strong assumption that all human annotators are alike in their motivations and abilities to do the tasks. Yes and no. There's a few more threads here that discuss this as if ChatGPT beats all humans in annotating data, more or less. I think that's the general assumption anyway. Also, the way the study was carried out it seems to me that the authors themselves made this assumption, that one group of human annotators is like the other, and you can measure the difference between them as if one was the ground truth and the other just trying to approximate it, where the reality is that they are probably both trying to approximate some completely subjective measure of goodness.

malshe 4y ago
Thanks for explaining how you used gvim for writing the comment! Btw, the authors may make that assumption but it's still a very strong assumption. On mTurk there is no reliable filter for the expertise the authors are looking for. So they used generic filters.