4 ms·
This is true. I set up a data validation project with mTurk in the past to validate scraped data by presenting a simply survey of the results. Basically "go to
by Minor49er 4y ago
This is true. I set up a data validation project with mTurk in the past to validate scraped data by presenting a simply survey of the results. Basically "go to this webpage. The title is X (True or False). The description is Y (True or False)" etc. There were several users who would speedrun the surveys which created a lot of false positives
mTurk has some ways to make results more accurate, such as being able to programmatically tank users who have given bad results and making it easy to give bonuses to users who give good results. But those have to be carefully designed within the application
- malshe 4y agoYes, that's a lot of work. I remember in one image annotation task we got back what looked like a uniform distribution of responses. This was a pilot to determine the quality of responses so we did not waste a lot of money on it. The more limits we use, the smaller is the pool of mTurkers making it impractical for large tasks.