2 ms·
Did anyone else read the paper? The summary doesn't seem very correct to me. From the summary: The results weren't pretty: in order to find a population of Tu
by nl 16y ago
Did anyone else read the paper? The summary doesn't seem very correct to me.
From the summary:
The results weren't pretty: in order to find a population of Turkers whose work was passable, the researchers first used Mechanical Turk to administer a test to 4,660 applicants. It was a multiple choice test to determine whether or not a Turker could identify the correct category for a business (Restaurant, Shopping, etc.) and verify, via its official website or by phone, its correct phone number and address.
79 passed. This was an extremely basic multiple choice test. It makes one wonder how the other 4,581 were smart enough to operate a web browser in the first place.
From the paper:
Of the 4,660 workers who took this test, only 1,658 (35.6%) workers earned a passing score, and over 25% of workers answered fewer than half of the questions correctly.
To investigate the high failure rate, we conversed with workers directly on TurkerNation and through private email. Based upon worker’s names and email addresses, we believe that we conversed with a representative sample of workers both inside and outside the United States. We found that the test was not too difficult and that most workers comprehended the questions. We believe that many applicants simply try to gain access to tasks as quickly as possible and do not actually put care into completing the test.
ie, 1658/4660 workers passed this test, NOT 79 (!!)
Then later they describe some additional filtering they put in place to attempt to find the best workers (they tried estimated location and time to complete task). Based on these filters they said: Using a combination of pre-screening and the test tasks described above, only 79 workers of 4,660 applicants qualified to process real business changes.