5 ms·
Case Study: Accuracy & Fraud on MTurk
- MediaSquirrel 15y ago"Humanoid’s software identified 8% of the workforce as engaged in fraud." <--that is a big number!
- sheraz 15y agoInteresting story -- I've attempted to use mturk a few times in the past, but it is actually a lot of work to get it right. Good instructions are really important, as well as a way to measure performance. Good luck to them as I think they might have made mturk easier to use for people like myself who see the value but couldn't figure out how to "make it work."
- MediaSquirrel 15y agogood instructions are crucial.
- bitstormer 15y agoGreat data, very cool how sophisticated algorithms can increase performance dramatically for human-powere tasks
- gcb 15y agoFails to mention how much they cost on top of the mturk prices to the client
- garethsprice 15y agoWhat does the author mean by fraud? Do people submit junk results and hope they slip through the screening process, or it it something more insidious?
- kailashbadu 15y agoJunk results, mostly, yes. That way they can complete more taks faster and still make money because most of the time their work is not manually approved.
- chrisyeh 15y agoI just used Humanoid for the first time. I had a job that would have taken about 5-10 hours (look up 50 ad agencies, record their Twitter account, blog URL, and their CEO's Twitter account and blog URL). Humanoid accomplished the job for under $35. Considering I would have had to do it myself, or nag an intern for a week to do it, I consider it money well spent. P.S. I don't know what results I would have gotten from going to Mturk directly, but the results from Humanoid were better/cheaper than my old habit of outsourcing to an Indian body shop. (Disclosure: I am an investor. But I'm giving honest feedback.)