5 ms·
I was using MTurk for labeling about 10 years ago. To see the other side I also did a 1 Month stint as a MTurk worker, earning about $300. It is absolutely ho
by sireat 4y ago
I was using MTurk for labeling about 10 years ago.
To see the other side I also did a 1 Month stint as a MTurk worker, earning about $300.
It is absolutely horrible work, and I used MTurk subreddit to find the "decent" jobs. I had the special firefox extension which ranked the job givers etc.
All jobs were below 1st world minimum wage and were incredibly depressing.
I think the adult content ones were the worst.
"Jian Yang: No! That's very boring work"
Worst thing was that you could not stop to think, you had to keep going if you wanted to at least make a few bucks an hour.
There are two solutions to the labeling issue:
Pay well (at least $20 an hour no matter the location).
If the workers feel exploited you do not get good results no matter the sanctions.
Even better solution to labeling is to find people who give a shit about your task.
That is how I've done 50k lines of labeling training our OCR for a rare font for Tesseract. We have a few volunteers and they know this is important work (preserving cultural heritage, non-profit, national library, etc).
Old reCaptcha had a similar "feel good" element.
Compare it to newest Google reCaptcha - you know those labels are going to be used for evil at some point in the future.
- MonkeyMalarky 4y agoI'm eager to see what nefarious things Google will do once they've mastered the art of identifying all the busses in a photo!
- stjohnswarts 4y agotheir motorcycle, hills, signal light, and traffic light AI will break the world when it's over!
- htrp 4y agohttps://xkcd.com/1897/ https://xkcd.com/1897/
- a9h74j 4y agoWhat tells me to start worrying if they get very good at identifying paperclips?
- drewcoo 4y agoThere are ways to detect problems with results due to things like fatigue or attenuation. And there are ways to deal with those problems (like forced break time and switching up tasks). It's a wonder that none of that is built in. But maybe things like MTurk aren't built to maximize worker effectiveness because it costs too much. Are there better quality crowd-sourcing options?
- simiones 4y ago> Compare it to newest Google reCaptcha - you know those labels are going to be used for evil at some point in the future. I for one always try at least once to get something wrong, often several times if it doesn't go through immediately, depending on urgency of my task. I hate being made to work for someone else like this.
- htrp 4y ago> To see the other side I also did a 1 Month stint as a MTurk worker, earning about $300. are you me? I did the same thing (worked as a mTurk labeler for 2 weeks) which convinced me to never use mTurk for anything even remotely important. I've been able to use semi-supervised approaches with actual domain experts reviewing outputs.