5 ms·

Ever since CrowdFlower (now Figure8) decided to terminate its SaaS-offering and focus only on enterprise, Amazon Mechanical Turk has been impossible to use. Cr

Ever since CrowdFlower (now Figure8) decided to terminate its SaaS-offering and focus only on enterprise, Amazon Mechanical Turk has been impossible to use. CrowdFlower maintained statistics about the accuracy of individual workers. Additionally, they made it easy to include gold-standard "test" questions, which weeds out workers who are not doing the specific task correctly. Without this sort of quality-control platform, mechanical turk is just unreliable crowdsourced-labor infrastructure. I understand that AWS doesn't want to build too much on the services they provide but, really, mechanical turk sucks. I'm working with a researcher who is conducting experiments using turk, and he has a half-written shoddy version of what crowdflower offered. It seems that everyone using turk has to reinvent the wheel. Why can't someone provide decent quality control over turk results?

imglorp 5y ago
Sounds like a business opportunity? Quality/metrics layer as a Saas with MT on backend, add a few percent to the price per work unit?
- kuzee 5y ago
  Sounds sweet. I bet a lot of companies relying on mturk built this for themselves and then sell a higher value service with better margins. You could build something right in the middle. I know Stanford's research teams all use a common interface to mturk that keeps profiles of turkers on their side so they know who to solicit for upcoming surveys, conduct longitudinal studies, etc. I've always wondered why more universities didn't follow suit. I built a side project called cogmint based on the insight that simple scoring and ranking of workers was valuable. I ended up building my own worker interface instead of using mturk because it wasn't much additional effort on top of the scoring logic I was building anyway. Perhaps other serious companies came to the same conclusion I did with my hobby project.
- catillac 5y ago
  Sounds a little meta! Managed managed labeling workflow. I do wish Amazon would do a better job with this, and I also have no doubt if a company like this sprang into existence and was successful, Amazon would clamp down on it in some way — whether by changing terms to disallow such a business, cutting off MT access to the business, or just copying it.
- x0xMaximus 5y ago
  As AMT’s largest requester, we have longitudinal (~3yr) worker quality data for over 85% of respondents. Feel free to DM for details
  - georgeutsin 5y ago
    How do you deal with respondents who just created their account?
    - x0xMaximus 5y ago
      Technical and Demographic user metrics are federated across the platform (we supply tasks to other labor markets online, essentially where bots and farms typically attack) so we leverage priors and shared similarity weights. Behavioral user metrics aren’t the only way to evaluate a respondent
- Beldin 5y ago
  Given that a business pivoted away from exactly this, when they had this up and running, I'd advise putting in some effort to find out why they did so first.
rootsudo 5y ago
Because you're forgetting the other side of the equation, what sort of people are doing mechanical turk? Are they 1:1 your peers, colleagues? Or are they underpaid, most likely emerging economy countries, and very overworked labor that you're paying for? You can browse to the mechanical turk subreddits and you'll hear the other side of the table that they spend 15-20-hours on a task to get what, a few dollars? Pay more, get better results. The system itself is designed to exploit.
- catillac 5y ago
  I have definitely upped the pay and not achieved better results. I don’t think it’s quite as simple a solution as you’re making it out to be. Realistically no matter how much you pay people who don’t have the medium to advanced skills to do the job for whatever reason, they won’t have those skills, at least in the short term.
  - georgeutsin 5y ago
    Those skills are typically built through training anyway, but MTurk doesn't seem to have the necessary training pipeline.
- tedunangst 5y ago
  How many of your peers and colleagues are regularly checking mturk for suitable well paying tasks?
- notyourwork 5y ago
  Yep! The old saying "you get what you pay for" applies to Mechanical Turk like everything else. We tried outsourcing a project to Mechanical Turk and ended up spending more time on reliability and quality controls than it saved is in doing the manual work ourselves.
carom 5y ago
Mechanical Turk custom tasks are insanity as well. I was doing an object detection project and threw together a labeling UI. Integrated the Turk custom task, create and complete, but then I realized these tasks don't show up in the UI anywhere. You start implementing an entire UI for Amazon before you just bail and contract the one good Turk worker you've found.
kuzee 5y ago
Same! I created Cogmint.com ("cognition minting") to solve this problem for myself. You can submit known correct answers for questions, and those questions are then used as ground truth to score worker accuracy. Workers are then scored on their similarity to known correct answers and other workers that have accurately answered questions. It works surprisingly well for how simple it is. It's been a fun challenge to create simple methods of scoring similarity across different task types. It's a side project, so don't rely on it for mission critical things, but I rely on it for some production tasks, so it's stable. It currently supports classification (choose from a set of possible answers) and has beta support for bounding box task types. String input task types are coming very soon. I'd love to see if it can help you out, I'll waive the fees: I'm not in it for the money I just like making things useful and reliable. Reach out and say hi!
- sokoloff 5y ago
  We can find your site, but not your email. (Email field in your HN profile is always hidden; to share an email, put it in “about” on your profile.)
- georgeutsin 5y ago
  The problem of providing quality control is that there are a lot of edge cases; even known “high accuracy” turkers may have bad judgement sometimes, which means that every piece of data needs to be validated anyway, whether it be the researcher themselves or another paid contractor. My undergrad thesis was to build https://tagbull.com https://tagbull.com, where we tried to have turkers validate the work of other turkers by breaking up a label into sub tasks, and getting multi-turker consensus on those before moving forward. The main issue we ran into is that the incentive system is incredibly misaligned with the responsibility that the turkers have. It’s very difficult to build trust, especially with a crowd of people who haven’t signed contracts, and who face virtually no repercussions for doing bad work, whether intentionally or unintentionally.
- webmaven 5y ago
  > You can submit known correct answers for questions, and those questions are then used as ground truth to score worker accuracy. Workers are then scored on their similarity to known correct answers and other workers that have accurately answered questions. It works surprisingly well for how simple it is. Have you noticed problems that show up with questions whose answers have a bimodal distribution (ie. The gold standard question actually has two or more correct answers)? In one sense, this is just a labeling quality problem with the 'gold standard' data, but to a lesser extent these same issues may crop up in the data being labeled when using similarity or clustering to rate or classify the workers and transitively apply that to the other results they produce.
mrgordon 5y ago
CrowdFlower / Figure Eight is now part of Appen FYI
jamessb 5y ago
> I'm working with a researcher who is conducting experiments using turk, and he has a half-written shoddy version of what crowdflower offered For doing research, I think Prolific [1] is a popular alternative. They include features like pre-screening based on demographic features (including participants' approval rates in past studies) [2], and attention check questions [3]. [1]: https://www.prolific.co https://www.prolific.co [2]: https://researcher-help.prolific.co/hc/en-gb/articles/360009221093-Can-I-prescreen-for-certain-demographics-or-target-certain-participants-Which-demographic-filters-do-you-offer- https://researcher-help.prolific.co/hc/en-gb/articles/360009... [3]: https://researcher-help.prolific.co/hc/en-gb/articles/360009223553-Using-attention-checks-as-a-measure-of-data-quality https://researcher-help.prolific.co/hc/en-gb/articles/360009...
orp 5y ago
My startup, tasq.ai, was founded to solve this exact problem, as we have a live, successful product with all of the features you describe and more. We're happy to have a conversation - feel free to reach out to me via my profile email or you can find our contact details at https://www.tasq.ai/ https://www.tasq.ai/

imglorp 5y ago

Sounds like a business opportunity? Quality/metrics layer as a Saas with MT on backend, add a few percent to the price per work unit?

kuzee 5y ago
Sounds sweet. I bet a lot of companies relying on mturk built this for themselves and then sell a higher value service with better margins. You could build something right in the middle. I know Stanford's research teams all use a common interface to mturk that keeps profiles of turkers on their side so they know who to solicit for upcoming surveys, conduct longitudinal studies, etc. I've always wondered why more universities didn't follow suit. I built a side project called cogmint based on the insight that simple scoring and ranking of workers was valuable. I ended up building my own worker interface instead of using mturk because it wasn't much additional effort on top of the scoring logic I was building anyway. Perhaps other serious companies came to the same conclusion I did with my hobby project.
catillac 5y ago
Sounds a little meta! Managed managed labeling workflow. I do wish Amazon would do a better job with this, and I also have no doubt if a company like this sprang into existence and was successful, Amazon would clamp down on it in some way — whether by changing terms to disallow such a business, cutting off MT access to the business, or just copying it.
x0xMaximus 5y ago
As AMT’s largest requester, we have longitudinal (~3yr) worker quality data for over 85% of respondents. Feel free to DM for details
- georgeutsin 5y ago
  How do you deal with respondents who just created their account?
  - x0xMaximus 5y ago
    Technical and Demographic user metrics are federated across the platform (we supply tasks to other labor markets online, essentially where bots and farms typically attack) so we leverage priors and shared similarity weights. Behavioral user metrics aren’t the only way to evaluate a respondent
Beldin 5y ago
Given that a business pivoted away from exactly this, when they had this up and running, I'd advise putting in some effort to find out why they did so first.

rootsudo 5y ago

Because you're forgetting the other side of the equation, what sort of people are doing mechanical turk? Are they 1:1 your peers, colleagues? Or are they underpaid, most likely emerging economy countries, and very overworked labor that you're paying for? You can browse to the mechanical turk subreddits and you'll hear the other side of the table that they spend 15-20-hours on a task to get what, a few dollars? Pay more, get better results. The system itself is designed to exploit.

catillac 5y ago
I have definitely upped the pay and not achieved better results. I don’t think it’s quite as simple a solution as you’re making it out to be. Realistically no matter how much you pay people who don’t have the medium to advanced skills to do the job for whatever reason, they won’t have those skills, at least in the short term.
- georgeutsin 5y ago
  Those skills are typically built through training anyway, but MTurk doesn't seem to have the necessary training pipeline.
tedunangst 5y ago
How many of your peers and colleagues are regularly checking mturk for suitable well paying tasks?
notyourwork 5y ago
Yep! The old saying "you get what you pay for" applies to Mechanical Turk like everything else. We tried outsourcing a project to Mechanical Turk and ended up spending more time on reliability and quality controls than it saved is in doing the manual work ourselves.

carom 5y ago

Mechanical Turk custom tasks are insanity as well. I was doing an object detection project and threw together a labeling UI. Integrated the Turk custom task, create and complete, but then I realized these tasks don't show up in the UI anywhere. You start implementing an entire UI for Amazon before you just bail and contract the one good Turk worker you've found.

kuzee 5y ago

Same! I created Cogmint.com ("cognition minting") to solve this problem for myself. You can submit known correct answers for questions, and those questions are then used as ground truth to score worker accuracy. Workers are then scored on their similarity to known correct answers and other workers that have accurately answered questions. It works surprisingly well for how simple it is. It's been a fun challenge to create simple methods of scoring similarity across different task types. It's a side project, so don't rely on it for mission critical things, but I rely on it for some production tasks, so it's stable. It currently supports classification (choose from a set of possible answers) and has beta support for bounding box task types. String input task types are coming very soon. I'd love to see if it can help you out, I'll waive the fees: I'm not in it for the money I just like making things useful and reliable. Reach out and say hi!

sokoloff 5y ago
We can find your site, but not your email. (Email field in your HN profile is always hidden; to share an email, put it in “about” on your profile.)
georgeutsin 5y ago
The problem of providing quality control is that there are a lot of edge cases; even known “high accuracy” turkers may have bad judgement sometimes, which means that every piece of data needs to be validated anyway, whether it be the researcher themselves or another paid contractor. My undergrad thesis was to build https://tagbull.com https://tagbull.com, where we tried to have turkers validate the work of other turkers by breaking up a label into sub tasks, and getting multi-turker consensus on those before moving forward. The main issue we ran into is that the incentive system is incredibly misaligned with the responsibility that the turkers have. It’s very difficult to build trust, especially with a crowd of people who haven’t signed contracts, and who face virtually no repercussions for doing bad work, whether intentionally or unintentionally.
webmaven 5y ago
> You can submit known correct answers for questions, and those questions are then used as ground truth to score worker accuracy. Workers are then scored on their similarity to known correct answers and other workers that have accurately answered questions. It works surprisingly well for how simple it is. Have you noticed problems that show up with questions whose answers have a bimodal distribution (ie. The gold standard question actually has two or more correct answers)? In one sense, this is just a labeling quality problem with the 'gold standard' data, but to a lesser extent these same issues may crop up in the data being labeled when using similarity or clustering to rate or classify the workers and transitively apply that to the other results they produce.

mrgordon 5y ago

CrowdFlower / Figure Eight is now part of Appen FYI

jamessb 5y ago

> I'm working with a researcher who is conducting experiments using turk, and he has a half-written shoddy version of what crowdflower offered For doing research, I think Prolific [1] is a popular alternative. They include features like pre-screening based on demographic features (including participants' approval rates in past studies) [2], and attention check questions [3]. [1]: https://www.prolific.co https://www.prolific.co [2]: https://researcher-help.prolific.co/hc/en-gb/articles/360009221093-Can-I-prescreen-for-certain-demographics-or-target-certain-participants-Which-demographic-filters-do-you-offer- https://researcher-help.prolific.co/hc/en-gb/articles/360009... [3]: https://researcher-help.prolific.co/hc/en-gb/articles/360009223553-Using-attention-checks-as-a-measure-of-data-quality https://researcher-help.prolific.co/hc/en-gb/articles/360009...

orp 5y ago

My startup, tasq.ai, was founded to solve this exact problem, as we have a live, successful product with all of the features you describe and more. We're happy to have a conversation - feel free to reach out to me via my profile email or you can find our contact details at https://www.tasq.ai/ https://www.tasq.ai/