6 ms·

A way to use Mechanical Turk from outside the US. I've needed it with every client I ever worked with, and I need it now and know a dozen companies who do. Cro

by crdb 10y ago

A way to use Mechanical Turk from outside the US. I've needed it with every client I ever worked with, and I need it now and know a dozen companies who do. CrowdFlower has a de facto monopoly on the outside of the US supply and charges an enormous premium for it - enough to turn off most of them. I certainly don't want to pay several thousand a month for the right to submit jobs, although I was ok with the 25% premium in the old days. If it's an alternative marketplace, it has to have excellent automation via API. I'm not going to use this for questionnaires, I'll be submitting thousands of jobs automatically. If you have built this already, please email me.

thenomad 10y ago
I could not agree more. It's quite ridiculous that there's no non-US alternative that's affordable.
Freeboots 10y ago
Out of curiosity, what do you use it for? I'm intrigued by platforms like MT, but have never had a real use myself.
- bruno2223 10y ago
  +1
- crdb 10y ago
  For my clients, the standard use cases for e-commerce. For example, you might have 100,000 products plus another 5,000 new ones listed per month. Your human error rate might be 3%. When a product arrives, its buyer (the corporate person responsible for signing on the brand) gives it a category and it shows up on the site. So 3% of the new products are erroneously categorised by their buyer. You give X human votes per product, say, 10. "Which of these categories applies to this product?" If the votes agree with each other, you can just input that as the category. If there's a split, you can feed those products to a more open ended question like "what category is this product" and use that as a starting point for either renaming the categories, finding a new category, or tagging: if it's a 50/50 split, you can just tag the product with both categories. This is all automated and used to cost me around $200 per 5,000 products. Another example might be training a data set for a machine learning algorithm. You send a data set to be trained by Turk and use it as training/test sets. I'm keeping this one deliberately vague for now as AFAIK we are the only ones in our space doing this and I don't even want to mention the space due to a relatively smart competitor. As a hypothetical example, you might be trying to predict the category from the description, gender, picture features, buyer and brand, and the above categorisation tasks can be used to train the algorithms you're testing out.
  - Freeboots 10y ago
    Interesting! Thanks
kveykva 10y ago
Check out https://www.scaleapi.com/ https://www.scaleapi.com/ not sure if they work outside of the US - but they are a MT alternative
monsterzz 10y ago
You can try Yandex.Toloka. It has API (unfortunately not compatible with MT)
- crdb 10y ago
  Thank you. This is the best available solution and I didn't know about it. We'll probably go with it for now, until we hit a scale of tasks that justifies setting up a US subsidiary and dealing with the IRS.
bshimmin 10y ago
I've used Fancy Hands (https://www.fancyhands.com/ https://www.fancyhands.com/) before and had good results with it, though it's probably not really price-competitive with MT; I paid about $15 for a quite large amount of scraping work that would've been annoying to write software to do.
- crdb 10y ago
  Looks like a neat site, but it doesn't allow us to structure the task (use an HTML template). The flexibility is important.
whamlastxmas 10y ago
How do they region lock it? Can't you just make a US business entity and pay through that, if the origin of payment is the issue?
- crdb 10y ago
  I could, but it's time consuming and I'd have to deal with the IRS. By not operating in either the US or the EU, I save myself a lot of administrative trouble and potential tax liabilities.
  - whamlastxmas 10y ago
    So you essentially just want a broker between yourself and MT? If all I have to do is provide you an account, and you pay up front with bitcoin, I'm happy to do this for you with 25% premium. All it involves for me is going out and buying prepaid debit cards of whatever amount and attaching them to the account. I fill them up with 75% of whatever dollar amount you send me.
    - crdb 10y ago
      Thanks for offering, but I doubt this methodology will scale. We really need a stable company with a product we can trust will both be around in 5 years and that can handle large volumes of tasks.
Eridrus 10y ago
Do you use custom software to interact with MTurk, or is there already some good software for interacting with it already? I have an image dataset I want classified and I've been putting it off for a while...
- crdb 10y ago
  Sites like CrowdFlower have point-and-click interfaces that allow you to just upload a CSV and drag and drop its columns as fields in your task template. What I used to do was host the images on a server, publicly, then upload the URL of the images and use it to display the images within the task. It's OK for one-offs. You get a thousand free rows with the CrowdFlower trial [1] and that's probably the most painless way to go for a classification problem. In our case though, everything will be written in-house. Automation is how we keep headcount low. [1] https://www.crowdflower.com/plans/ https://www.crowdflower.com/plans/
ayw 10y ago
Hey there—I'm CEO and cofounder of Scale API (YC S16, https://scaleapi.com https://scaleapi.com), which is a developer-first API for human tasks. We allow customers outside the US, and are much more focused on quality than MTurk. Definitely give us a shot, and feel free to email me if you have any questions! alex@scaleapi.com
- crdb 10y ago
  Hi Alex, thanks for your email. Replying here for the benefit of other readers. We can't use your product for the same reason we can't use most of the alternatives on the market: there's no way to push tasks with custom HTML/CSS templates, as is possible with MTurk. This is important at scale. Do let me know if this ever changes.
bbcbasic 10y ago
I've used https://microworkers.com/ https://microworkers.com/ before (many years ago) and seems to do the job quite well.

by crdb 10y ago

thenomad 10y ago

I could not agree more. It's quite ridiculous that there's no non-US alternative that's affordable.

Freeboots 10y ago

Out of curiosity, what do you use it for? I'm intrigued by platforms like MT, but have never had a real use myself.

bruno2223 10y ago
+1
crdb 10y ago
For my clients, the standard use cases for e-commerce. For example, you might have 100,000 products plus another 5,000 new ones listed per month. Your human error rate might be 3%. When a product arrives, its buyer (the corporate person responsible for signing on the brand) gives it a category and it shows up on the site. So 3% of the new products are erroneously categorised by their buyer. You give X human votes per product, say, 10. "Which of these categories applies to this product?" If the votes agree with each other, you can just input that as the category. If there's a split, you can feed those products to a more open ended question like "what category is this product" and use that as a starting point for either renaming the categories, finding a new category, or tagging: if it's a 50/50 split, you can just tag the product with both categories. This is all automated and used to cost me around $200 per 5,000 products. Another example might be training a data set for a machine learning algorithm. You send a data set to be trained by Turk and use it as training/test sets. I'm keeping this one deliberately vague for now as AFAIK we are the only ones in our space doing this and I don't even want to mention the space due to a relatively smart competitor. As a hypothetical example, you might be trying to predict the category from the description, gender, picture features, buyer and brand, and the above categorisation tasks can be used to train the algorithms you're testing out.
- Freeboots 10y ago
  Interesting! Thanks

kveykva 10y ago

Check out https://www.scaleapi.com/ https://www.scaleapi.com/ not sure if they work outside of the US - but they are a MT alternative

monsterzz 10y ago

You can try Yandex.Toloka. It has API (unfortunately not compatible with MT)

crdb 10y ago
Thank you. This is the best available solution and I didn't know about it. We'll probably go with it for now, until we hit a scale of tasks that justifies setting up a US subsidiary and dealing with the IRS.

bshimmin 10y ago

I've used Fancy Hands (https://www.fancyhands.com/ https://www.fancyhands.com/) before and had good results with it, though it's probably not really price-competitive with MT; I paid about $15 for a quite large amount of scraping work that would've been annoying to write software to do.

crdb 10y ago
Looks like a neat site, but it doesn't allow us to structure the task (use an HTML template). The flexibility is important.

whamlastxmas 10y ago

How do they region lock it? Can't you just make a US business entity and pay through that, if the origin of payment is the issue?

crdb 10y ago
I could, but it's time consuming and I'd have to deal with the IRS. By not operating in either the US or the EU, I save myself a lot of administrative trouble and potential tax liabilities.
- whamlastxmas 10y ago
  So you essentially just want a broker between yourself and MT? If all I have to do is provide you an account, and you pay up front with bitcoin, I'm happy to do this for you with 25% premium. All it involves for me is going out and buying prepaid debit cards of whatever amount and attaching them to the account. I fill them up with 75% of whatever dollar amount you send me.
  - crdb 10y ago
    Thanks for offering, but I doubt this methodology will scale. We really need a stable company with a product we can trust will both be around in 5 years and that can handle large volumes of tasks.

Eridrus 10y ago

Do you use custom software to interact with MTurk, or is there already some good software for interacting with it already? I have an image dataset I want classified and I've been putting it off for a while...

crdb 10y ago
Sites like CrowdFlower have point-and-click interfaces that allow you to just upload a CSV and drag and drop its columns as fields in your task template. What I used to do was host the images on a server, publicly, then upload the URL of the images and use it to display the images within the task. It's OK for one-offs. You get a thousand free rows with the CrowdFlower trial [1] and that's probably the most painless way to go for a classification problem. In our case though, everything will be written in-house. Automation is how we keep headcount low. [1] https://www.crowdflower.com/plans/ https://www.crowdflower.com/plans/

ayw 10y ago

Hey there—I'm CEO and cofounder of Scale API (YC S16, https://scaleapi.com https://scaleapi.com), which is a developer-first API for human tasks. We allow customers outside the US, and are much more focused on quality than MTurk. Definitely give us a shot, and feel free to email me if you have any questions! alex@scaleapi.com

crdb 10y ago
Hi Alex, thanks for your email. Replying here for the benefit of other readers. We can't use your product for the same reason we can't use most of the alternatives on the market: there's no way to push tasks with custom HTML/CSS templates, as is possible with MTurk. This is important at scale. Do let me know if this ever changes.

bbcbasic 10y ago

I've used https://microworkers.com/ https://microworkers.com/ before (many years ago) and seems to do the job quite well.