3 ms·

If you are hosting your own data, Amazon knows the answer but not the question. So in say a photo classification.. they know "turk xxx said green" they don't

If you are hosting your own data, Amazon knows the answer but not the question. So in say a photo classification.. they know "turk xxx said green" they don't know the photo. Seems like not too big of a deal, right? I can share with you all of my answer datasets, they are useless without the questions.

amelius 8y ago
The turks access your data (the questions) through Amazon, and I don't see how this will happen without Amazon having access to that data.
- Harvey-Specter 8y ago
  This doesn't have to be the case. For example, if you're paying mturk workers to fill out a survey you can give them a link to some third party (ie: Qualtrics) which will generate a code upon survey completion which the mturker enters as the answer to the only question on Amazon. If you're really concerned about a third party having access to your data, you could just host the survey yourself.
  - amelius 8y ago
    Ok, but Amazon also checks for turk performance, e.g. to keep out bots, and people who just submit wrong answers to make a quick buck. It doesn't seem trivial to build upon that by returning a code. (A related question: would it be possible to use Amazon Turk to train an algorithm that answers Amazon Turk questions?)
    - Harvey-Specter 8y ago
      You would still use mTurk to recruit participants, so you still benefit from their filters for acceptance rate etc. You would just implement the survey (probably with some attention checks to weed out people who just randomly answer questions), and at survey completion you give them a code to submit to complete the mTurk hit. Then you would only accept the submissions that have passed the attention checks, and pay those mTurkers.
- brianwawok 8y ago
  External Questions https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMturkAPI/ApiReference_ExternalQuestionArticle.html https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMturkAPI/A... This seems one of the more popular ways to do turks. You send people off to your own site, to show them stuff. All you really provide is a url, like www.mysite.com/questionId=123456... and render whatever you want when that appears

amelius 8y ago

The turks access your data (the questions) through Amazon, and I don't see how this will happen without Amazon having access to that data.

Harvey-Specter 8y ago
This doesn't have to be the case. For example, if you're paying mturk workers to fill out a survey you can give them a link to some third party (ie: Qualtrics) which will generate a code upon survey completion which the mturker enters as the answer to the only question on Amazon. If you're really concerned about a third party having access to your data, you could just host the survey yourself.
- amelius 8y ago
  Ok, but Amazon also checks for turk performance, e.g. to keep out bots, and people who just submit wrong answers to make a quick buck. It doesn't seem trivial to build upon that by returning a code. (A related question: would it be possible to use Amazon Turk to train an algorithm that answers Amazon Turk questions?)
  - Harvey-Specter 8y ago
    You would still use mTurk to recruit participants, so you still benefit from their filters for acceptance rate etc. You would just implement the survey (probably with some attention checks to weed out people who just randomly answer questions), and at survey completion you give them a code to submit to complete the mTurk hit. Then you would only accept the submissions that have passed the attention checks, and pay those mTurkers.
brianwawok 8y ago
External Questions https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMturkAPI/ApiReference_ExternalQuestionArticle.html https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMturkAPI/A... This seems one of the more popular ways to do turks. You send people off to your own site, to show them stuff. All you really provide is a url, like www.mysite.com/questionId=123456... and render whatever you want when that appears