5 ms·

So many social science papers rely on MTurk which is concerning given the poor compensation. With such low payment structures, MTurk is incentivizing people to

So many social science papers rely on MTurk which is concerning given the poor compensation. With such low payment structures, MTurk is incentivizing people to rush through responses and give responses that they anticipate the researchers want to see.

notafraudster 7y ago
(Note: I am an academic social scientist who has done an MTurk pilot but never published anything based on MTurk responses) Yes, this is true, but this is true outside MTurk as well. If you rely on the YouGov panel (people complete surveys to get points for t-shirts), or really anything else, there's a strong incentive to cheat. This is why good survey research is going to involve attempting to bound fraud through a variety of measures: attention checks ("What's your favorite color? Ignore this question and answer yellow."), looking for straight-lining (respondents always picking the leftmost answer), looking for unusual contradictions (i.e. asking the same question in two, opposite ways and looking for people who don't have the expected relation in their responses), looking at the distribution of completion speed and scrutinizing the lower quantiles, attempting to log participants who take the survey multiple times through dummy accounts, etc. Oddly, my experience was that Turkers were fairly conscientious. I know one theory for this is that many Turkers are in fact Turking on the job, and so their reserve wages are not about what they are being paid for the MTurk hit, they're about what they're being paid to sit in their desk and not work.
- ben_w 7y ago
  > "What's your favorite color? Ignore this question and answer yellow." I’ve noticed something similar to that in some of the YouGov surveys I’ve answered over the last decade. I wonder what it says about me that I sometimes write, in the errors/comments section at the end of the surveys, that I have seen surprising questions such as “what do you think about Channel 4?” when I have previously answered that I haven’t watched Channel 4 recently.
  - Operyl 7y ago
    In this scenario, one might not watch Channel 4 for a reason, so asking for that reason is sensible. “User A doesn’t watch channel 4, this is what they think about it.”
    - ben_w 7y ago
      Could be; while it seemed very wrong when I saw it, I didn’t record the exact words and it may have made more sense than it seemed to.
- ghaff 7y ago
  Just last week, I saw a presentation about some research results that used MTurk subjects. Some of the results were counter-intuitive, albeit fairly consistent across experiments. However, the subject matter was such (basically trust in the results of ML models with differing degrees of complexity/transparency) that I couldn't help wondering if the results might be greatly affected if the outcomes were genuinely important to people. As you say though, it's the same problem with just about any study of this type whether MTurks, some other online panel, or students earning a little bit of beer money.
- bjnord 7y ago
  I used to work in assessments (educational testing, employee satisfaction, etc.), and the MMPI and other psychological testing would sprinkle in items like "I have never had hair on any part of my body (agree/disagree)" and use them to calculate an honesty/reliability scale, to help assess whether the results were usable or the respondent was giving random or misleading answers.

notafraudster 7y ago

(Note: I am an academic social scientist who has done an MTurk pilot but never published anything based on MTurk responses) Yes, this is true, but this is true outside MTurk as well. If you rely on the YouGov panel (people complete surveys to get points for t-shirts), or really anything else, there's a strong incentive to cheat. This is why good survey research is going to involve attempting to bound fraud through a variety of measures: attention checks ("What's your favorite color? Ignore this question and answer yellow."), looking for straight-lining (respondents always picking the leftmost answer), looking for unusual contradictions (i.e. asking the same question in two, opposite ways and looking for people who don't have the expected relation in their responses), looking at the distribution of completion speed and scrutinizing the lower quantiles, attempting to log participants who take the survey multiple times through dummy accounts, etc. Oddly, my experience was that Turkers were fairly conscientious. I know one theory for this is that many Turkers are in fact Turking on the job, and so their reserve wages are not about what they are being paid for the MTurk hit, they're about what they're being paid to sit in their desk and not work.

ben_w 7y ago
> "What's your favorite color? Ignore this question and answer yellow." I’ve noticed something similar to that in some of the YouGov surveys I’ve answered over the last decade. I wonder what it says about me that I sometimes write, in the errors/comments section at the end of the surveys, that I have seen surprising questions such as “what do you think about Channel 4?” when I have previously answered that I haven’t watched Channel 4 recently.
- Operyl 7y ago
  In this scenario, one might not watch Channel 4 for a reason, so asking for that reason is sensible. “User A doesn’t watch channel 4, this is what they think about it.”
  - ben_w 7y ago
    Could be; while it seemed very wrong when I saw it, I didn’t record the exact words and it may have made more sense than it seemed to.
ghaff 7y ago
Just last week, I saw a presentation about some research results that used MTurk subjects. Some of the results were counter-intuitive, albeit fairly consistent across experiments. However, the subject matter was such (basically trust in the results of ML models with differing degrees of complexity/transparency) that I couldn't help wondering if the results might be greatly affected if the outcomes were genuinely important to people. As you say though, it's the same problem with just about any study of this type whether MTurks, some other online panel, or students earning a little bit of beer money.
bjnord 7y ago
I used to work in assessments (educational testing, employee satisfaction, etc.), and the MMPI and other psychological testing would sprinkle in items like "I have never had hair on any part of my body (agree/disagree)" and use them to calculate an honesty/reliability scale, to help assess whether the results were usable or the respondent was giving random or misleading answers.