5 ms·
(Note: I am an academic social scientist who has done an MTurk pilot but never published anything based on MTurk responses) Yes, this is true, but this is true
by notafraudster 8y ago
(Note: I am an academic social scientist who has done an MTurk pilot but never published anything based on MTurk responses)
Yes, this is true, but this is true outside MTurk as well. If you rely on the YouGov panel (people complete surveys to get points for t-shirts), or really anything else, there's a strong incentive to cheat.
This is why good survey research is going to involve attempting to bound fraud through a variety of measures: attention checks ("What's your favorite color? Ignore this question and answer yellow."), looking for straight-lining (respondents always picking the leftmost answer), looking for unusual contradictions (i.e. asking the same question in two, opposite ways and looking for people who don't have the expected relation in their responses), looking at the distribution of completion speed and scrutinizing the lower quantiles, attempting to log participants who take the survey multiple times through dummy accounts, etc.
Oddly, my experience was that Turkers were fairly conscientious. I know one theory for this is that many Turkers are in fact Turking on the job, and so their reserve wages are not about what they are being paid for the MTurk hit, they're about what they're being paid to sit in their desk and not work.
- ben_w 8y ago> "What's your favorite color? Ignore this question and answer yellow." I’ve noticed something similar to that in some of the YouGov surveys I’ve answered over the last decade. I wonder what it says about me that I sometimes write, in the errors/comments section at the end of the surveys, that I have seen surprising questions such as “what do you think about Channel 4?” when I have previously answered that I haven’t watched Channel 4 recently.
- Operyl 8y agoIn this scenario, one might not watch Channel 4 for a reason, so asking for that reason is sensible. “User A doesn’t watch channel 4, this is what they think about it.”
- ben_w 8y agoCould be; while it seemed very wrong when I saw it, I didn’t record the exact words and it may have made more sense than it seemed to.
- ghaff 8y agoJust last week, I saw a presentation about some research results that used MTurk subjects. Some of the results were counter-intuitive, albeit fairly consistent across experiments. However, the subject matter was such (basically trust in the results of ML models with differing degrees of complexity/transparency) that I couldn't help wondering if the results might be greatly affected if the outcomes were genuinely important to people. As you say though, it's the same problem with just about any study of this type whether MTurks, some other online panel, or students earning a little bit of beer money.
- bjnord 8y agoI used to work in assessments (educational testing, employee satisfaction, etc.), and the MMPI and other psychological testing would sprinkle in items like "I have never had hair on any part of my body (agree/disagree)" and use them to calculate an honesty/reliability scale, to help assess whether the results were usable or the respondent was giving random or misleading answers.