3 ms·
I think this is probably a poor metric for it. Mechanical Turk is full of people trying to race their way through surveys to make a few pennies. So there's a mo
by somenameforme 3y ago
I think this is probably a poor metric for it. Mechanical Turk is full of people trying to race their way through surveys to make a few pennies. So there's a motivation to just create some output that can't be automatically filtered out, like missing a 'please select D as the answer for this question' attention test, while otherwise just going as quickly as possible. It's a dream scenario for any sort of automation, no matter how crude. It's one step above MMO gold farming.
Tangentially related, Mechanical Turk and its various less well known clones which come down to the exact same thing, are increasingly the status quo for social science studies. They keep making really shocking discoveries with like 99.99999% statistical certainty given the huge sample sizes these services enable. Kind of weird how often they fail to replicate.
- AndrewKemendo 3y agoThe fact that MTurkers are paid makes them unreliable for reflecting genuine human responses. That they can’t replicate indicates that the sampling population isn’t consistent representation of population demographics and so there’s no real signal there that can be applied to a general population like most failure to replicate studies
- incrudible 3y agoNot paying would be a heavy bias, too. Do we need something like jury duty?
- mahathu 3y agoThe person you replied to was being sarcastic, but you're still right.
- mike_hearn 3y agoSociology's reliance on MTurk is absurd but most professionally run polls are now the same thing, just dressed up to look nicer. This is why polls keep reporting lots of weird results which are then taken as absolute fact by researchers and pundits. I saw one the other day from a professional polling firm in which exactly 7% of "the public" said they had attended a protest, no matter what the protest was about. They asked about six or seven things people might protest about, and for every single one, 7% of the panel said they'd been on such a protest. Taken at face value it led to nonsense like concluding millions of people had attended protest marches about CDBC and 15 minute cities. Unfortunately, the panels these firms use are extremely unrepresentative of the public but are advertised as perfect proxies. People then accept this claim unthinkingly. It's a problem.