4 ms·

It's worth digging into the methodology here: the study participants were found through Mechanical Turk, and the proxy for cognitive ability is a vocabulary qui

It's worth digging into the methodology here: the study participants were found through Mechanical Turk, and the proxy for cognitive ability is a vocabulary quiz. You can just as well use this study to show that the kind of highly educated people whose circumstances are dire enough they have to work for pennies an hour have been socialized into giving correct answers about questions pertaining to freedom of speech.

harvey9 4y ago
Thanks for this. Second time in two days that a MTurk-based study has been highlighted as having questionable methodology.
weberer 4y ago
>It's worth digging into the methodology here: the study participants were found through Mechanical Turk That seems to be most studies these days. And 10 years ago it was mostly "we've recruited 200 students from [university performing the study]" and people complained about that too. And if you do a phone survey, then its "you're biasing toward the people who pick up calls from unknown numbers". There's always going to be some sort of sampling bias, and this study doesn't seem to be any sort of special outlier.
- sejje 4y ago
  > There's always going to be some sort of sampling bias I don't agree, I think we'll get pretty good at this. I agree there currently is pretty much always a sampling bias, though. Maybe the exception is when the study samples a particular group already, rather than the population at large.
- oh_my_goodness 4y ago
  I agree, but unfortunately that doesn't address the complaint. If surveys are in fact whacked out by bias (idk), then we won't get certain kinds of meaningful data from those surveys. It's tough for the surveyors, but that doesn't overcome the objection. I think in some fields this may be more obvious. If particles are too big for the LHC to find them, then we won't get meaningful data about those particles from those experiments. We can point out that we built a ridiculously, unbelievably huge collider and what do people expect? It's a valid point, but too bad for us.
- naniwaduni 4y ago
  Maybe we could fund the IRS to to conduct more audits by funneling them the grant money that otherwise funds mturk survey "research"! (This definitely seems like a great idea and nothing can possibly go wrong)
TigeriusKirk 4y ago
They also analyzed responses from 21 waves of the General Social Survey, which showed similar results.
deleted 4y ago
[deleted]
Consultant32452 4y ago
It's easier to train a smart dog than a dumb dog.

harvey9 4y ago

Thanks for this. Second time in two days that a MTurk-based study has been highlighted as having questionable methodology.

weberer 4y ago

>It's worth digging into the methodology here: the study participants were found through Mechanical Turk That seems to be most studies these days. And 10 years ago it was mostly "we've recruited 200 students from [university performing the study]" and people complained about that too. And if you do a phone survey, then its "you're biasing toward the people who pick up calls from unknown numbers". There's always going to be some sort of sampling bias, and this study doesn't seem to be any sort of special outlier.

sejje 4y ago
> There's always going to be some sort of sampling bias I don't agree, I think we'll get pretty good at this. I agree there currently is pretty much always a sampling bias, though. Maybe the exception is when the study samples a particular group already, rather than the population at large.
oh_my_goodness 4y ago
I agree, but unfortunately that doesn't address the complaint. If surveys are in fact whacked out by bias (idk), then we won't get certain kinds of meaningful data from those surveys. It's tough for the surveyors, but that doesn't overcome the objection. I think in some fields this may be more obvious. If particles are too big for the LHC to find them, then we won't get meaningful data about those particles from those experiments. We can point out that we built a ridiculously, unbelievably huge collider and what do people expect? It's a valid point, but too bad for us.
naniwaduni 4y ago
Maybe we could fund the IRS to to conduct more audits by funneling them the grant money that otherwise funds mturk survey "research"! (This definitely seems like a great idea and nothing can possibly go wrong)

TigeriusKirk 4y ago

They also analyzed responses from 21 waves of the General Social Survey, which showed similar results.

deleted 4y ago

[deleted]

Consultant32452 4y ago

It's easier to train a smart dog than a dumb dog.