4 ms·

(I'm asking in all earnestness) Are online surveys now an accepted part of the social science repertoire? How do they compare to previous methods such as phone

(I'm asking in all earnestness) Are online surveys now an accepted part of the social science repertoire? How do they compare to previous methods such as phone surveys, which I imagine had more uniform distribution to the population but had their own participation bias problems?

eli 10y ago
I'm not a social scientist, but online polls have been widely used and accepted for years. All forms of polling have bias and these can be adjusted for. Internet-only polls fared well in 538's analysis of 2012 election polling: http://fivethirtyeight.blogs.nytimes.com/2012/11/10/which-polls-fared-best-and-worst-in-the-2012-presidential-race/?_r=0 http://fivethirtyeight.blogs.nytimes.com/2012/11/10/which-po...
- throwawayrol 10y ago
  Internet polls are significantly different from Amazon turk; see my sibling post. Internet polling isn't nearly as problematic, especially for a study like this. crowd-sourcing platforms need their own bias adjustments and I'm not even convinced that it's possible to adjust for the bias caused by using these platforms, unless you really are interested in putting a lower bound on human ability. Not a statement about the quality of people on Turk, just the amount of effort they often (rightfully!) put into below-developed-world-minimum-wage work...
detaro 10y ago
Not an answer, but additional info: I was curious how they found participants and looked it up in the paper: they posted their surveys on Amazon Mechanical Turk and paid participants 75 cents to complete it.
- pbreit 10y ago
  So all the survey subject members were Mechanical Turk users? That sounds like a terrible representation.
  - Houshalter 10y ago
    Sadly it's still a thousand times better than the book it's disputing, which has little or no empirical backing.
    - forgotpwtomain 10y ago
      Bad statistics aren't better than no statistics.
      - LionessLover 10y ago
        Looking at your comment I find this funny. All you make is an assertion.
        forgotpwtomain 10y ago
        If you take a group that's not representative of the population and you know nothing about the underlying distribution - that's not giving you better information it's just letting you be mislead into making statements which are erroneous.
throwawayrol 10y ago
Yes. Even and especially surveys where the participants are not only compensated for their time, but are also selected from a pool of people who are specifically seeking out such piece-male work (a la Amazon Turk). There's a big difference b/w posting flyers/ads in target-rich environments and providing a low friction way for people to get involved in the study. But if you do something on Amazon Turk you're basically guaranteeing a very particular type of person, because random guy with a high paying job probably isn't going to have a turk account and won't have the patience to create one just for your survey. In some cases you can make a good case that this doesn't totally invalidate any sort of generality in the results, but for the most part I throw out papers -- on face -- that proceed in this ways without extended, evidence-based discussion of why the hell I can expect any sort of conclusion except "our result generalizes to a population of people who really need an extra dollar or two (literally) and are looking for pretty much any way to get it."
btn 10y ago
"Social science" isn't a singular amorphous blob, and these methods aren't uniformly accepted. Online surveys are certainly becoming more popular as they are significantly cheaper to conduct than the alternatives, and yield publishable results that garner media attention. There are peer reviewers that will be sympathetic to these issues, regardless of the method's robustness. However, there are others that would say this reeks of dredging (p-hacking) in a very murky pool of data. Their "scepticism" rarely makes the New York Times (or a bestselling book), though.
LionessLover 10y ago
I think the (IMHO valid) point here is that that survey is a whole of a lot better than the anecdotes in the book that prompted them to do the research. It isn't about providing definitive evidence - but about providing better one.

eli 10y ago

I'm not a social scientist, but online polls have been widely used and accepted for years. All forms of polling have bias and these can be adjusted for. Internet-only polls fared well in 538's analysis of 2012 election polling: http://fivethirtyeight.blogs.nytimes.com/2012/11/10/which-polls-fared-best-and-worst-in-the-2012-presidential-race/?_r=0 http://fivethirtyeight.blogs.nytimes.com/2012/11/10/which-po...

throwawayrol 10y ago
Internet polls are significantly different from Amazon turk; see my sibling post. Internet polling isn't nearly as problematic, especially for a study like this. crowd-sourcing platforms need their own bias adjustments and I'm not even convinced that it's possible to adjust for the bias caused by using these platforms, unless you really are interested in putting a lower bound on human ability. Not a statement about the quality of people on Turk, just the amount of effort they often (rightfully!) put into below-developed-world-minimum-wage work...

detaro 10y ago

Not an answer, but additional info: I was curious how they found participants and looked it up in the paper: they posted their surveys on Amazon Mechanical Turk and paid participants 75 cents to complete it.

pbreit 10y ago
So all the survey subject members were Mechanical Turk users? That sounds like a terrible representation.
- Houshalter 10y ago
  Sadly it's still a thousand times better than the book it's disputing, which has little or no empirical backing.
  - forgotpwtomain 10y ago
    Bad statistics aren't better than no statistics.
    - LionessLover 10y ago
      Looking at your comment I find this funny. All you make is an assertion.
      - forgotpwtomain 10y ago
        If you take a group that's not representative of the population and you know nothing about the underlying distribution - that's not giving you better information it's just letting you be mislead into making statements which are erroneous.

throwawayrol 10y ago

Yes. Even and especially surveys where the participants are not only compensated for their time, but are also selected from a pool of people who are specifically seeking out such piece-male work (a la Amazon Turk). There's a big difference b/w posting flyers/ads in target-rich environments and providing a low friction way for people to get involved in the study. But if you do something on Amazon Turk you're basically guaranteeing a very particular type of person, because random guy with a high paying job probably isn't going to have a turk account and won't have the patience to create one just for your survey. In some cases you can make a good case that this doesn't totally invalidate any sort of generality in the results, but for the most part I throw out papers -- on face -- that proceed in this ways without extended, evidence-based discussion of why the hell I can expect any sort of conclusion except "our result generalizes to a population of people who really need an extra dollar or two (literally) and are looking for pretty much any way to get it."

btn 10y ago

"Social science" isn't a singular amorphous blob, and these methods aren't uniformly accepted. Online surveys are certainly becoming more popular as they are significantly cheaper to conduct than the alternatives, and yield publishable results that garner media attention. There are peer reviewers that will be sympathetic to these issues, regardless of the method's robustness. However, there are others that would say this reeks of dredging (p-hacking) in a very murky pool of data. Their "scepticism" rarely makes the New York Times (or a bestselling book), though.

LionessLover 10y ago

I think the (IMHO valid) point here is that that survey is a whole of a lot better than the anecdotes in the book that prompted them to do the research. It isn't about providing definitive evidence - but about providing better one.