4 ms·

> Methodology > Reviews.org surveyed 1,000 Americans 18 years and older with a +/- 4% margin of error and a confidence level of 95%. The survey results were we

> Methodology > Reviews.org surveyed 1,000 Americans 18 years and older with a +/- 4% margin of error and a confidence level of 95%. The survey results were weighted to reflect characteristics of the United States population using available data from the US census. Absent them explicitly saying otherwise, I think we need to assume that the survey was done of reviews.org visitors. Reviews.org has reviews for exactly three categories of services: internet providers, mobile phone plans and services, and TV and streaming. Weighting for US demographics isn't going to make this sample very representative—this survey is of people who are browsing reviews for a set of products that most people don't think too hard about, which also happens to be a set of products that is tightly related to screen time use.

eckmLJE 2y ago
> we need to assume that the survey was done of reviews.org visitors. That's pretty unlikely. Reviews.org likely engages one of many vendors (like Qualtrics[1]) that will solicit responses online for a survey you design and provide to them through their survey building tool. [1] https://www.qualtrics.com/research-services/online-sample/ https://www.qualtrics.com/research-services/online-sample/
- lolinder 2y ago
  I've read dozens of these survey methodologies, and the reputable ones always tell you who they recruited to do the survey and what sampling method was used (if only in broad strokes). Given that this one does neither, we have to assume the laziest approach possible. If they wanted us to take their survey seriously they'd have a real methodology section.
  - eckmLJE 2y ago
    Sure, they could be lazy in a lot of ways in how they execute this survey leading to unreliable results -- indeed, fielding it through something like qualtrics could be said to be "lazier" than attempting to solicit and engage readers of your own individual site. It would be pretty hard and expensive to get enough responses yourself for even a lazy, low-quality result, vs. the at-scale offerings of a vendor.

eckmLJE 2y ago

> we need to assume that the survey was done of reviews.org visitors. That's pretty unlikely. Reviews.org likely engages one of many vendors (like Qualtrics[1]) that will solicit responses online for a survey you design and provide to them through their survey building tool. [1] https://www.qualtrics.com/research-services/online-sample/ https://www.qualtrics.com/research-services/online-sample/

lolinder 2y ago
I've read dozens of these survey methodologies, and the reputable ones always tell you who they recruited to do the survey and what sampling method was used (if only in broad strokes). Given that this one does neither, we have to assume the laziest approach possible. If they wanted us to take their survey seriously they'd have a real methodology section.
- eckmLJE 2y ago
  Sure, they could be lazy in a lot of ways in how they execute this survey leading to unreliable results -- indeed, fielding it through something like qualtrics could be said to be "lazier" than attempting to solicit and engage readers of your own individual site. It would be pretty hard and expensive to get enough responses yourself for even a lazy, low-quality result, vs. the at-scale offerings of a vendor.