4 ms·
Systematic bias in participants would be a far bigger worry to me than sample error from few participants. I'd take a properly randomised poll with 200 partici
by kqr 5y ago
Systematic bias in participants would be a far bigger worry to me than sample error from few participants.
I'd take a properly randomised poll with 200 participants over a non-random one with two million any day of the week. At least if participation is truly randomised you can calculate the maximum possible extent of the error. With bias, who knows?
- retrac 5y agoYep. People tend to underestimate the power of truly random sampling. For questions with a roughly equal split, a poll of just 100 people properly selected randomly from a population of 100 million has 99% confidence of being within 12% of the value you would get if you sampled the entire 100 million. This falls to 4% with a sample of 1000.
- karatinversion 5y agoAnd the other way, you can’t make up for bad sampling with response count; the classic case study is the Litery Digest poll of the 1936 US presidential election, which polled over 2 million people but got the result wrong by over 30 percentage points.
- dredmorbius 5y agoRight. "Dewey Beats Truman" is another notorious example: https://news.ycombinator.com/item?id=29757119 https://news.ycombinator.com/item?id=29757119 As household telephone service became near-universal, telephone surveys did become highly valuable and generally accurate in measuring public opinon. That's reversed over the past decade or so as both practices (refusal to pick up unknown calls) and service (landline service is at or below 20% in many parts of the US) have radically reformed. Political pollsters have commented on this at length, and it's a major concern. See Nate Silver at https://fivethirtyeight.com https://fivethirtyeight.com among others.