5 ms·
"How accurate are online polls?" [M]ost online polls that use participants who volunteer to take part do not have a proven record of accuracy. There are at lea
by dredmorbius 5y ago
"How accurate are online polls?"
[M]ost online polls that use participants who volunteer to take part do not have a proven record of accuracy. There are at least two reasons for this. One is that not everyone in the U.S. uses the internet, and those who do not are demographically different from the rest of the public. Another reason is that people who volunteer for polls may be different from other people in ways that could make the poll unrepresentative. At worst, online polls can be seriously biased if people who hold a particular point of view are more motivated to participate than those with a different point of view.
https://www.pewresearch.org/2010/12/29/how-accurate-are-online-polls/ https://www.pewresearch.org/2010/12/29/how-accurate-are-onli...
Rather than serve as accurate assessments of public opinion or beliefs, online polls at best surface sentiments and potential areas of interest. Some are mere amusements. Many serve a darker purpose of advocating for a specific cause or ideology (a "push poll" https://en.wikipedia.org/wiki/Push_poll https://en.wikipedia.org/wiki/Push_poll), fishing for either insights on how audiences might be manipulated through advertising or propaganda, or outright soliciting personal information that is of use in hacking into accounts by guessing passwords or so-called "security questions".
- jart 5y agoPew Research is a biased source because CGI Script polling poses an existential threat to their business model.
- dredmorbius 5y agoNo. Pew Research (and pretty much any other credible research institution or academic) are domain experts, understand sampling methodology, and the very-well-known biases which occur from biased and/or self-selecting samples. Public polling can be exceedingly accurate based on surprisingly small samples (roughly 300 in most national political polls, for example, and even that is generous), so long as the sample is in fact truly random. Oh, and that the responses are not similarly filtered. One of the most famous cases of a poll which failed due to sampling bias was in the 1948 US Presidential election, and resulted in the Chicago Tribune erroneous headline, "Dewey Defeats Truman". This was drilled hard in my own stats education several decades ago and remains a sharp lesson in the risks of biased sampling. For a good descrition of the error, see: https://web.archive.org/web/20180823071102/https://textbook.ds100.org/ch/02/design_dewey_truman.html https://web.archive.org/web/20180823071102/https://textbook....
- jart 5y agoWere you under the impression that the HN voting feature was intended to predict U.S. election outcomes? I can see you like Pew but their magical authority on sampling and biases doesn't apply here.
- dredmorbius 5y agoMy point is that your statement is an unsubstantiated shallow dismissal which suggests a very poor understanding of statistical and research methods. Pew was simply the first well-articulated reference I turned up. There are numerous others. You've offered nothing other than your uninformed opinion and casual insults and accusations. Please don't do that.
- jart 5y ago
- nopenopenopeno 5y agoI guess we’ll just have to accept that we may never know how many people were previously aware that they could create polls on HN.
- deleted 5y ago[deleted]
- Nbox9 5y agoIt’s not just that online polls do not provide valid data, it’s that they provide misleading data.
- roenxi 5y agoPotato potato, as they say. It wouldn't be surprising if there was no link at all between reality and the poll. Consider as a worse case; there is a political question with a fairly clear reality ("the legislators should legislate the value of exp(1) to be 2.5 to make life easier"). The silent majority ignores the silly poll, and the lunatics who believe this is a good idea all put in a vote because they want to promote their mad cause. The poll could therefore be biased to be exactly wrong. May as well call it invalid data and save the time thinking about it.
- dredmorbius 5y agoOpinion polls, properly constructed and conducted, measure opinion. They do not measure any truth other than the truth of a specific opinon. Polling for the value of e is rather like polling for the piloting of a jetliner. You could certainly do it. It would end rather badly in virtually all instances.
- roenxi 5y agoThe point is the poll wouldn't be picking up the opinion of the crowd. It would be picking up the least representative sample of people; those who have an incentive for the poll to mis-represent popular opinion. I only picked exp() as an example because it can't start a political argument.
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- mike741 5y agoThese criticisms are not limited to just online polls. One advantage that online polls have is a larger sample size. Gallop polls, for example, tend to have an extremely small sample size of less than 2,000 people while an online poll can reach hundreds of thousands. Both online and offline have advantages and disadvantages. At the end of the day, if you're being shown a poll that you didn't pay for or participate in then it's likely intended to influence rather than educate.
- kqr 5y agoSystematic bias in participants would be a far bigger worry to me than sample error from few participants. I'd take a properly randomised poll with 200 participants over a non-random one with two million any day of the week. At least if participation is truly randomised you can calculate the maximum possible extent of the error. With bias, who knows?
- retrac 5y agoYep. People tend to underestimate the power of truly random sampling. For questions with a roughly equal split, a poll of just 100 people properly selected randomly from a population of 100 million has 99% confidence of being within 12% of the value you would get if you sampled the entire 100 million. This falls to 4% with a sample of 1000.
- karatinversion 5y agoAnd the other way, you can’t make up for bad sampling with response count; the classic case study is the Litery Digest poll of the 1936 US presidential election, which polled over 2 million people but got the result wrong by over 30 percentage points.
- dredmorbius 5y agoRight. "Dewey Beats Truman" is another notorious example: https://news.ycombinator.com/item?id=29757119 https://news.ycombinator.com/item?id=29757119 As household telephone service became near-universal, telephone surveys did become highly valuable and generally accurate in measuring public opinon. That's reversed over the past decade or so as both practices (refusal to pick up unknown calls) and service (landline service is at or below 20% in many parts of the US) have radically reformed. Political pollsters have commented on this at length, and it's a major concern. See Nate Silver at https://fivethirtyeight.com https://fivethirtyeight.com among others.
- jodrellblank 5y ago> “Another reason is that people who volunteer for polls may be different from other people in ways that could make the poll unrepresentative.” See also “most of what you read on the Internet is written by insane people” on r/Slatestarcodex - https://www.reddit.com/r/slatestarcodex/comments/9rvroo/most_of_what_you_read_on_the_internet_is_written/ https://www.reddit.com/r/slatestarcodex/comments/9rvroo/most...
- shusaku 5y agoIt’s definitely worth remembering that just as polls can be misleading about sentiment, so can the voting of comments.
- Semaphor 5y agoThis is always very apparent on /r/SampleSize (a subreddit dedicated to posting and taking surveys): While Reddit as a whole has far fewer women than men, women often outsize men, Germans are usually the 3rd biggest group after USA-ians and British, and transgender sometimes make up 15%. Luckily, most people there (judging by the comments) are aware of that.
- tzs 5y agoI wonder what could be done if HN itself were to analyze data from an HN poll, combining it with other data they have on the participants? Quite a lot about a person on HN can be inferred from that person's comment history. Inferences could also be made from their IP address history if that is logged. Upvote, downvote, flagging, and vouching history if logged would also provide some information. I wonder if there would be enough to correct for some of the biases?
- kwkelly 5y agoMost serious online pollsters are using random sampling through panel providers and stratify based on known demographic statistics. Samples are re-weighted by raking (which reduces bias but increases variance) to better match population statistics. This of course doesn’t mean all sources of bias are eliminated, but you also can’t eliminate all sources of bias in phone polling. Just like those taking polls on the internet, certain slices of the population that you can’t control for or don’t know about may be more or less willing to participate.