3 ms·
Surely it's incorrect to make a blanket statement that voluntary response data is worthless. Worthless means there is zero information in the data, which would
by pg 18y ago
Surely it's incorrect to make a blanket statement that voluntary response data is worthless. Worthless means there is zero information in the data, which would only be true in extreme cases.
- Hexstream 18y agoNo information is better than misinformation.
- tomsaffell 18y agoTrue some of the time, but not all of the time.
- ErrantX 18y agoI'd argue that entirely depends on the context :) it's a bit of a blanket statement.
- jbert 18y agoIt's effectively worthless because it needs to be qualified so heavily. Dropping the qualification (as will happen sometimes) makes it more like misinformation, which is worse than worthless, so it averages out to zero... "The median poll selection of a HN visitor who clicked on a poll which was on the front page on the following weekdays (and national holidays in the following countries:...) was..." etc. If the information content is sufficiently hard to extract usefully that it would be easier to redo the sample in a sensible way, then you could call the sample worthless. (A bit like uneconomic oil reserves).
- IsaacSchlueter 18y agoSo, doesn't that just mean that a statistically significant portion of the regular HN users would have to respond? Of course, you'd have to figure out how to define "regular HN users" and what would be a significant portion of them. If this poll were to get that much response, it would also tell you something interesting: that HN users respond to polls in statistically significant numbers :) So, voluntary response polls are not worthless if enough people voluntarily respond to them, but this is a tricky problem.
- jbert 18y ago> So, doesn't that just mean that a statistically significant portion of the regular HN users would have to respond? No. If 50% of HN visitors voted but 99% of those over 40 didn't (due to whatever confounding factor you like - say embarrasment), you'd still wouldn't be able to extrapolate from the sample to the category of "HN visitors".
- tokenadult 18y agoCorrect. It doesn't matter how large a sample size is if the sample is biased. This should be something that every high school student knows after any high school statistics course, but it is a crucial consideration that is widely ignored.
- whatusername 18y agoalthough we can all think of an edge case.. (especially if we know how the sample is biased) eg: a gender survey in the general population that transgender respondees do not answer -- ends up with a 99% sample size (all but the transgenders) -- and thus gives a pretty significant bit of data on male:female ratios, etc.. Agreed in principle though.
- tokenadult 18y agoI think Professor Velleman promotes "Voluntary response data are worthless" as a slogan for the same reason an earlier generation of statisticians taught their students the slogan "correlation does not imply causation." That's because common human cognitive errors run strongly in one direction on each issue, so the slogan has take the cognitive error head-on. Of course, a distinct pattern in voluntary responses tells us SOMETHING (maybe about what kind of people come forward to respond), just as a correlation tells us SOMETHING (maybe about a lurking variable correlated with both things we observe), but it doesn't tell us enough to warrant a firm conclusion about facts of the world. The Literary Digest poll http://aurora.wells.edu/~srs/Math151-Fall02/Litdigest.htm http://aurora.wells.edu/~srs/Math151-Fall02/Litdigest.htm is a spectacular historical example of a voluntary response poll that didn't give a correct picture of reality at all.
- deleted 18y ago[deleted]