3 ms·
As I commented previously when we had a poll on the ages of HNers, the data can't be relied on to make such an inference. That's because the date are not from a
by tokenadult 16y ago
As I commented previously when we had a poll on the ages of HNers, the data can't be relied on to make such an inference. That's because the date are not from a random sample of the relevant population. One professor of statistics, who is a co-author of a highly regarded AP statistics textbook, has tried to popularize the phrase that "voluntary response data are worthless" to go along with the phrase "correlation does not imply causation." Other statistics teachers are gradually picking up this phrase.
-----Original Message----- From: Paul Velleman [SMTPfv2@cornell.edu] Sent: Wednesday, January 14, 1998 5:10 PM To: apstat-l@etc.bc.ca; Kim Robinson Cc: mmbalach@mtu.edu Subject: Re: qualtiative study
Sorry Kim, but it just aint so. Voluntary response data are worthless. One excellent example is the books by Shere Hite. She collected many responses from biased lists with voluntary response and drew conclusions that are roundly contradicted by all responsible studies. She claimed to be doing only qualitative work, but what she got was just plain garbage. Another famous example is the Literary Digest "poll". All you learn from voluntary response is what is said by those who choose to respond. Unless the respondents are a substantially large fraction of the population, they are very likely to be a biased -- possibly a very biased -- subset. Anecdotes tell you nothing at all about the state of the world. They can't be "used only as a description" because they describe nothing but themselves.
http://mathforum.org/kb/thread.jspa?threadID=194473&tstart=36420 http://mathforum.org/kb/thread.jspa?threadID=194473&tsta...
For more on the distinction between statistics and mathematics, see
http://statland.org/MAAFIXED.PDF http://statland.org/MAAFIXED.PDF
and
http://escholarship.org/uc/item/6hb3k0nz http://escholarship.org/uc/item/6hb3k0nz
I think Professor Velleman promotes "Voluntary response data are worthless" as a slogan for the same reason an earlier generation of statisticians taught their students the slogan "correlation does not imply causation." That's because common human cognitive errors run strongly in one direction on each issue, so the slogan has take the cognitive error head-on. Of course, a distinct pattern in voluntary responses tells us SOMETHING (maybe about what kind of people come forward to respond), just as a correlation tells us SOMETHING (maybe about a lurking variable correlated with both things we observe), but it doesn't tell us enough to warrant a firm conclusion about facts of the world. The Literary Digest poll
http://historymatters.gmu.edu/d/5168/ http://historymatters.gmu.edu/d/5168/
http://www.math.uah.edu/stat/data/LiteraryDigest.pdf http://www.math.uah.edu/stat/data/LiteraryDigest.pdf
is a spectacular historical example of a voluntary response poll that didn't give a correct picture of reality at all.
- Alex3917 16y ago"voluntary response data are worthless" There are ways of correcting for non-responders though. For example, there is the Monitoring The Future study that asks high school kids about their history of drug use. Almost everyone who is in school that day answers the questions, but that still leaves everyone who was absent that day. So what they do is they ask the kids the number of days of school they have missed so far this year, and then they correlate the number of missed days with drug usage. Using this information they can correct for every kid who was absent, because they know how many absentees there were and can figure out the distribution of how many of those kids have skipped one day of school so far, two days of school so far, etc. And then they can assign drug usage rates accordingly.
- obfuscate 16y agoWell, they can assign reported history of drug use correctly. Do they do anything similar to correct for lying, or just label their data as "self-reports of drug use"?
- Alex3917 16y ago"Do they do anything similar to correct for lying" There is an entire literature on the validity of self-reported drug use that they take into account. For example, there is a phenomenon called recanting, where someone will admit to using a drug one year and then deny it in future years. They correct for this by ignoring when someone recants if they have already admitted to using the drug on at least two previous surveys. (Which would be administered at least two years apart.) What they found was that the people most likely to recant were the ones who got jobs where they would now be fired for admitting drug use. For example, cops recanted drug use after previously admitting to it at twice the rate of the general public. So that's how they made the decision to discount recanting for people who had already admitted to drug use twice or more times. As for people who try to lie and claim they are taking drugs that they haven't been, this is corrected for by asking about the same drug multiple times in different ways. I think they may also salt the questions by asking about drugs that don't exist. You can obviously read all about the methodology of Monitoring the Future by downloading it, and here is an enormous PDF with tons of studies on the validity of self-reported drug use in general: http://archives.drugabuse.gov/pdf/monographs/monograph167/download167.html http://archives.drugabuse.gov/pdf/monographs/monograph167/do... All in all the methods they are using seem fairly accurate. It would definitely be a little higher if they corrected for more stuff, so for example when they say that 82% of Americans have used marijuana, it's actually probably closer to 85 or 86% when you correct for additional things that they haven't.