5 ms·

There's scope for pretty massive sampling errors with any door to door survey like this - you only end up polling people who are home (presumably during the day

There's scope for pretty massive sampling errors with any door to door survey like this - you only end up polling people who are home (presumably during the day), that answer their door to strangers (and not screen via camera) and are willing to answer a whole bunch of questions. That may be representative of the wider society, but amongst my acquaintances it is not.

reaperducer 7y ago
There's scope for pretty massive sampling errors with any door to door survey like this Not necessarily. While it's common for people on HN to eschew the results of surveys because of presumed sample bias, that's rarely what happens. I took a couple of statistics classes in college that spelled out the math involved, and essentially if you survey enough people within a population of a certain size, unintentional bias gets reduced to the point where it becomes a non-factor. It's all very interesting if you're into math. I am not, but statistics courses were required for my journalism degree. ("Unintentional" meaning there is such a thing as "intentional" bias. For example, if you're deliberately surveying Toyota owners about Toyotas, or something like that.)
- cookie_monsta 7y ago
  Yes, I don't know the maths but this study surveyed 2,000 people from a pool of around 54 million (18% of Britain's 66 million people are under 16 [1]). I don't think those numbers qualify as being enough people within a population of a certain size to avoid unintentional bias. [1] https://www.ons.gov.uk/peoplepopulationandcommunity/populationandmigration/populationestimates/articles/overviewoftheukpopulation/november2018 https://www.ons.gov.uk/peoplepopulationandcommunity/populati...
  - reaperducer 7y ago
    2,000 might actually be oversampling for 66,000,000 people. When the United States had about 300,000,000 people the number for statistical veracity was around 975. Again, my statistics classes were a long time ago, so the numbers I remember are inexact. But once I groked the math, it was surprising how few people you needed to interview.
    - cookie_monsta 7y ago
      Yes, there is a comment further down about how population size doesn't matter, that all you need is a sample size of around 1,000. Fascinating. I stand corrected.
- 0xffff2 7y ago
  I too have taken a couple of college statistics classes. I think you're misunderstanding/misrepresenting things a bit here. Yes, the internet at large loves to dismiss survey results for any and all perceived flaw, no matter how minute. That said, they aren't always wrong. In this particular case, everything you've said is true only if you assume a reasonably random sample. If your sampling method itself introduces biases, no increase in sample size is going to eliminate those biases. I'm not enough of a statistician to feel particularly confident in passing judgment over the sampling method here, but I think it's fair to question it, despite the fact that I'm ordinarily far more likely to take the statistics at face value than the average internet commenter.
aclimatt 7y ago
I don't believe they're going door to door and knocking on random doors. It's not said explicitly but it's implied that they're sending postal mail and following up with people who respond for in-home interviews. The methodology is outlined here: https://oxis.oii.ox.ac.uk/research/methodology/ https://oxis.oii.ox.ac.uk/research/methodology/
- vharuck 7y ago
  Mail wasn't involved except for picking random addresses from the list of known postal addresses. The interviewers went door-to-door. They also chose a household resident over 14 at random, not just the door answerer. And a 50% successful response rate is pretty good for a random survey. Really, though, I wish people who question survey methodology would mention specific problems with the methodology. One problem I see is that part of the weighting is based on "ACORN" categories, which aren't well-defined and remind me of junk sold by marketing companies. Of course, I assume the data's based on actual demographics, so it might be good for weighting even if it's bad at explaining.
  - cookie_monsta 7y ago
    Where does it say that they chose a household resident over 14 at random, not just the door answerer and what would that process even look like?
    - vharuck 7y ago
      In the methodology PDF linked by another commenter up this chain[0]: >At each address respondents for interview were selected by asking the person who answered the door if it would be possible to interview the person normally resident at that household ages 14 or over with the next birthday. If birthdays weren't known by the door-answerer, then the interviewer picked a subject by first letter of the first name. But that only happened in 5% of cases. [0] https://oxis.oii.ox.ac.uk/research/methodology/ https://oxis.oii.ox.ac.uk/research/methodology/
      - 7y ago

reaperducer 7y ago

There's scope for pretty massive sampling errors with any door to door survey like this Not necessarily. While it's common for people on HN to eschew the results of surveys because of presumed sample bias, that's rarely what happens. I took a couple of statistics classes in college that spelled out the math involved, and essentially if you survey enough people within a population of a certain size, unintentional bias gets reduced to the point where it becomes a non-factor. It's all very interesting if you're into math. I am not, but statistics courses were required for my journalism degree. ("Unintentional" meaning there is such a thing as "intentional" bias. For example, if you're deliberately surveying Toyota owners about Toyotas, or something like that.)

cookie_monsta 7y ago
Yes, I don't know the maths but this study surveyed 2,000 people from a pool of around 54 million (18% of Britain's 66 million people are under 16 [1]). I don't think those numbers qualify as being enough people within a population of a certain size to avoid unintentional bias. [1] https://www.ons.gov.uk/peoplepopulationandcommunity/populationandmigration/populationestimates/articles/overviewoftheukpopulation/november2018 https://www.ons.gov.uk/peoplepopulationandcommunity/populati...
- reaperducer 7y ago
  2,000 might actually be oversampling for 66,000,000 people. When the United States had about 300,000,000 people the number for statistical veracity was around 975. Again, my statistics classes were a long time ago, so the numbers I remember are inexact. But once I groked the math, it was surprising how few people you needed to interview.
  - cookie_monsta 7y ago
    Yes, there is a comment further down about how population size doesn't matter, that all you need is a sample size of around 1,000. Fascinating. I stand corrected.
0xffff2 7y ago
I too have taken a couple of college statistics classes. I think you're misunderstanding/misrepresenting things a bit here. Yes, the internet at large loves to dismiss survey results for any and all perceived flaw, no matter how minute. That said, they aren't always wrong. In this particular case, everything you've said is true only if you assume a reasonably random sample. If your sampling method itself introduces biases, no increase in sample size is going to eliminate those biases. I'm not enough of a statistician to feel particularly confident in passing judgment over the sampling method here, but I think it's fair to question it, despite the fact that I'm ordinarily far more likely to take the statistics at face value than the average internet commenter.

aclimatt 7y ago

I don't believe they're going door to door and knocking on random doors. It's not said explicitly but it's implied that they're sending postal mail and following up with people who respond for in-home interviews. The methodology is outlined here: https://oxis.oii.ox.ac.uk/research/methodology/ https://oxis.oii.ox.ac.uk/research/methodology/

vharuck 7y ago
Mail wasn't involved except for picking random addresses from the list of known postal addresses. The interviewers went door-to-door. They also chose a household resident over 14 at random, not just the door answerer. And a 50% successful response rate is pretty good for a random survey. Really, though, I wish people who question survey methodology would mention specific problems with the methodology. One problem I see is that part of the weighting is based on "ACORN" categories, which aren't well-defined and remind me of junk sold by marketing companies. Of course, I assume the data's based on actual demographics, so it might be good for weighting even if it's bad at explaining.
- cookie_monsta 7y ago
  Where does it say that they chose a household resident over 14 at random, not just the door answerer and what would that process even look like?
  - vharuck 7y ago
    In the methodology PDF linked by another commenter up this chain[0]: >At each address respondents for interview were selected by asking the person who answered the door if it would be possible to interview the person normally resident at that household ages 14 or over with the next birthday. If birthdays weren't known by the door-answerer, then the interviewer picked a subject by first letter of the first name. But that only happened in 5% of cases. [0] https://oxis.oii.ox.ac.uk/research/methodology/ https://oxis.oii.ox.ac.uk/research/methodology/
    - 7y ago