4 ms·

Unfortunately, unless you go door to door knocking... you're gonna get biased data.

ineedasername 7y ago
Or truly random sampling that includes appropriate proportions of sub groups, e.g, socioeconomic status etc. Otherwise you have to know on all of the doors. But it's still a lot: for a 95% confidence level and 2% margin of error, for NYC you would need to test about 2,400 people. Assuming a response rate of 10%, you need to knock on 24,000 doors.
- austingulati 7y ago
  Wouldn't the ability to refuse taking the test introduce bias? i.e. the 10% that do take it still may not be representative of the entire population. Likely better than the grocery store sample, but still not ideal
  - qqqwerty 7y ago
    Yeah, that is still an issue. You can offer compensation to incentivize participation. And you can pre-select your sample (instead of random door-to-door) and require multiple follow ups with the selected persons to reduce nonresponse bias. You can also do a separate phone survey in addition to surveying the test participants. Questions like "Do you think you have had COVID?" and "How many times per week do you leave the house?". If the responses for the test participants vary significantly from the phone survey participants, you can try and weight your data accordingly.
  - ineedasername 7y ago
    Sure, phone surveys for political purposes (presidential approval ratings etc.) have to deal with that all of the time. There are methods for estimating non-response impact. [0] One method of mitigating it that I've seen it to reach out again to non-responders. You then analyze their results to see how they differ from the baseline responders to estimate the non-responder population. If there's little/no difference, you can be fairly confident the risk of bias is low. It's called non-response follow up, and is a pretty common method. There's also literature that suggests that you don't discard outlier values in the actual responders as they may help approximate the non-responder population, i.e., the non-outliers represent typical responders while outliers are more likely to represent non-responders [1] [0]https://www.warc.com/content/paywall/article/jar/research_note_a_model_for_diagnosing_and_reducing_nonresponse_bias/14027 https://www.warc.com/content/paywall/article/jar/research_no... [1] https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-019-0757-1 https://bmcmedresmethodol.biomedcentral.com/articles/10.1186...
    - ineedasername 7y ago
      Wow, downvoted for providing factual information on how researchers mitigate non-response bias. Didn't think there was anything controversial there. I'm willing to accept the additional downvoted for this comment though.
      - austingulati 7y ago
        Not downvoted by me! Appreciate you sharing this information, it is helping me better understand how researchers mitigate non-response bias. Interesting stuff.
jonas21 7y ago
Even with door to door knocking, you're going to introduce a bias -- toward people who are around to answer the door, which would tend to undersample essential workers.
- AznHisoka 7y ago
  Or people who will try to talk to you from their balcony rather than answering the door.

Or truly random sampling that includes appropriate proportions of sub groups, e.g, socioeconomic status etc. Otherwise you have to know on all of the doors. But it's still a lot: for a 95% confidence level and 2% margin of error, for NYC you would need to test about 2,400 people. Assuming a response rate of 10%, you need to knock on 24,000 doors.

austingulati 7y ago
Wouldn't the ability to refuse taking the test introduce bias? i.e. the 10% that do take it still may not be representative of the entire population. Likely better than the grocery store sample, but still not ideal
- qqqwerty 7y ago
  Yeah, that is still an issue. You can offer compensation to incentivize participation. And you can pre-select your sample (instead of random door-to-door) and require multiple follow ups with the selected persons to reduce nonresponse bias. You can also do a separate phone survey in addition to surveying the test participants. Questions like "Do you think you have had COVID?" and "How many times per week do you leave the house?". If the responses for the test participants vary significantly from the phone survey participants, you can try and weight your data accordingly.
- ineedasername 7y ago
  Sure, phone surveys for political purposes (presidential approval ratings etc.) have to deal with that all of the time. There are methods for estimating non-response impact. [0] One method of mitigating it that I've seen it to reach out again to non-responders. You then analyze their results to see how they differ from the baseline responders to estimate the non-responder population. If there's little/no difference, you can be fairly confident the risk of bias is low. It's called non-response follow up, and is a pretty common method. There's also literature that suggests that you don't discard outlier values in the actual responders as they may help approximate the non-responder population, i.e., the non-outliers represent typical responders while outliers are more likely to represent non-responders [1] [0]https://www.warc.com/content/paywall/article/jar/research_note_a_model_for_diagnosing_and_reducing_nonresponse_bias/14027 https://www.warc.com/content/paywall/article/jar/research_no... [1] https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-019-0757-1 https://bmcmedresmethodol.biomedcentral.com/articles/10.1186...
  - ineedasername 7y ago
    Wow, downvoted for providing factual information on how researchers mitigate non-response bias. Didn't think there was anything controversial there. I'm willing to accept the additional downvoted for this comment though.
    - austingulati 7y ago
      Not downvoted by me! Appreciate you sharing this information, it is helping me better understand how researchers mitigate non-response bias. Interesting stuff.

jonas21 7y ago

Even with door to door knocking, you're going to introduce a bias -- toward people who are around to answer the door, which would tend to undersample essential workers.

AznHisoka 7y ago
Or people who will try to talk to you from their balcony rather than answering the door.