4 ms·
> one assumes there is a correct answer, and presumably that is A, because there is a clear argument for it (you wish to sample from the population distribution
by pdonis 4y ago
> one assumes there is a correct answer, and presumably that is A, because there is a clear argument for it (you wish to sample from the population distribution, which can be viewed as a mixture of an initial-respondents distribution and an initial-non-respondents distribution and so you cannot claim to have sampled from the mixture until you've sampled from both mixture components, since they are probably statistically distinct)
But you can't survey non-respondents--because they don't respond!
What option A will actually result in is having a mixture of people who responded on the first try, and people who responded on the second try. In principle, neither of those will be representative of non-respondents. Whether this creates a significant problem in practice will depend on lots of other assumptions.
My own first choice would be either E or D, depending on (a) whether you have enough resources available to send 30 more emails, and (b) how much statistical power you are sacrificing with a sample of only 90 instead of 120. (If you were doing things properly, the number of emails you send out initially would be larger than the actual number you needed to get enough statistical power, by some factor that would depend on what fraction of people you expected to respond. In which case D would be the obvious option.)
- montroser 4y agoThe idea with A is that you will get a more representative sample if you follow up specifically with the people who did not initially respond. Yes, some will never respond, but the ones that do answer will increase the quality of the data, moreso than an additional random sample. The people who did not initially respond are more likely to be busy/lazy/unengaged, and without as many responses from that cohort, the data will be skewed.
- pdonis 4y ago> The idea with A is that you will get a more representative sample if you follow up specifically with the people who did not initially respond. Perhaps. But you are also creating a potential confounder, since you now have two categories of responders instead of one.
- ksenzee 4y agoThat seems like a pretty small potential confounder. It doesn't seem to bother the US Census, for example, that they get some people's response via mail right away, and others they have to go knock on the door and pester them to respond. Any random sample of people is going to include some who really like answering questionnaires, and some who hate it or are too distracted or busy. Designing a survey where you plan to contact people multiple times seems perfectly normal and reasonable.
- dragonwriter 4y ago> That seems like a pretty small potential confounder. It doesn't seem to bother the US Census, for example, that they get some people's response via mail right away, and others they have to go knock on the door and pester them to respond. Any random sample [...] The decennial census is not a random sample, though the Census Bureau does sruveys that are random samples separately.
- Retric 4y agoThe Census uses this technique when doing random samples. One of the ways to improve accuracy is to put a lot of effort into contacting a random sample of non respondents which simply isn’t viable at scale.
- pdonis 4y agoThe US Census is not doing sociological research on people's opinions. They're collecting factual information. Big difference.
- wombatpm 4y agoIf you want social science research checkout the General Social Survey by NORC. They have been following the same cohorts of people for the last 50 years with biannual surveys. The data is open to view at https://gssdataexplorer.norc.org/ https://gssdataexplorer.norc.org/
- aj7 4y agoOption A is absolutely the worst choice. The non-responders have already selected themselves into a non-random set, and making a request to them alters the entire experiment.
- mlyle 4y ago> The non-responders have already selected themselves into a non-random set Yes, and you're clawing back a non-random portion of that non-random set, which can be expected to improve the sample quality overall.
- pdonis 4y agoWhy would this be expected to improve sample quality? You want a random sample, not a non-random piece of a non-random subset. Non-random sampling just introduces more confounders and makes your sample quality worse.