4 ms·

It seems you could possibly apply this same logic to exclude any subgroup. For example, you must sample the mall-goer population. Some people "seem approachabl

It seems you could possibly apply this same logic to exclude any subgroup. For example, you must sample the mall-goer population. Some people "seem approachable"/etc. ("respond to your first email"), and you approach them. You are worried that others will be annoyed if you approach. If this assumption is true (which it may not be), their responses could be biased and confounding. Should you still survey them? I believe the stats textbooks would say yes, since it will result in a more representative sample. I assume it would be a detriment to the survey if you exclude some groups merely because they aren't compatible with your survey-taking techniques. All surveys have some inherent measurement error, and there are some techniques to detect biases/inconsistencies without excluding entire groups.

pdonis 4y ago
> It seems you could possibly apply this same logic to exclude any subgroup. No, just any subgroup that, by construction, is not reachable with whatever technique you are using. The key point is that there will always be such subgroups. See further comments below. > Should you still survey them? If you mean, should you still send out surveys the first time even though you don't know how people will respond? If you're going to do a survey at all, you have no choice, so this question is pointless. If you mean, should you continue to pester people who didn't respond the first time, on the grounds that "it will result in a more representative sample", I would question whether that is really the case. See further comments below. > I assume it would be a detriment to the survey if you exclude some groups merely because they aren't compatible with your survey-taking techniques. It's a "detriment" if you fail to realize that there is no way to avoid it. Taking a survey inherently means you get responses from people who respond to surveys and not from people who don't. Going back and asking a second time for a response from people who didn't respond the first time inherently means you get responses from people who will respond the second time and not from people who won't. And so on. There is no way to avoid the fact that any survey-taking technique will inherently exclude some proportion of the population. Once you realize this, you realize that the idea of "getting a more representive sample" is based on a false premise, that you can somehow "include" everyone if you just go about it the right way. You can't.
- psyklic 4y ago
  "Getting a more representative sample" is premised on getting as representative a sample as possible, not necessarily perfectly. Using language such as "pester" is making assumptions about the group. In fact, many likely were busy or have just forgotten to reply. As I mentioned above, there are techniques for assessing biased/inconsistent responses.
  - pdonis 4y ago
    > "Getting a more representative sample" is premised on getting as representative a sample as possible But representative of what? You can't get a sample that is "representative" of non-responders--because they don't respond. If you ask once and include whoever responds in your sample, your sample is representative (you hope) of people who respond. If you ask non-responders again, your sample now includes people who responded when asked once, and people who responded when asked twice. But the "asked twice" part introduces an extra variable: did the fact that you asked them twice change something that you would rather had not been changed? Given that, it's not clear that the second sample is any more "representative" of anything useful than the first.
    - psyklic 4y ago
      Representative of the population under study. In practice, "extra variables" are naturally present in every study. For example, surveying mall-goers in the morning vs evening. Or surveying people at one end of the mall vs the other. These variables are likely far more confounding than "must send a followup email". People asked in the evening might be more annoyed, but you shouldn't assume this a priori and decide to skip surveying them. Just as "busy people" in the studied population might forget to initially respond to an email and need a reminder. Just as a personal anecdote, I've dealt with this issue quite a lot. In my surveys, the people who respond initially are almost always eager to give glowing reviews. If we didn't send follow-ups, we would have extremely positively biased results. Maybe it is true that sending reminders makes respondents slightly more negative than they normally would be. However, we'd completely exclude "unsatisfied people" otherwise and the survey results would be worthless.

pdonis 4y ago

> It seems you could possibly apply this same logic to exclude any subgroup. No, just any subgroup that, by construction, is not reachable with whatever technique you are using. The key point is that there will always be such subgroups. See further comments below. > Should you still survey them? If you mean, should you still send out surveys the first time even though you don't know how people will respond? If you're going to do a survey at all, you have no choice, so this question is pointless. If you mean, should you continue to pester people who didn't respond the first time, on the grounds that "it will result in a more representative sample", I would question whether that is really the case. See further comments below. > I assume it would be a detriment to the survey if you exclude some groups merely because they aren't compatible with your survey-taking techniques. It's a "detriment" if you fail to realize that there is no way to avoid it. Taking a survey inherently means you get responses from people who respond to surveys and not from people who don't. Going back and asking a second time for a response from people who didn't respond the first time inherently means you get responses from people who will respond the second time and not from people who won't. And so on. There is no way to avoid the fact that any survey-taking technique will inherently exclude some proportion of the population. Once you realize this, you realize that the idea of "getting a more representive sample" is based on a false premise, that you can somehow "include" everyone if you just go about it the right way. You can't.

psyklic 4y ago
"Getting a more representative sample" is premised on getting as representative a sample as possible, not necessarily perfectly. Using language such as "pester" is making assumptions about the group. In fact, many likely were busy or have just forgotten to reply. As I mentioned above, there are techniques for assessing biased/inconsistent responses.
- pdonis 4y ago
  > "Getting a more representative sample" is premised on getting as representative a sample as possible But representative of what? You can't get a sample that is "representative" of non-responders--because they don't respond. If you ask once and include whoever responds in your sample, your sample is representative (you hope) of people who respond. If you ask non-responders again, your sample now includes people who responded when asked once, and people who responded when asked twice. But the "asked twice" part introduces an extra variable: did the fact that you asked them twice change something that you would rather had not been changed? Given that, it's not clear that the second sample is any more "representative" of anything useful than the first.
  - psyklic 4y ago
    Representative of the population under study. In practice, "extra variables" are naturally present in every study. For example, surveying mall-goers in the morning vs evening. Or surveying people at one end of the mall vs the other. These variables are likely far more confounding than "must send a followup email". People asked in the evening might be more annoyed, but you shouldn't assume this a priori and decide to skip surveying them. Just as "busy people" in the studied population might forget to initially respond to an email and need a reminder. Just as a personal anecdote, I've dealt with this issue quite a lot. In my surveys, the people who respond initially are almost always eager to give glowing reviews. If we didn't send follow-ups, we would have extremely positively biased results. Maybe it is true that sending reminders makes respondents slightly more negative than they normally would be. However, we'd completely exclude "unsatisfied people" otherwise and the survey results would be worthless.