3 ms·

> Only choice A tries to solve this problem. No, it doesn't. It doesn't get any information from non-responders; it can't, because they don't respond! It only

by pdonis 4y ago

> Only choice A tries to solve this problem. No, it doesn't. It doesn't get any information from non-responders; it can't, because they don't respond! It only gets information from some people who respond on the second try instead of the first.

next_xibalba 4y ago
There is nothing in the question which suggests some of them (or all) wouldn't respond to a second, third, or nth prompt. In many surveys, non-respondents receive multiple outreach attempts.
- pdonis 4y ago
  > There is nothing in the question which suggests some of them (or all) wouldn't respond to a second, third, or nth prompt. Of course not. But making second, third, or nth requests is not "sampling from non-responders". It's just creating more and more different groups of responders. Which doesn't improve anything statistically that I can see; in fact it might make things worse, because you are introducing more and more potential confounders.
  - psyklic 4y ago
    When this is a concern, I do two analyses -- one with and one without the potentially confounding group. It is still better to obtain the data, since otherwise we knowingly have a biased sample. (Somewhat similar to only surveying people who "seem approachable.")
    - pdonis 4y ago
      > When this is a concern, I do two analyses -- one with and one without the potentially confounding group. If this analysis tells you there is no significant difference between the subgroups, that's nice, because you can then just lump them into one and not worry about it any more. (This also tells you you didn't actually need to send out the second email, but what's done is done.) But if this analysis tells you there is a significant difference between the subgroups, you don't know if it's something that was there already or something that you created by sending out a second email to people who had not responded to the first. So you're not actually better off than if you hadn't sent the second email; you're worse off, because you now have a confounding variable present and you don't know what to do about it. > otherwise we knowingly have a biased sample You have a biased sample regardless of whether you send out a second email or not.
      - psyklic 4y ago
        It seems you could possibly apply this same logic to exclude any subgroup. For example, you must sample the mall-goer population. Some people "seem approachable"/etc. ("respond to your first email"), and you approach them. You are worried that others will be annoyed if you approach. If this assumption is true (which it may not be), their responses could be biased and confounding. Should you still survey them? I believe the stats textbooks would say yes, since it will result in a more representative sample. I assume it would be a detriment to the survey if you exclude some groups merely because they aren't compatible with your survey-taking techniques. All surveys have some inherent measurement error, and there are some techniques to detect biases/inconsistencies without excluding entire groups.
        pdonis 4y ago
        > It seems you could possibly apply this same logic to exclude any subgroup. No, just any subgroup that, by construction, is not reachable with whatever technique you are using. The key point is that there will always be such subgroups. See further comments below. > Should you still survey them? If you mean, should you still send out surveys the first time even though you don't know how people will respond? If you're going to do a survey at all, you have no choice, so this question is pointless. If you mean, should you continue to pester people who didn't respond the first time, on the grounds that "it will result in a more representative sample", I would question whether that is really the case. See further comments below. > I assume it would be a detriment to the survey if you exclude some groups merely because they aren't compatible with your survey-taking techniques. It's a "detriment" if you fail to realize that there is no way to avoid it. Taking a survey inherently means you get responses from people who respond to surveys and not from people who don't. Going back and asking a second time for a response from people who didn't respond the first time inherently means you get responses from people who will respond the second time and not from people who won't. And so on. There is no way to avoid the fact that any survey-taking technique will inherently exclude some proportion of the population. Once you realize this, you realize that the idea of "getting a more representive sample" is based on a false premise, that you can somehow "include" everyone if you just go about it the right way. You can't.
jefftk 4y ago
Choice A will likely get some answers from students who didn't respond the first time, increasing the representativeness of your sample. It's only futile if you conflate as "non-responders" the people who didn't respond on the first opportunity with the smaller number who wouldn't respond even given a second opportunity.
- pdonis 4y ago
  > Choice A will likely get some answers from students who didn't respond the first time, increasing the representativeness of your sample. Possibly. But it also introduces a potential confounder. > It's only futile if you conflate as "non-responders" the people who didn't respond on the first opportunity with the smaller number who wouldn't respond even given a second opportunity. No, the non-responders are the ones who never respond, period. Which, as I said, means you can't sample them to see what they're like. What you're doing, instead, is creating two different subsamples of responders: those who responded on the first try, and those who responded on the second try. Which might make things worse, since now you have an additional variable that you created by sending a second email to those who didn't respond the first time.
- aj7 4y ago
  Option A is absolutely the worst choice. The non-responders have already selected themselves into a non-random set, and making a request to them alters the entire experiment.
  - pdonis 4y ago
    > The non-responders have already selected themselves into a non-random set, and making a request to them alters the entire experiment. This is a better statement than any of mine of the point I've been trying to make.

by pdonis 4y ago

next_xibalba 4y ago

There is nothing in the question which suggests some of them (or all) wouldn't respond to a second, third, or nth prompt. In many surveys, non-respondents receive multiple outreach attempts.

pdonis 4y ago
> There is nothing in the question which suggests some of them (or all) wouldn't respond to a second, third, or nth prompt. Of course not. But making second, third, or nth requests is not "sampling from non-responders". It's just creating more and more different groups of responders. Which doesn't improve anything statistically that I can see; in fact it might make things worse, because you are introducing more and more potential confounders.
- psyklic 4y ago
  When this is a concern, I do two analyses -- one with and one without the potentially confounding group. It is still better to obtain the data, since otherwise we knowingly have a biased sample. (Somewhat similar to only surveying people who "seem approachable.")
  - pdonis 4y ago
    > When this is a concern, I do two analyses -- one with and one without the potentially confounding group. If this analysis tells you there is no significant difference between the subgroups, that's nice, because you can then just lump them into one and not worry about it any more. (This also tells you you didn't actually need to send out the second email, but what's done is done.) But if this analysis tells you there is a significant difference between the subgroups, you don't know if it's something that was there already or something that you created by sending out a second email to people who had not responded to the first. So you're not actually better off than if you hadn't sent the second email; you're worse off, because you now have a confounding variable present and you don't know what to do about it. > otherwise we knowingly have a biased sample You have a biased sample regardless of whether you send out a second email or not.
    - psyklic 4y ago
      It seems you could possibly apply this same logic to exclude any subgroup. For example, you must sample the mall-goer population. Some people "seem approachable"/etc. ("respond to your first email"), and you approach them. You are worried that others will be annoyed if you approach. If this assumption is true (which it may not be), their responses could be biased and confounding. Should you still survey them? I believe the stats textbooks would say yes, since it will result in a more representative sample. I assume it would be a detriment to the survey if you exclude some groups merely because they aren't compatible with your survey-taking techniques. All surveys have some inherent measurement error, and there are some techniques to detect biases/inconsistencies without excluding entire groups.
      - pdonis 4y ago
        > It seems you could possibly apply this same logic to exclude any subgroup. No, just any subgroup that, by construction, is not reachable with whatever technique you are using. The key point is that there will always be such subgroups. See further comments below. > Should you still survey them? If you mean, should you still send out surveys the first time even though you don't know how people will respond? If you're going to do a survey at all, you have no choice, so this question is pointless. If you mean, should you continue to pester people who didn't respond the first time, on the grounds that "it will result in a more representative sample", I would question whether that is really the case. See further comments below. > I assume it would be a detriment to the survey if you exclude some groups merely because they aren't compatible with your survey-taking techniques. It's a "detriment" if you fail to realize that there is no way to avoid it. Taking a survey inherently means you get responses from people who respond to surveys and not from people who don't. Going back and asking a second time for a response from people who didn't respond the first time inherently means you get responses from people who will respond the second time and not from people who won't. And so on. There is no way to avoid the fact that any survey-taking technique will inherently exclude some proportion of the population. Once you realize this, you realize that the idea of "getting a more representive sample" is based on a false premise, that you can somehow "include" everyone if you just go about it the right way. You can't.

jefftk 4y ago

Choice A will likely get some answers from students who didn't respond the first time, increasing the representativeness of your sample. It's only futile if you conflate as "non-responders" the people who didn't respond on the first opportunity with the smaller number who wouldn't respond even given a second opportunity.

pdonis 4y ago
> Choice A will likely get some answers from students who didn't respond the first time, increasing the representativeness of your sample. Possibly. But it also introduces a potential confounder. > It's only futile if you conflate as "non-responders" the people who didn't respond on the first opportunity with the smaller number who wouldn't respond even given a second opportunity. No, the non-responders are the ones who never respond, period. Which, as I said, means you can't sample them to see what they're like. What you're doing, instead, is creating two different subsamples of responders: those who responded on the first try, and those who responded on the second try. Which might make things worse, since now you have an additional variable that you created by sending a second email to those who didn't respond the first time.
aj7 4y ago
Option A is absolutely the worst choice. The non-responders have already selected themselves into a non-random set, and making a request to them alters the entire experiment.
- pdonis 4y ago
  > The non-responders have already selected themselves into a non-random set, and making a request to them alters the entire experiment. This is a better statement than any of mine of the point I've been trying to make.