3 ms·

> We have access to individual-level desktop browsing data of a representative sample of the U.S. population via the market research firm Nielsen. Participants

by gzer0 5y ago

> We have access to individual-level desktop browsing data of a representative sample of the U.S. population via the market research firm Nielsen. Participants are incentivized to install a software that records all web browsing activity and fill in a survey of basic demographics, such as gender, employment, age, education, and income This seems to be an inherently flawed collection methodology. The users that one would expect to be involved with these "install this software, download this app and earn free money!!" schemes would also typically be associated with certain activities that would not necessarily reflect the overall population. The experiment groups themselves are flawed, but then again, I also cannot think of an ethical/legal way to conduct this kind of research.

reaperducer 5y ago
Selection bias is kind of a HN go-to for dismissing this people don't agree with. Bit something I learned in my college statistics courses is that with a large enough sample, self-selection bias stops becoming a problem. It was a long time ago, so I can't explain the math here. But from what I remember, you need a surprisingly small sample size to actually achieve real representation.
- pid-1 5y ago
  The math is called law of large numbers. It might work well, badly or not work at all depending on the distribution and the sampling methodology. Statistics as an area is full of gotchas, I never dismiss this sort of complain unless I have robust assumptions about the distribution being studied.
  - disgruntledphd2 5y ago
    Who needs a distribution when you have a collection of means?
  - dragonwriter 5y ago
    > The math is called law of large numbers That reduces sampling error, not non-sampling error.
- stdbrouw 5y ago
  Erm, no, self-selection bias never disappears or even attenuates with sample size, the best you can hope for is to reweight the sample according to known demographics of your target population, which does indeed get a bit easier with larger samples.
  - disgruntledphd2 5y ago
    > Erm, no, self-selection bias never disappears or even attenuates with sample size, the best you can hope for is to reweight the sample according to known demographics of your target population, Which, to be fair, is literally Neilsen's entire product (for TV at least). I mean, I guess that everyone here understands selection bias at a deep level, but to think that people who sell representative data to big corporations (and have done for longer than many of us have been alive) don't have a similar level of understanding is just weird.
  - dragonwriter 5y ago
    > the best you can hope for is to reweight the sample according to known demographics of your target population, which does indeed get a bit easier with larger samples. That doesn't actually deal with self-selection bias if self-selection correlates with the feature of interest within the demographic groups, which is probably the normal case.
- richdougherty 5y ago
  > Bit something I learned in my college statistics courses is that with a large enough sample, self-selection bias stops becoming a problem. What you might have learned is that small (random) samples might not represent the full population, but as you get bigger (random) samples they tend to get closer to the true values. However, if you sample badly, errors will persist even when you sample more. Example - estimating height in a population: - If I get a perfect random sample of people then I can estimate population height really well, even with a smallish sample. The estimate gets better the more people I (randomly) sample. - However if there's selection bias in my sampling and I only sample women, then no matter how many women I sample I'm going to be getting a bad estimate of height across the full population, because I'm excluding men who are taller. Sample size can't overcome selection bias, you need to use other techniques to manage it.
- dragonwriter 5y ago
  > Bit something I learned in my college statistics courses is that with a large enough sample, self-selection bias stops becoming a problem. That's...not true, until you are so close to the whole population that your maximum error from excluding part of the population is less than the error that would otherwise be introduced by bias. With larger samples, sampling error of a random sample is reduced, but non-sampling error is (with the above caveat) not.

by gzer0 5y ago

reaperducer 5y ago

Selection bias is kind of a HN go-to for dismissing this people don't agree with. Bit something I learned in my college statistics courses is that with a large enough sample, self-selection bias stops becoming a problem. It was a long time ago, so I can't explain the math here. But from what I remember, you need a surprisingly small sample size to actually achieve real representation.

pid-1 5y ago
The math is called law of large numbers. It might work well, badly or not work at all depending on the distribution and the sampling methodology. Statistics as an area is full of gotchas, I never dismiss this sort of complain unless I have robust assumptions about the distribution being studied.
- disgruntledphd2 5y ago
  Who needs a distribution when you have a collection of means?
- dragonwriter 5y ago
  > The math is called law of large numbers That reduces sampling error, not non-sampling error.
stdbrouw 5y ago
Erm, no, self-selection bias never disappears or even attenuates with sample size, the best you can hope for is to reweight the sample according to known demographics of your target population, which does indeed get a bit easier with larger samples.
- disgruntledphd2 5y ago
  > Erm, no, self-selection bias never disappears or even attenuates with sample size, the best you can hope for is to reweight the sample according to known demographics of your target population, Which, to be fair, is literally Neilsen's entire product (for TV at least). I mean, I guess that everyone here understands selection bias at a deep level, but to think that people who sell representative data to big corporations (and have done for longer than many of us have been alive) don't have a similar level of understanding is just weird.
- dragonwriter 5y ago
  > the best you can hope for is to reweight the sample according to known demographics of your target population, which does indeed get a bit easier with larger samples. That doesn't actually deal with self-selection bias if self-selection correlates with the feature of interest within the demographic groups, which is probably the normal case.
richdougherty 5y ago
> Bit something I learned in my college statistics courses is that with a large enough sample, self-selection bias stops becoming a problem. What you might have learned is that small (random) samples might not represent the full population, but as you get bigger (random) samples they tend to get closer to the true values. However, if you sample badly, errors will persist even when you sample more. Example - estimating height in a population: - If I get a perfect random sample of people then I can estimate population height really well, even with a smallish sample. The estimate gets better the more people I (randomly) sample. - However if there's selection bias in my sampling and I only sample women, then no matter how many women I sample I'm going to be getting a bad estimate of height across the full population, because I'm excluding men who are taller. Sample size can't overcome selection bias, you need to use other techniques to manage it.
dragonwriter 5y ago
> Bit something I learned in my college statistics courses is that with a large enough sample, self-selection bias stops becoming a problem. That's...not true, until you are so close to the whole population that your maximum error from excluding part of the population is less than the error that would otherwise be introduced by bias. With larger samples, sampling error of a random sample is reduced, but non-sampling error is (with the above caveat) not.