9 ms·

The Magic of Sampling, and Its Limitations

86 points by gen220 4y ago

jandalf 4y ago
I have a question regarding the direct computation of the error sizes via the inverse error function. Does this not assume that the sample error is normally distributed whereas it is hypergeometric (binomial in approximation)? Has anybody a link which explains this way of error estimation in detail? I would have used an exact confidence interval + its half width, but I am no expert in statistics.
- ericpauley 4y ago
  The binomial distribution quickly converges towards the normal distribution for large n, so this relaxation is pretty commonly used.
- 6gvONxR4sf7o 4y ago
  The law of large numbers comes into play here. The average of a thing quickly becomes gaussian in a huge variety of circumstances.
ckelly 4y ago
It has always surprised me that many technology professionals (and business professionals in general) don't have a strong intuition for the power of sampling. For example, in this case, the author states: "With 100 samples, our estimates are accurate to within about 5%. The magic of sampling is that we can derive accurate estimates about a very large population using a relatively small number of samples. In the last scenario (100 billion M&MS), we have 1% accuracy despite only sampling 0.00001% of the M&Ms." I bet many would think n=100 would be worthless once the population reaches millions, or especially billions. One HN-related piece of evidence for that is when I pointed out what margin of error would be for a n=164 survey sample, I got downvoted hard! https://news.ycombinator.com/item?id=8050801 https://news.ycombinator.com/item?id=8050801 But I saw this hundreds of times talking to customers when I ran a survey sampling product out of YC.
- kevin_thibedeau 4y ago
  > I bet many would think n=100 would be worthless once the population reaches millions, or especially billions. That depends on a uniform distribution of the population and an unbiased sampling method. One of the polls in the 2016 US presidential election [1] would shift Trump's position by a full percentage point based on input from a single man depending on which week he participated in the panel. [1] https://www.nytimes.com/2016/10/13/upshot/how-one-19-year-old-illinois-man-is-distorting-national-polling-averages.html https://www.nytimes.com/2016/10/13/upshot/how-one-19-year-ol...
  - Dylan16807 4y ago
    > a full percentage point If your goal is to be within 10%, that's not a problem.
  - rcxdude 4y ago
    Yes, but if your sampling is biased then the solution is not simply a larger sample size. The only way in which larger sample sizes help is giving more power to any methods you use to try to control for sampling bias.
- taeric 4y ago
  Yeah, I confess I'm having a hard time understanding this. How much of the underlying population do you have to accurately know for such a small sample to be worth so much? Edit: I see that the article describes some of the limitations. I'm curious on how to work with unknown populations. That said, it does have me revisiting some ideas. Looking forward to it.
  - CrazyStat 4y ago
    > How much of the underlying population do you have to accurately know for such a small sample to be worth so much? That's the magic of random sampling, you don't need to accurately know anything about the underlying population. If you do know things about the underlying population then you can do clever things like stratified sampling to get even more accurate measurements, but that's not necessary. The magic is that a randomly selected group of 100/1000/10000 is unlikely to be too different from the population as a whole, no matter what that population looks like. You do have to be able to sample randomly--truly randomly[1]--from the population, though, and that's often an issue. Picking 100 people randomly from the population of "likely voters in the next US presidential election" is a very nontrivial thing. To start with, that population is not even very well defined; who is likely to vote changes over time and is difficult to pin down. Pollsters do various things to try to account for this, but if they fail to predict say a surge in young voters their numbers will end up being off. Even if the population is clearly defined, it's not easy to survey a truly random sample from it. Some people are hard to reach. Some people don't want to talk to you, and whether or not they're willing to talk to you might be correlated with the thing you're interested in (like who they plan to vote for). You can do things to try to correct for that, but again if you get that wrong (and it's very hard to get right) your estimates will be off. And of course, if you're interested in things that are rare, like third party voters, you need a much larger sample to get an accurate read. If you sample 100 likely voters there's a pretty good chance you won't get a single person who plans to vote for the Libertarian Party candidate. [1] For the most basic form of random sampling, simple random sampling, you need not just every individual in the population to have the same probability of getting sampled, but every possible sample (i.e. every possible set of 100) needs to have the same probability of being sampled.
bigmattystyles 4y ago
Followed the link thinking I would get a refresher on Nyquist frequency - but they meant another type of sampling. But if like me you were seeking a refresher https://en.wikipedia.org/wiki/Nyquist_frequency https://en.wikipedia.org/wiki/Nyquist_frequency
- karmakaze 4y ago
  I thought so too, though I'm quite interested in both types. At one time long ago I thought we can't just rasterize everything, there must be cases where, or as resolutions increase, it won't scale well/make sense. That didn't turn out to be the case. Another story was at one company I was at, they were trying to make raster-driven PCB masks (I think that was how it worked, I was in a different dept). The traces must be continuous and robust so is normally done pen-plotter style. They were able to keep increasing the resolution of the laser rasterizing process to the point where it no longer mattered. I think now the default view is to work in pixels, voxels, etc. For all we know, the universe could work that way too. Like what's with dark energy? Space being created by a vacuum.
__mharrison__ 4y ago
Big data hates this article...

86 points by gen220 4y ago

I have a question regarding the direct computation of the error sizes via the inverse error function. Does this not assume that the sample error is normally distributed whereas it is hypergeometric (binomial in approximation)? Has anybody a link which explains this way of error estimation in detail? I would have used an exact confidence interval + its half width, but I am no expert in statistics.

ericpauley 4y ago
The binomial distribution quickly converges towards the normal distribution for large n, so this relaxation is pretty commonly used.
6gvONxR4sf7o 4y ago
The law of large numbers comes into play here. The average of a thing quickly becomes gaussian in a huge variety of circumstances.

ckelly 4y ago

It has always surprised me that many technology professionals (and business professionals in general) don't have a strong intuition for the power of sampling. For example, in this case, the author states: "With 100 samples, our estimates are accurate to within about 5%. The magic of sampling is that we can derive accurate estimates about a very large population using a relatively small number of samples. In the last scenario (100 billion M&MS), we have 1% accuracy despite only sampling 0.00001% of the M&Ms." I bet many would think n=100 would be worthless once the population reaches millions, or especially billions. One HN-related piece of evidence for that is when I pointed out what margin of error would be for a n=164 survey sample, I got downvoted hard! https://news.ycombinator.com/item?id=8050801 https://news.ycombinator.com/item?id=8050801 But I saw this hundreds of times talking to customers when I ran a survey sampling product out of YC.

kevin_thibedeau 4y ago
> I bet many would think n=100 would be worthless once the population reaches millions, or especially billions. That depends on a uniform distribution of the population and an unbiased sampling method. One of the polls in the 2016 US presidential election [1] would shift Trump's position by a full percentage point based on input from a single man depending on which week he participated in the panel. [1] https://www.nytimes.com/2016/10/13/upshot/how-one-19-year-old-illinois-man-is-distorting-national-polling-averages.html https://www.nytimes.com/2016/10/13/upshot/how-one-19-year-ol...
- Dylan16807 4y ago
  > a full percentage point If your goal is to be within 10%, that's not a problem.
- rcxdude 4y ago
  Yes, but if your sampling is biased then the solution is not simply a larger sample size. The only way in which larger sample sizes help is giving more power to any methods you use to try to control for sampling bias.
taeric 4y ago
Yeah, I confess I'm having a hard time understanding this. How much of the underlying population do you have to accurately know for such a small sample to be worth so much? Edit: I see that the article describes some of the limitations. I'm curious on how to work with unknown populations. That said, it does have me revisiting some ideas. Looking forward to it.
- CrazyStat 4y ago
  > How much of the underlying population do you have to accurately know for such a small sample to be worth so much? That's the magic of random sampling, you don't need to accurately know anything about the underlying population. If you do know things about the underlying population then you can do clever things like stratified sampling to get even more accurate measurements, but that's not necessary. The magic is that a randomly selected group of 100/1000/10000 is unlikely to be too different from the population as a whole, no matter what that population looks like. You do have to be able to sample randomly--truly randomly[1]--from the population, though, and that's often an issue. Picking 100 people randomly from the population of "likely voters in the next US presidential election" is a very nontrivial thing. To start with, that population is not even very well defined; who is likely to vote changes over time and is difficult to pin down. Pollsters do various things to try to account for this, but if they fail to predict say a surge in young voters their numbers will end up being off. Even if the population is clearly defined, it's not easy to survey a truly random sample from it. Some people are hard to reach. Some people don't want to talk to you, and whether or not they're willing to talk to you might be correlated with the thing you're interested in (like who they plan to vote for). You can do things to try to correct for that, but again if you get that wrong (and it's very hard to get right) your estimates will be off. And of course, if you're interested in things that are rare, like third party voters, you need a much larger sample to get an accurate read. If you sample 100 likely voters there's a pretty good chance you won't get a single person who plans to vote for the Libertarian Party candidate. [1] For the most basic form of random sampling, simple random sampling, you need not just every individual in the population to have the same probability of getting sampled, but every possible sample (i.e. every possible set of 100) needs to have the same probability of being sampled.

bigmattystyles 4y ago

Followed the link thinking I would get a refresher on Nyquist frequency - but they meant another type of sampling. But if like me you were seeking a refresher https://en.wikipedia.org/wiki/Nyquist_frequency https://en.wikipedia.org/wiki/Nyquist_frequency

karmakaze 4y ago
I thought so too, though I'm quite interested in both types. At one time long ago I thought we can't just rasterize everything, there must be cases where, or as resolutions increase, it won't scale well/make sense. That didn't turn out to be the case. Another story was at one company I was at, they were trying to make raster-driven PCB masks (I think that was how it worked, I was in a different dept). The traces must be continuous and robust so is normally done pen-plotter style. They were able to keep increasing the resolution of the laser rasterizing process to the point where it no longer mattered. I think now the default view is to work in pixels, voxels, etc. For all we know, the universe could work that way too. Like what's with dark energy? Space being created by a vacuum.

__mharrison__ 4y ago

Big data hates this article...