5 ms·

My statistics class at high school level taught the following: The number of samples you need is very difficult to calculate correctly, requiring deep analysis

My statistics class at high school level taught the following: The number of samples you need is very difficult to calculate correctly, requiring deep analysis of standard deviations and variances. But surprisingly, you can simply know you've reached large number status when over 10 items exist in each category. --------- Ex: when doing a heads vs tails coin flip experiment, you likely have a large number once you have over 10 heads and over 10 tails. No matter how biased the coin is. Or in this 'Lotto ticket' example, you have a large number of samples after gathering enough data to find over 10 Jackpot winners.

deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
jmount 3y ago
Very cool rule. I think you can justify it by approximating each category as an independent Poisson distribution. Then for each such processes the variance equals the mean. So once you have 10 successes in a bin, you have evidence of a probably good estimate for the arrival rate of that category. The book "The Probabilistic Method" calls a related idea "the Poisson paradigm." (10 a nice round number where the standard deviation is below the mean)
- jmount 3y ago
  Small proviso: this is only true for a reasonable number of categories (or you run into repeated experiment problems).
  - hgsgm 3y ago
    What's a reasonable number of categories? 10?
    - dredmorbius 3y ago
      That's defined by the phenomenon you're investigating. In the case of six-sided dice, there are precisely six categories, ideally with even odds of occurrence. With the lottery jackpot given, there are eight categories, with highly asymmetric probabilities and values. In real-world cases, you might be trying to distinguish two cases (treatment and control in a medical experiment), between multiple particles or isotopes (say, with physics or chemistry), amongst different political divisions (countries, states or provinces, counties, cities, or other), between political parties or candidates (which raises interesting questions over which and/or how many to include in consideration, in turn dependent on voting procedures, overall popularity, and impacts of non-winning candidates or parties on others), on multiple products, or on different behavioural characteristics in some domain (e.g., highly-active, occasionally-active, and lurking participants in online fora). There are times when categories are well and unambiguously defined. Others in which where you choose to draw divisions (say, in generational groups, or wealth or income brackets) is highly arbitrary. Even where there are a large number of potential categories, choosing some limited number for specific analysis (2, 3, 5, 10, etc.) and lumping the remaining into "other" may provide clearer insights and fewer distractions than choosing a large number of divisions.[1] In other cases, a very small number of individuals may account for an overwhelming majority of activity or outcome. I'd strongly argue that in this case, the analysis might be somewhat poorly focused, and that activities and outcomes rather than individuals are of greater interest.[2] What's key is to match your sampling and sample sizes to the phenomenon being studied. ________________________________ Notes: 1. Power law distribution / Zipf functions often mean that a very small number of participants has highly disproportionate impact or significance. 2. This is often the flip side of power law distributions. If we look at all book titles, there are a huge number of individual items to consider; there are roughly 300k annual English-language "traditional" publications, and over 1 million "nontraditional" (self-published, or publish-on-demand) titles. But if your focus is instead titles by percentage of revenue or number of sales, a top-n analysis (5, 10, 20, etc.) often captures much of the activity, frequently well over half. This is typical of any informational good: music, cinema, blogs, social media posts, etc.
  - Aerroon 3y ago
    My intuition is telling me that a coin flip would be the worst case scenario, because I think it would be the easiest to hit 10 examples in both categories. Every other mix of probabilities I can come up with would average more rolls of RNG than a coinflip. Am I mistaken?
tgv 3y ago
For heads or tails, that leaves a very large margin. In approx. 1 in 20 trials, you'll end up with a 10-20 split.
- dragontamer 3y ago
  Yeah, 95% confidence ratio (or approximately two standard deviations) is pretty standard with regards to statistical tests. You gotta draw the line somewhere. At high-school statistics level, its basically universally drawn at the 95% confidence level. If you wanna draw new lines elsewhere, you gotta make new rules yourself and recalculate all the rules of thumb.
  - User23 3y ago
    I remember my high school AP Psychology teacher mocking p=0.05 as practically meaningless. In retrospect it's funny for a psychologist to say that, but I guess it was because he was from the more empirically minded behaviorist cognitive school and from time to time they have done actual rigorous experiments[1] (in rodents). [1] For example as described by Feynman in Cargo Cult Science.
    - lisper 3y ago
      The problem is two-fold: 1. p=0.05 means that one result in 20 is going to be the result of chance. 2. It's generally pretty easy (especially in psychology) to do 20 experiments, cherry-pick -- and publish! -- the p=0.05 result, and throw away the others. The result is that published p=0.05 results are much more likely than 1 in 20 to be the result of chance.
      - dragontamer 3y ago
        So run a meta-study upon the results published by a set of authors and double-check to make sure that their results are normally distributed across the p-values associated with their studies. These problems are solved problems in the scientific community. Just announce that regular meta-studies will be done, expectations for authors to be normally distributed is published, and publicly show off the meta-study. ------------- In any case, the discussion point you're making is well beyond the high-school level needed for a general education. If someone needs to run their own experiment (A/B testing upon their website) and cannot afford a proper set of tests/statistics, they should instead rely upon high-school level heuristics to design their personal studies. This isn't a level of study about analyzing other people's results and finding flaws in other people's (possibly maliciously seeded) results. This is a heuristic about how to run your own experiments and how to prove something to yourself at a 95% confidence level. If you want to get published in the scientific community, the level of rigor is much higher of course, but no one tries to publish a scientific paper on just a high school education (which is where I was aiming my original comment at).
NelsonMinar 3y ago
That's a neat rule of thumb; is there a simple statistical argument for why 10 is the (not very large) number?
fiddlerwoaroof 3y ago
Does this work the other way? I.e. “you have enough buckets if adding one more puts the number of samples in the smallest bucket below 10?”

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

jmount 3y ago

Very cool rule. I think you can justify it by approximating each category as an independent Poisson distribution. Then for each such processes the variance equals the mean. So once you have 10 successes in a bin, you have evidence of a probably good estimate for the arrival rate of that category. The book "The Probabilistic Method" calls a related idea "the Poisson paradigm." (10 a nice round number where the standard deviation is below the mean)

jmount 3y ago
Small proviso: this is only true for a reasonable number of categories (or you run into repeated experiment problems).
- hgsgm 3y ago
  What's a reasonable number of categories? 10?
  - dredmorbius 3y ago
    That's defined by the phenomenon you're investigating. In the case of six-sided dice, there are precisely six categories, ideally with even odds of occurrence. With the lottery jackpot given, there are eight categories, with highly asymmetric probabilities and values. In real-world cases, you might be trying to distinguish two cases (treatment and control in a medical experiment), between multiple particles or isotopes (say, with physics or chemistry), amongst different political divisions (countries, states or provinces, counties, cities, or other), between political parties or candidates (which raises interesting questions over which and/or how many to include in consideration, in turn dependent on voting procedures, overall popularity, and impacts of non-winning candidates or parties on others), on multiple products, or on different behavioural characteristics in some domain (e.g., highly-active, occasionally-active, and lurking participants in online fora). There are times when categories are well and unambiguously defined. Others in which where you choose to draw divisions (say, in generational groups, or wealth or income brackets) is highly arbitrary. Even where there are a large number of potential categories, choosing some limited number for specific analysis (2, 3, 5, 10, etc.) and lumping the remaining into "other" may provide clearer insights and fewer distractions than choosing a large number of divisions.[1] In other cases, a very small number of individuals may account for an overwhelming majority of activity or outcome. I'd strongly argue that in this case, the analysis might be somewhat poorly focused, and that activities and outcomes rather than individuals are of greater interest.[2] What's key is to match your sampling and sample sizes to the phenomenon being studied. ________________________________ Notes: 1. Power law distribution / Zipf functions often mean that a very small number of participants has highly disproportionate impact or significance. 2. This is often the flip side of power law distributions. If we look at all book titles, there are a huge number of individual items to consider; there are roughly 300k annual English-language "traditional" publications, and over 1 million "nontraditional" (self-published, or publish-on-demand) titles. But if your focus is instead titles by percentage of revenue or number of sales, a top-n analysis (5, 10, 20, etc.) often captures much of the activity, frequently well over half. This is typical of any informational good: music, cinema, blogs, social media posts, etc.
- Aerroon 3y ago
  My intuition is telling me that a coin flip would be the worst case scenario, because I think it would be the easiest to hit 10 examples in both categories. Every other mix of probabilities I can come up with would average more rolls of RNG than a coinflip. Am I mistaken?

tgv 3y ago

For heads or tails, that leaves a very large margin. In approx. 1 in 20 trials, you'll end up with a 10-20 split.

dragontamer 3y ago
Yeah, 95% confidence ratio (or approximately two standard deviations) is pretty standard with regards to statistical tests. You gotta draw the line somewhere. At high-school statistics level, its basically universally drawn at the 95% confidence level. If you wanna draw new lines elsewhere, you gotta make new rules yourself and recalculate all the rules of thumb.
- User23 3y ago
  I remember my high school AP Psychology teacher mocking p=0.05 as practically meaningless. In retrospect it's funny for a psychologist to say that, but I guess it was because he was from the more empirically minded behaviorist cognitive school and from time to time they have done actual rigorous experiments[1] (in rodents). [1] For example as described by Feynman in Cargo Cult Science.
  - lisper 3y ago
    The problem is two-fold: 1. p=0.05 means that one result in 20 is going to be the result of chance. 2. It's generally pretty easy (especially in psychology) to do 20 experiments, cherry-pick -- and publish! -- the p=0.05 result, and throw away the others. The result is that published p=0.05 results are much more likely than 1 in 20 to be the result of chance.
    - dragontamer 3y ago
      So run a meta-study upon the results published by a set of authors and double-check to make sure that their results are normally distributed across the p-values associated with their studies. These problems are solved problems in the scientific community. Just announce that regular meta-studies will be done, expectations for authors to be normally distributed is published, and publicly show off the meta-study. ------------- In any case, the discussion point you're making is well beyond the high-school level needed for a general education. If someone needs to run their own experiment (A/B testing upon their website) and cannot afford a proper set of tests/statistics, they should instead rely upon high-school level heuristics to design their personal studies. This isn't a level of study about analyzing other people's results and finding flaws in other people's (possibly maliciously seeded) results. This is a heuristic about how to run your own experiments and how to prove something to yourself at a 95% confidence level. If you want to get published in the scientific community, the level of rigor is much higher of course, but no one tries to publish a scientific paper on just a high school education (which is where I was aiming my original comment at).

NelsonMinar 3y ago

That's a neat rule of thumb; is there a simple statistical argument for why 10 is the (not very large) number?

fiddlerwoaroof 3y ago

Does this work the other way? I.e. “you have enough buckets if adding one more puts the number of samples in the smallest bucket below 10?”