5 ms·

I think what you're missing is that sample sizes for statistical significance do not depend on the population size. The big caveat is that the sample is random

by em500 3y ago

I think what you're missing is that sample sizes for statistical significance do not depend on the population size. The big caveat is that the sample is random and representative, which you can't really verify beyond doubt. But if a sample is random and representative, you could estimate population statistics of millions (or even infinite) with a sample of just 30 or so. As sibling posts mentioned, a much bigger issue than sample size is representativeness: college psychology study samples tend to be drawn disproportionately from college students. They might not generalize well to the general population even if you increase the sample size ten fold or a hundred fold.

p-e-w 3y ago
> But if a sample is random and representative, you could estimate population statistics of millions (or even infinite) with a sample of just 30 or so. While that statement is technically correct, it is useless in practice. In order to determine whether a sample is representative (of the trait you are trying to measure), you need to know what the true mean value of the trait is – but if you knew that, you wouldn't need to conduct the experiment in the first place. That's why it is important to have a large sample size, because under some very general assumptions, the distortions caused by outliers become smaller the more samples you look at. In the limit (that is, when the sample size equals the population size) those distortions disappear entirely, and the sample mean becomes the true population mean.
- mjburgess 3y ago
  To add to this, the assumptions behind any use of statistics here are wild when applied to pyschology. 1. We assume that people have stable measurable independent properties f1(x1.., t), f2(x1.., t), ... 2. We assume these are modelled by a linear model, f1(t) = ax + b, etc. 3. We assume the parameters (a, b) of this model causally determine `f1` st. each corresponds to an independent cause of `f1` 4. We assume that we have some reliable measurement process, say m, which measures each property producing samples of that a,b: m(f1) = random samples of a,b 5. Then we assume that these samples are representative of the true values, ie., that mean(a samples) = a + random_error The problem with the field is that (1 - 4) are false -- arguing about 5 is a little like arguing what time a broken clock should be set to.
  - cptaj 3y ago
    Psychology is a joke.
    - FrustratedMonky 3y ago
      String Theory is a joke.
      - throwaway426079 3y ago
        A psychologist and a physicist walk into a bar...
        datameta 3y ago
        The psychologist says, "I merely perceived walking into it". The physicist replies, "At the atomic level I did not, technically, touch it."
        kelipso 3y ago
        Ummm the bar doesn't actually exist.
  - startupsfail 3y ago
    I remember a talk at NIPS, a number of years back, where a psychologist was criticizing statistical approaches used by ML practitioners. And telling us that unlike ML practitioners (who still live in the dark ages), modern psychology researchers had realized their wrongs ways of the past. And now they register their studies and the statistical methodology upfront. They review it, register the experiments and only then conduct these, publishing both positive and negative results of the registered study.
    - mjburgess 3y ago
      The issue is that psychologists still do not understand the non-statistical premises of their statistical applications. Without a causal model of a human mind (and its relationship to the world) -- what exactly is there a "statistics" of? You cannot really have a scientific psychology; the necessary experiments are impossible or unethical.
- tmoertel 3y ago
  > > But if a sample is random and representative, you could estimate population statistics of millions (or even infinite) with a sample of just 30 or so. > While that statement is technically correct, it is useless in practice. I don't really follow your argument for small sample sizes being useless in practice. Isn't the whole point of things like confidence/credible intervals, the bootstrap, and other common statistical methods, to quantify the uncertainty of estimates due to the sampling procedure, including sample size and the possibility of outliers? And, while we may not know that our sample is representative, isn't random sampling from the population of interest a method that will generate a sample that is representative with high probability? That these methods are well known in statistical practice suggests that samples are generally useful, even when they are not large. The existence of pathological distributions where we might get misleading results (e.g., the classical "wealth over the people in a room if you throw Bill Gates in there") doesn't make these methods useless, does it?
  - p-e-w 3y ago
    > isn't random sampling from the population of interest a method that will generate a sample that is representative with high probability? That depends on the sample size. The comment I replied to claimed that "you could estimate population statistics of millions (or even infinite) with a sample of just 30 or so". Which is certainly not true if the sample is drawn randomly, except under very specific circumstances (such as population variance being extremely small). You don't need a billionaire in order for income distribution samples to be wildly misleading. It's enough for the variance to be high, so that very small samples are almost guaranteed to have a mean that differs substantially from that of the whole population.
    - tmoertel 3y ago
      Draw at random just six people from a population of millions. Take the min and max wealth of those six people. That procedure produces a confidence interval that contains the true median wealth of the population more than 95% of the time. Even if the population wealth variance is high. How did I just produce a reliable estimate using a sample size of only six? By quantifying the uncertainty of the estimate. There's a trade off between sample size and uncertainty, and that trade off allows small sample sizes to be useful for many practical applications, most of which can accommodate uncertainty. Again, small samples are informative. Whether they are informative enough for a particular application doesn't determine whether they are informative enough for all applications in general. I certainly wouldn't call them "useless in practice."
fragebogen 3y ago
good point! yes, agree with what you're saying, i think that was what i was alluding to. i didn't find the selection process in the article, but given that such a study takes a long time to carry through, isn't it absurdly risky to sample such a small group knowing that if it turns out decades later that the sample isn't really representative and you can't really infer anything about the general public?
rvba 3y ago
What is the sample? Psychology students and people who want to be paid 10 dollars to fill out a survey? People who agree to have a long survey via telephone? Doesnt sound representative for me at all

by em500 3y ago

p-e-w 3y ago

> But if a sample is random and representative, you could estimate population statistics of millions (or even infinite) with a sample of just 30 or so. While that statement is technically correct, it is useless in practice. In order to determine whether a sample is representative (of the trait you are trying to measure), you need to know what the true mean value of the trait is – but if you knew that, you wouldn't need to conduct the experiment in the first place. That's why it is important to have a large sample size, because under some very general assumptions, the distortions caused by outliers become smaller the more samples you look at. In the limit (that is, when the sample size equals the population size) those distortions disappear entirely, and the sample mean becomes the true population mean.

mjburgess 3y ago
To add to this, the assumptions behind any use of statistics here are wild when applied to pyschology. 1. We assume that people have stable measurable independent properties f1(x1.., t), f2(x1.., t), ... 2. We assume these are modelled by a linear model, f1(t) = ax + b, etc. 3. We assume the parameters (a, b) of this model causally determine `f1` st. each corresponds to an independent cause of `f1` 4. We assume that we have some reliable measurement process, say m, which measures each property producing samples of that a,b: m(f1) = random samples of a,b 5. Then we assume that these samples are representative of the true values, ie., that mean(a samples) = a + random_error The problem with the field is that (1 - 4) are false -- arguing about 5 is a little like arguing what time a broken clock should be set to.
- cptaj 3y ago
  Psychology is a joke.
  - FrustratedMonky 3y ago
    String Theory is a joke.
    - throwaway426079 3y ago
      A psychologist and a physicist walk into a bar...
      - datameta 3y ago
        The psychologist says, "I merely perceived walking into it". The physicist replies, "At the atomic level I did not, technically, touch it."
        kelipso 3y ago
        Ummm the bar doesn't actually exist.
- startupsfail 3y ago
  I remember a talk at NIPS, a number of years back, where a psychologist was criticizing statistical approaches used by ML practitioners. And telling us that unlike ML practitioners (who still live in the dark ages), modern psychology researchers had realized their wrongs ways of the past. And now they register their studies and the statistical methodology upfront. They review it, register the experiments and only then conduct these, publishing both positive and negative results of the registered study.
  - mjburgess 3y ago
    The issue is that psychologists still do not understand the non-statistical premises of their statistical applications. Without a causal model of a human mind (and its relationship to the world) -- what exactly is there a "statistics" of? You cannot really have a scientific psychology; the necessary experiments are impossible or unethical.
tmoertel 3y ago
> > But if a sample is random and representative, you could estimate population statistics of millions (or even infinite) with a sample of just 30 or so. > While that statement is technically correct, it is useless in practice. I don't really follow your argument for small sample sizes being useless in practice. Isn't the whole point of things like confidence/credible intervals, the bootstrap, and other common statistical methods, to quantify the uncertainty of estimates due to the sampling procedure, including sample size and the possibility of outliers? And, while we may not know that our sample is representative, isn't random sampling from the population of interest a method that will generate a sample that is representative with high probability? That these methods are well known in statistical practice suggests that samples are generally useful, even when they are not large. The existence of pathological distributions where we might get misleading results (e.g., the classical "wealth over the people in a room if you throw Bill Gates in there") doesn't make these methods useless, does it?
- p-e-w 3y ago
  > isn't random sampling from the population of interest a method that will generate a sample that is representative with high probability? That depends on the sample size. The comment I replied to claimed that "you could estimate population statistics of millions (or even infinite) with a sample of just 30 or so". Which is certainly not true if the sample is drawn randomly, except under very specific circumstances (such as population variance being extremely small). You don't need a billionaire in order for income distribution samples to be wildly misleading. It's enough for the variance to be high, so that very small samples are almost guaranteed to have a mean that differs substantially from that of the whole population.
  - tmoertel 3y ago
    Draw at random just six people from a population of millions. Take the min and max wealth of those six people. That procedure produces a confidence interval that contains the true median wealth of the population more than 95% of the time. Even if the population wealth variance is high. How did I just produce a reliable estimate using a sample size of only six? By quantifying the uncertainty of the estimate. There's a trade off between sample size and uncertainty, and that trade off allows small sample sizes to be useful for many practical applications, most of which can accommodate uncertainty. Again, small samples are informative. Whether they are informative enough for a particular application doesn't determine whether they are informative enough for all applications in general. I certainly wouldn't call them "useless in practice."

fragebogen 3y ago

good point! yes, agree with what you're saying, i think that was what i was alluding to. i didn't find the selection process in the article, but given that such a study takes a long time to carry through, isn't it absurdly risky to sample such a small group knowing that if it turns out decades later that the sample isn't really representative and you can't really infer anything about the general public?

rvba 3y ago

What is the sample? Psychology students and people who want to be paid 10 dollars to fill out a survey? People who agree to have a long survey via telephone? Doesnt sound representative for me at all