7 ms·

number of samples = 92 hard for me believe that you would infer anything statistically meaningful on population scale from that small of a sample, or am i miss

number of samples = 92 hard for me believe that you would infer anything statistically meaningful on population scale from that small of a sample, or am i missing something? is it common to work with so small sample sizes in psychology?

cubefox 3y ago
It's a meta-analysis, so they analyzed 92 studies.
- spangry 3y ago
  Yep, the article confirms: "Ninety-two studies satisfied the inclusion criteria. Because many of these studies reported data from several samples, the number of samples, 113, was greater than the total number of studies. The total number of participants for the 113 samples was 50,120. A total of 1,682 estimates of change were compiled."
- fragebogen 3y ago
  thanks! that wasn't clear to me.
adhesive_wombat 3y ago
"Our study was inconclusive: one third of patients were cured, one third died and the other guy got away".
just_a_monkfish 3y ago
[dead]
moi2388 3y ago
Yes it is. There are quite some studies where the sample size is 20 or 30 people per group, and sometimes even 15-20 in total. Combina that with the fact that they often use psychology students as experiment group and the fact that most research doesn’t go further than assume a normal distribution, and you might start to understand why more than 2/3rds of all psychology research cannot be reproduced.
em500 3y ago
I think what you're missing is that sample sizes for statistical significance do not depend on the population size. The big caveat is that the sample is random and representative, which you can't really verify beyond doubt. But if a sample is random and representative, you could estimate population statistics of millions (or even infinite) with a sample of just 30 or so. As sibling posts mentioned, a much bigger issue than sample size is representativeness: college psychology study samples tend to be drawn disproportionately from college students. They might not generalize well to the general population even if you increase the sample size ten fold or a hundred fold.
- p-e-w 3y ago
  > But if a sample is random and representative, you could estimate population statistics of millions (or even infinite) with a sample of just 30 or so. While that statement is technically correct, it is useless in practice. In order to determine whether a sample is representative (of the trait you are trying to measure), you need to know what the true mean value of the trait is – but if you knew that, you wouldn't need to conduct the experiment in the first place. That's why it is important to have a large sample size, because under some very general assumptions, the distortions caused by outliers become smaller the more samples you look at. In the limit (that is, when the sample size equals the population size) those distortions disappear entirely, and the sample mean becomes the true population mean.
  - mjburgess 3y ago
    To add to this, the assumptions behind any use of statistics here are wild when applied to pyschology. 1. We assume that people have stable measurable independent properties f1(x1.., t), f2(x1.., t), ... 2. We assume these are modelled by a linear model, f1(t) = ax + b, etc. 3. We assume the parameters (a, b) of this model causally determine `f1` st. each corresponds to an independent cause of `f1` 4. We assume that we have some reliable measurement process, say m, which measures each property producing samples of that a,b: m(f1) = random samples of a,b 5. Then we assume that these samples are representative of the true values, ie., that mean(a samples) = a + random_error The problem with the field is that (1 - 4) are false -- arguing about 5 is a little like arguing what time a broken clock should be set to.
p-e-w 3y ago
> is it common to work with so small sample sizes in psychology? Yes. Even much smaller sample sizes are common. I've seen "results" published in supposedly reputable journals with sample sizes as small as 9. Not 9 other studies that were meta-analyzed, mind you. 9 individual people. This is just one of several reasons why most of psychology is junk science, if not outright fraudulent. Statistical methods is usually a required course for students in such disciplines, so I do not buy the "they don't know better" excuse. Not that you need to know statistics in order to understand that you can't conclude anything from a sample size of 9.
- hackernewds 3y ago
  Psychology isn't accepted as a hard science, perhaps an applied science. Since there really isn't a precise follow to the scientific method, hypothesis - inference - conclusion.
  - FrustratedMonky 3y ago
    So Astronomy is not a hard science? Because there is only observation, no testing of hypothesis?
    - naillo 3y ago
      They can test it in the limited sense of simulation. Not ideal but it's more than even psychologists can do
      - FrustratedMonky 3y ago
        That is so completely false that I fear you are responding purely by a bias against psychology and have not read anything about it. Of course you can form a hypothesis, and test it on real live humans, gather data and check results.
  - willcipriano 3y ago
    I don't think a lot of people know this. The evidence required for a physiological intervention is lower than a surgical or pharmaceutical intervention. The AMA used to lump psychology with things like chiropractic, acupuncture and homeopathy until they were sued by chiropractors under antitrust law in 1987.
lamename 3y ago
It's a meta-analysis, which means they evaluated 92 papers in the literature.
williamcotton 3y ago
There are many reasons that qualitative research is better than quantitative research. Understanding context and nuance, exploratory studies, complex phenomena, and social dynamics and interactions are more often better served by having fewer subjects.
stevenhuang 3y ago
Never only look at sample size. Something can be statistically meaningful with a small sample size if the effect size is large for instance.
fzeindl 3y ago
While this is a meta analysis in general I suspect it is not easy to get and retain study participants for decade-long studies.
jschveibinz 3y ago
You think that’s small? Try reading a paper in education and learning. Some of the seminal papers have less than 30 subjects. And these papers have driven teaching policies for decades.

cubefox 3y ago

It's a meta-analysis, so they analyzed 92 studies.

spangry 3y ago
Yep, the article confirms: "Ninety-two studies satisfied the inclusion criteria. Because many of these studies reported data from several samples, the number of samples, 113, was greater than the total number of studies. The total number of participants for the 113 samples was 50,120. A total of 1,682 estimates of change were compiled."
fragebogen 3y ago
thanks! that wasn't clear to me.

adhesive_wombat 3y ago

"Our study was inconclusive: one third of patients were cured, one third died and the other guy got away".

just_a_monkfish 3y ago

[dead]

moi2388 3y ago

Yes it is. There are quite some studies where the sample size is 20 or 30 people per group, and sometimes even 15-20 in total. Combina that with the fact that they often use psychology students as experiment group and the fact that most research doesn’t go further than assume a normal distribution, and you might start to understand why more than 2/3rds of all psychology research cannot be reproduced.

em500 3y ago

I think what you're missing is that sample sizes for statistical significance do not depend on the population size. The big caveat is that the sample is random and representative, which you can't really verify beyond doubt. But if a sample is random and representative, you could estimate population statistics of millions (or even infinite) with a sample of just 30 or so. As sibling posts mentioned, a much bigger issue than sample size is representativeness: college psychology study samples tend to be drawn disproportionately from college students. They might not generalize well to the general population even if you increase the sample size ten fold or a hundred fold.

p-e-w 3y ago
> But if a sample is random and representative, you could estimate population statistics of millions (or even infinite) with a sample of just 30 or so. While that statement is technically correct, it is useless in practice. In order to determine whether a sample is representative (of the trait you are trying to measure), you need to know what the true mean value of the trait is – but if you knew that, you wouldn't need to conduct the experiment in the first place. That's why it is important to have a large sample size, because under some very general assumptions, the distortions caused by outliers become smaller the more samples you look at. In the limit (that is, when the sample size equals the population size) those distortions disappear entirely, and the sample mean becomes the true population mean.
- mjburgess 3y ago
  To add to this, the assumptions behind any use of statistics here are wild when applied to pyschology. 1. We assume that people have stable measurable independent properties f1(x1.., t), f2(x1.., t), ... 2. We assume these are modelled by a linear model, f1(t) = ax + b, etc. 3. We assume the parameters (a, b) of this model causally determine `f1` st. each corresponds to an independent cause of `f1` 4. We assume that we have some reliable measurement process, say m, which measures each property producing samples of that a,b: m(f1) = random samples of a,b 5. Then we assume that these samples are representative of the true values, ie., that mean(a samples) = a + random_error The problem with the field is that (1 - 4) are false -- arguing about 5 is a little like arguing what time a broken clock should be set to.

p-e-w 3y ago

> is it common to work with so small sample sizes in psychology? Yes. Even much smaller sample sizes are common. I've seen "results" published in supposedly reputable journals with sample sizes as small as 9. Not 9 other studies that were meta-analyzed, mind you. 9 individual people. This is just one of several reasons why most of psychology is junk science, if not outright fraudulent. Statistical methods is usually a required course for students in such disciplines, so I do not buy the "they don't know better" excuse. Not that you need to know statistics in order to understand that you can't conclude anything from a sample size of 9.

hackernewds 3y ago
Psychology isn't accepted as a hard science, perhaps an applied science. Since there really isn't a precise follow to the scientific method, hypothesis - inference - conclusion.
- FrustratedMonky 3y ago
  So Astronomy is not a hard science? Because there is only observation, no testing of hypothesis?
  - naillo 3y ago
    They can test it in the limited sense of simulation. Not ideal but it's more than even psychologists can do
    - FrustratedMonky 3y ago
      That is so completely false that I fear you are responding purely by a bias against psychology and have not read anything about it. Of course you can form a hypothesis, and test it on real live humans, gather data and check results.
- willcipriano 3y ago
  I don't think a lot of people know this. The evidence required for a physiological intervention is lower than a surgical or pharmaceutical intervention. The AMA used to lump psychology with things like chiropractic, acupuncture and homeopathy until they were sued by chiropractors under antitrust law in 1987.

lamename 3y ago

It's a meta-analysis, which means they evaluated 92 papers in the literature.

williamcotton 3y ago

There are many reasons that qualitative research is better than quantitative research. Understanding context and nuance, exploratory studies, complex phenomena, and social dynamics and interactions are more often better served by having fewer subjects.

stevenhuang 3y ago

Never only look at sample size. Something can be statistically meaningful with a small sample size if the effect size is large for instance.

fzeindl 3y ago

While this is a meta analysis in general I suspect it is not easy to get and retain study participants for decade-long studies.

jschveibinz 3y ago

You think that’s small? Try reading a paper in education and learning. Some of the seminal papers have less than 30 subjects. And these papers have driven teaching policies for decades.