5 ms·

> Any scientific study with less than 200 participants should be discarded ... Imagine you're investigating a drug for hair loss. You gather a group of 31 bald

> Any scientific study with less than 200 participants should be discarded ... Imagine you're investigating a drug for hair loss. You gather a group of 31 bald white men. 15 of them take a placebo and 16 take the experimental drug for 24 weeks. If, after 24 weeks, you find 1 of 15 controls with a full head of hair, and 14 of 16 test subjects with a full head of hair, is 31 too small a sample size?

3pt14159 9y ago
Yes. Let me rephrase what you phrased: Imagine you're investigating a drug for hair loss, like 20000 other researchers hellbent on achieving their goals. You gather a group of 31 bald white men. 15 of them take a placebo and 16 take the experimental drug for 24 weeks, but because your random allocation put more extra bald people in your non-control group you fixed it by balancing the two groups. Your counterpart that didn't have the same problem, uh, didn't make any adjustments. If, after 24 weeks, you find 1 of 15 controls with a full head of hair by some function you define, and 14 of 16 test subjects with a full head of hair again by some function that you define oh and we didn't adjust for anything but of course in the case where the drugs didn't do anything we did because the non-control group were all above average in both age and some other random things like testosterone levels. Is 31 too small a sample size? 31 is always too small of a sample size. These scientific studies are almost always bullshit and we all know it and we'd do something about it if we really cared but we don't because nobody is incentivized to fix it.
- sov 9y ago
  This has nothing directly to do with sample size. First, that's obviously not a rephrasing of what they said. They, very clearly, illustrated a valid case for a well-powered, low sample size study. They presuppose a case where 31 virtually identically-haired white men undergo a treatment and, voila, it works decidedly well. Second, you're begging the question. You state that the drug doesn't work, yet have concocted a scenario in which it clearly and explicitly works. Even in your "cheatsy" example, 15 of the 16 people go from "bald" to "full head of hair." If the "full head of hair" metric is wrong it has nothing to do with the sample size, rather, we'll find this by examining how the study partitioned groups, pre-trial measurements and its claims to have calculated effectiveness over placebo (ie: actually reading the study). Third, you've neglected statistical power (which is actually your argument here) for an incorrect statement about low-sample size bias. You state that the mechanism of action for this bias is a pre-partition between [truly bald, nearly haired] and then, given maybe a bit of luck, let good ol' father time take over and show, wow, the (useless) drug works! Maybe everyone experienced a 1.5x follicular density growth, but only because you've been able to presort the group do we see the non-placebo benefit more than the other... except this has nothing whatsoever to do specifically with sample size! If you can sort a trial with 31 people into 15 "truly bald" and 16 "near-haired" you bet you can sort a trial of 200 participants into 96 "truly balled" and 104 "nearly haired" or, heck, just don't even randomize your group selection at all! Or run the study 20 times and pick the successful one! Or any number of other tricks, which are real and meaningful and require someone to actually read the study to uncover instead of just looking at a clearly, very misunderstood, number.
- function_seven 9y ago
  You had to introduce a lot of assumptions to invalidate the n=31. So let me lay it out in more detail. I recruited 31 white men who self-report having no hair on at least 500cm^2 of their head. They all had hair as adolescents, and lost it gradually throughout their 20s or early 30s. They're all at least 37 now. I then sort them into the two groups by assigning sequential IDs in random order. Those with an odd ID are in Group A, those with an even ID are in Group B. I then give each participant in both groups a placebo for 24 weeks, having them check in every 2 weeks to measure hair growth, if any. At 24 weeks, I randomly assign two new courses of treatment to each group. I do this blindly, giving Group A pill X and Group B pill Y. Another 24 weeks (with check-ins every 2 weeks) and I find that Group A (taking pill X) all have full heads of hair. Well, except for Paul and Maury. Meanwhile, Tom, in Group B (who took pill Y), is the only member of his group to have a full head of hair. Statistically-speaking, my n=31 was a great sample size, and I'm also pretty sure Tom discovered he had sugar pills and stole the real stuff from Paul or Maury. :) To address the possibility of seeing these results due to a confounding variable (like above average testosterone), this is about as likely as shuffling a deck of 31 cards, and all the red ones ending up in the front and all the black ones in the back. About 0.0000003%
  - jcranberry 9y ago
    Ah, but you did not precisely define the area in cm^2 of hair which constitutes a full head of hair vs bald!???!? Clearly this is a sham and you needed at least 31^47 participants in this study you charlatan!
    - function_seven 9y ago
      Shit. I just rechecked my data, and it appears the "full-head-of-hair" group still had 482cm² of shiny scalp. I need to fire my measurement tech and issue a recall...
  - carlmr 9y ago
    The point is though, that if enough people do the same study, and only the positive results are published, you're increasing the odds a lot of getting a positive result from someone. That's why companies often finance studies in their drugs. If enough people study their drugs, something which doesn't have a measureable effect, might have a measureable effect in some of these studies. At which sample size this becomes reasonable is a good question though.

3pt14159 9y ago

Yes. Let me rephrase what you phrased: Imagine you're investigating a drug for hair loss, like 20000 other researchers hellbent on achieving their goals. You gather a group of 31 bald white men. 15 of them take a placebo and 16 take the experimental drug for 24 weeks, but because your random allocation put more extra bald people in your non-control group you fixed it by balancing the two groups. Your counterpart that didn't have the same problem, uh, didn't make any adjustments. If, after 24 weeks, you find 1 of 15 controls with a full head of hair by some function you define, and 14 of 16 test subjects with a full head of hair again by some function that you define oh and we didn't adjust for anything but of course in the case where the drugs didn't do anything we did because the non-control group were all above average in both age and some other random things like testosterone levels. Is 31 too small a sample size? 31 is always too small of a sample size. These scientific studies are almost always bullshit and we all know it and we'd do something about it if we really cared but we don't because nobody is incentivized to fix it.

sov 9y ago
This has nothing directly to do with sample size. First, that's obviously not a rephrasing of what they said. They, very clearly, illustrated a valid case for a well-powered, low sample size study. They presuppose a case where 31 virtually identically-haired white men undergo a treatment and, voila, it works decidedly well. Second, you're begging the question. You state that the drug doesn't work, yet have concocted a scenario in which it clearly and explicitly works. Even in your "cheatsy" example, 15 of the 16 people go from "bald" to "full head of hair." If the "full head of hair" metric is wrong it has nothing to do with the sample size, rather, we'll find this by examining how the study partitioned groups, pre-trial measurements and its claims to have calculated effectiveness over placebo (ie: actually reading the study). Third, you've neglected statistical power (which is actually your argument here) for an incorrect statement about low-sample size bias. You state that the mechanism of action for this bias is a pre-partition between [truly bald, nearly haired] and then, given maybe a bit of luck, let good ol' father time take over and show, wow, the (useless) drug works! Maybe everyone experienced a 1.5x follicular density growth, but only because you've been able to presort the group do we see the non-placebo benefit more than the other... except this has nothing whatsoever to do specifically with sample size! If you can sort a trial with 31 people into 15 "truly bald" and 16 "near-haired" you bet you can sort a trial of 200 participants into 96 "truly balled" and 104 "nearly haired" or, heck, just don't even randomize your group selection at all! Or run the study 20 times and pick the successful one! Or any number of other tricks, which are real and meaningful and require someone to actually read the study to uncover instead of just looking at a clearly, very misunderstood, number.
function_seven 9y ago
You had to introduce a lot of assumptions to invalidate the n=31. So let me lay it out in more detail. I recruited 31 white men who self-report having no hair on at least 500cm^2 of their head. They all had hair as adolescents, and lost it gradually throughout their 20s or early 30s. They're all at least 37 now. I then sort them into the two groups by assigning sequential IDs in random order. Those with an odd ID are in Group A, those with an even ID are in Group B. I then give each participant in both groups a placebo for 24 weeks, having them check in every 2 weeks to measure hair growth, if any. At 24 weeks, I randomly assign two new courses of treatment to each group. I do this blindly, giving Group A pill X and Group B pill Y. Another 24 weeks (with check-ins every 2 weeks) and I find that Group A (taking pill X) all have full heads of hair. Well, except for Paul and Maury. Meanwhile, Tom, in Group B (who took pill Y), is the only member of his group to have a full head of hair. Statistically-speaking, my n=31 was a great sample size, and I'm also pretty sure Tom discovered he had sugar pills and stole the real stuff from Paul or Maury. :) To address the possibility of seeing these results due to a confounding variable (like above average testosterone), this is about as likely as shuffling a deck of 31 cards, and all the red ones ending up in the front and all the black ones in the back. About 0.0000003%
- jcranberry 9y ago
  Ah, but you did not precisely define the area in cm^2 of hair which constitutes a full head of hair vs bald!???!? Clearly this is a sham and you needed at least 31^47 participants in this study you charlatan!
  - function_seven 9y ago
    Shit. I just rechecked my data, and it appears the "full-head-of-hair" group still had 482cm² of shiny scalp. I need to fire my measurement tech and issue a recall...
- carlmr 9y ago
  The point is though, that if enough people do the same study, and only the positive results are published, you're increasing the odds a lot of getting a positive result from someone. That's why companies often finance studies in their drugs. If enough people study their drugs, something which doesn't have a measureable effect, might have a measureable effect in some of these studies. At which sample size this becomes reasonable is a good question though.