4 ms·

It isn't. People talking about P values forget about p-hacking and selection bias. Any scientific study with less than 200 participants should be discarded and

It isn't. People talking about P values forget about p-hacking and selection bias. Any scientific study with less than 200 participants should be discarded and the failure to routinely do so has more to do with scientific laziness and the pro-profit incentive most journals have than anything else.

tomcooks 9y ago
Is that 200 a random value or is it the result of a study (with hopefully at least 200 partecipants)
QAPereo 9y ago
This is also the cause of rotating advice for decades, eg “eggs are/good/bad/good/bad/good” which is s feedback loop between terrible science, predatory media, and a credulous public.
canjobear 9y ago
The number of participants required to show an effect should be shown by power analysis. It depends a lot on the effect being investigated. An arbitrary number like 200 is not helpful.
sov 9y ago
That's not entirely true and doesn't really convey the right message about p-values or study size. P-hacking and selection bias are definite concerns, but they're also concerns for studies with 200, or more, participants. Rather, we should put more of an emphasis on the statistical power of the study. We're not yet in the beautiful golden world of pre-registered studies, but even high-n studies can have huge disparities due to bad statistical power (eg: The Control Group is Out of Control).
- 3pt14159 9y ago
  I'm essentially right. Studies with less than 200 people are essentially useless. It is more complicated than I originally let on, but not so much more complicated to muck up the takeaway. Most of our junk science has to do with these low population studies or with studies simultaneously studying high numbers of attributes.
  - sov 9y ago
    Err, I don't quite agree with your description. The takeaway from your post was that low-n is worthless and high-n is ineffable, full stop. The size of the trial has very little to do with whether or not we are to believe the results insofar as it affects its statistical power. The fact that p-hacking and selection bias exist doesn't immediately imply that low-n studies are wrong. Those aren't problems unique to small sample size studies. For sure, statistical power isn't the catch-all thing to examine either--rather, pre-registration of studies would alleviate a great deal of falsities, but none of any of this has to do strictly with low sample size trials as your post implies. Let's say the drug studied was a guaranteed 100% limb regenerative drug for amputees. Would you really require a 200 person study to prove that Examplinol successfully regenerates limbs? I hope the answer to that is "obviously not; it will be very clear whether or not it works with a low sample size." Of course, how would we change the study if it wasn't supposed to work in 100% of people? What if Sample Pharmaceuticals indicated that it only worked for 50% of people? Or 10%? Or 1%? Would it suffice to use 200 people each trial?
- xaedes 9y ago
  I had to look up what you meant by pre-registered studies. In case anyone else wonders: "Unlike traditional scientific publishing, in which manuscripts are peer reviewed only after studies have been completed, registered reports are reviewed before scientists collect data. If the scientific question and methods are deemed sound, the authors are then offered "in-principle acceptance" of their article, which virtually guarantees publication regardless of how the results turn out." https://www.theguardian.com/science/blog/2013/jun/05/trust-in-science-study-pre-registration https://www.theguardian.com/science/blog/2013/jun/05/trust-i...
function_seven 9y ago
> Any scientific study with less than 200 participants should be discarded ... Imagine you're investigating a drug for hair loss. You gather a group of 31 bald white men. 15 of them take a placebo and 16 take the experimental drug for 24 weeks. If, after 24 weeks, you find 1 of 15 controls with a full head of hair, and 14 of 16 test subjects with a full head of hair, is 31 too small a sample size?
- 3pt14159 9y ago
  Yes. Let me rephrase what you phrased: Imagine you're investigating a drug for hair loss, like 20000 other researchers hellbent on achieving their goals. You gather a group of 31 bald white men. 15 of them take a placebo and 16 take the experimental drug for 24 weeks, but because your random allocation put more extra bald people in your non-control group you fixed it by balancing the two groups. Your counterpart that didn't have the same problem, uh, didn't make any adjustments. If, after 24 weeks, you find 1 of 15 controls with a full head of hair by some function you define, and 14 of 16 test subjects with a full head of hair again by some function that you define oh and we didn't adjust for anything but of course in the case where the drugs didn't do anything we did because the non-control group were all above average in both age and some other random things like testosterone levels. Is 31 too small a sample size? 31 is always too small of a sample size. These scientific studies are almost always bullshit and we all know it and we'd do something about it if we really cared but we don't because nobody is incentivized to fix it.
  - sov 9y ago
    This has nothing directly to do with sample size. First, that's obviously not a rephrasing of what they said. They, very clearly, illustrated a valid case for a well-powered, low sample size study. They presuppose a case where 31 virtually identically-haired white men undergo a treatment and, voila, it works decidedly well. Second, you're begging the question. You state that the drug doesn't work, yet have concocted a scenario in which it clearly and explicitly works. Even in your "cheatsy" example, 15 of the 16 people go from "bald" to "full head of hair." If the "full head of hair" metric is wrong it has nothing to do with the sample size, rather, we'll find this by examining how the study partitioned groups, pre-trial measurements and its claims to have calculated effectiveness over placebo (ie: actually reading the study). Third, you've neglected statistical power (which is actually your argument here) for an incorrect statement about low-sample size bias. You state that the mechanism of action for this bias is a pre-partition between [truly bald, nearly haired] and then, given maybe a bit of luck, let good ol' father time take over and show, wow, the (useless) drug works! Maybe everyone experienced a 1.5x follicular density growth, but only because you've been able to presort the group do we see the non-placebo benefit more than the other... except this has nothing whatsoever to do specifically with sample size! If you can sort a trial with 31 people into 15 "truly bald" and 16 "near-haired" you bet you can sort a trial of 200 participants into 96 "truly balled" and 104 "nearly haired" or, heck, just don't even randomize your group selection at all! Or run the study 20 times and pick the successful one! Or any number of other tricks, which are real and meaningful and require someone to actually read the study to uncover instead of just looking at a clearly, very misunderstood, number.

tomcooks 9y ago

Is that 200 a random value or is it the result of a study (with hopefully at least 200 partecipants)

QAPereo 9y ago

This is also the cause of rotating advice for decades, eg “eggs are/good/bad/good/bad/good” which is s feedback loop between terrible science, predatory media, and a credulous public.

canjobear 9y ago

The number of participants required to show an effect should be shown by power analysis. It depends a lot on the effect being investigated. An arbitrary number like 200 is not helpful.

sov 9y ago

That's not entirely true and doesn't really convey the right message about p-values or study size. P-hacking and selection bias are definite concerns, but they're also concerns for studies with 200, or more, participants. Rather, we should put more of an emphasis on the statistical power of the study. We're not yet in the beautiful golden world of pre-registered studies, but even high-n studies can have huge disparities due to bad statistical power (eg: The Control Group is Out of Control).

3pt14159 9y ago
I'm essentially right. Studies with less than 200 people are essentially useless. It is more complicated than I originally let on, but not so much more complicated to muck up the takeaway. Most of our junk science has to do with these low population studies or with studies simultaneously studying high numbers of attributes.
- sov 9y ago
  Err, I don't quite agree with your description. The takeaway from your post was that low-n is worthless and high-n is ineffable, full stop. The size of the trial has very little to do with whether or not we are to believe the results insofar as it affects its statistical power. The fact that p-hacking and selection bias exist doesn't immediately imply that low-n studies are wrong. Those aren't problems unique to small sample size studies. For sure, statistical power isn't the catch-all thing to examine either--rather, pre-registration of studies would alleviate a great deal of falsities, but none of any of this has to do strictly with low sample size trials as your post implies. Let's say the drug studied was a guaranteed 100% limb regenerative drug for amputees. Would you really require a 200 person study to prove that Examplinol successfully regenerates limbs? I hope the answer to that is "obviously not; it will be very clear whether or not it works with a low sample size." Of course, how would we change the study if it wasn't supposed to work in 100% of people? What if Sample Pharmaceuticals indicated that it only worked for 50% of people? Or 10%? Or 1%? Would it suffice to use 200 people each trial?
xaedes 9y ago
I had to look up what you meant by pre-registered studies. In case anyone else wonders: "Unlike traditional scientific publishing, in which manuscripts are peer reviewed only after studies have been completed, registered reports are reviewed before scientists collect data. If the scientific question and methods are deemed sound, the authors are then offered "in-principle acceptance" of their article, which virtually guarantees publication regardless of how the results turn out." https://www.theguardian.com/science/blog/2013/jun/05/trust-in-science-study-pre-registration https://www.theguardian.com/science/blog/2013/jun/05/trust-i...

function_seven 9y ago

> Any scientific study with less than 200 participants should be discarded ... Imagine you're investigating a drug for hair loss. You gather a group of 31 bald white men. 15 of them take a placebo and 16 take the experimental drug for 24 weeks. If, after 24 weeks, you find 1 of 15 controls with a full head of hair, and 14 of 16 test subjects with a full head of hair, is 31 too small a sample size?

3pt14159 9y ago
Yes. Let me rephrase what you phrased: Imagine you're investigating a drug for hair loss, like 20000 other researchers hellbent on achieving their goals. You gather a group of 31 bald white men. 15 of them take a placebo and 16 take the experimental drug for 24 weeks, but because your random allocation put more extra bald people in your non-control group you fixed it by balancing the two groups. Your counterpart that didn't have the same problem, uh, didn't make any adjustments. If, after 24 weeks, you find 1 of 15 controls with a full head of hair by some function you define, and 14 of 16 test subjects with a full head of hair again by some function that you define oh and we didn't adjust for anything but of course in the case where the drugs didn't do anything we did because the non-control group were all above average in both age and some other random things like testosterone levels. Is 31 too small a sample size? 31 is always too small of a sample size. These scientific studies are almost always bullshit and we all know it and we'd do something about it if we really cared but we don't because nobody is incentivized to fix it.
- sov 9y ago
  This has nothing directly to do with sample size. First, that's obviously not a rephrasing of what they said. They, very clearly, illustrated a valid case for a well-powered, low sample size study. They presuppose a case where 31 virtually identically-haired white men undergo a treatment and, voila, it works decidedly well. Second, you're begging the question. You state that the drug doesn't work, yet have concocted a scenario in which it clearly and explicitly works. Even in your "cheatsy" example, 15 of the 16 people go from "bald" to "full head of hair." If the "full head of hair" metric is wrong it has nothing to do with the sample size, rather, we'll find this by examining how the study partitioned groups, pre-trial measurements and its claims to have calculated effectiveness over placebo (ie: actually reading the study). Third, you've neglected statistical power (which is actually your argument here) for an incorrect statement about low-sample size bias. You state that the mechanism of action for this bias is a pre-partition between [truly bald, nearly haired] and then, given maybe a bit of luck, let good ol' father time take over and show, wow, the (useless) drug works! Maybe everyone experienced a 1.5x follicular density growth, but only because you've been able to presort the group do we see the non-placebo benefit more than the other... except this has nothing whatsoever to do specifically with sample size! If you can sort a trial with 31 people into 15 "truly bald" and 16 "near-haired" you bet you can sort a trial of 200 participants into 96 "truly balled" and 104 "nearly haired" or, heck, just don't even randomize your group selection at all! Or run the study 20 times and pick the successful one! Or any number of other tricks, which are real and meaningful and require someone to actually read the study to uncover instead of just looking at a clearly, very misunderstood, number.