4 ms·

That's not entirely true and doesn't really convey the right message about p-values or study size. P-hacking and selection bias are definite concerns, but they

by sov 9y ago

That's not entirely true and doesn't really convey the right message about p-values or study size. P-hacking and selection bias are definite concerns, but they're also concerns for studies with 200, or more, participants. Rather, we should put more of an emphasis on the statistical power of the study. We're not yet in the beautiful golden world of pre-registered studies, but even high-n studies can have huge disparities due to bad statistical power (eg: The Control Group is Out of Control).

3pt14159 9y ago
I'm essentially right. Studies with less than 200 people are essentially useless. It is more complicated than I originally let on, but not so much more complicated to muck up the takeaway. Most of our junk science has to do with these low population studies or with studies simultaneously studying high numbers of attributes.
- sov 9y ago
  Err, I don't quite agree with your description. The takeaway from your post was that low-n is worthless and high-n is ineffable, full stop. The size of the trial has very little to do with whether or not we are to believe the results insofar as it affects its statistical power. The fact that p-hacking and selection bias exist doesn't immediately imply that low-n studies are wrong. Those aren't problems unique to small sample size studies. For sure, statistical power isn't the catch-all thing to examine either--rather, pre-registration of studies would alleviate a great deal of falsities, but none of any of this has to do strictly with low sample size trials as your post implies. Let's say the drug studied was a guaranteed 100% limb regenerative drug for amputees. Would you really require a 200 person study to prove that Examplinol successfully regenerates limbs? I hope the answer to that is "obviously not; it will be very clear whether or not it works with a low sample size." Of course, how would we change the study if it wasn't supposed to work in 100% of people? What if Sample Pharmaceuticals indicated that it only worked for 50% of people? Or 10%? Or 1%? Would it suffice to use 200 people each trial?
xaedes 9y ago
I had to look up what you meant by pre-registered studies. In case anyone else wonders: "Unlike traditional scientific publishing, in which manuscripts are peer reviewed only after studies have been completed, registered reports are reviewed before scientists collect data. If the scientific question and methods are deemed sound, the authors are then offered "in-principle acceptance" of their article, which virtually guarantees publication regardless of how the results turn out." https://www.theguardian.com/science/blog/2013/jun/05/trust-in-science-study-pre-registration https://www.theguardian.com/science/blog/2013/jun/05/trust-i...

by sov 9y ago

3pt14159 9y ago

I'm essentially right. Studies with less than 200 people are essentially useless. It is more complicated than I originally let on, but not so much more complicated to muck up the takeaway. Most of our junk science has to do with these low population studies or with studies simultaneously studying high numbers of attributes.

sov 9y ago
Err, I don't quite agree with your description. The takeaway from your post was that low-n is worthless and high-n is ineffable, full stop. The size of the trial has very little to do with whether or not we are to believe the results insofar as it affects its statistical power. The fact that p-hacking and selection bias exist doesn't immediately imply that low-n studies are wrong. Those aren't problems unique to small sample size studies. For sure, statistical power isn't the catch-all thing to examine either--rather, pre-registration of studies would alleviate a great deal of falsities, but none of any of this has to do strictly with low sample size trials as your post implies. Let's say the drug studied was a guaranteed 100% limb regenerative drug for amputees. Would you really require a 200 person study to prove that Examplinol successfully regenerates limbs? I hope the answer to that is "obviously not; it will be very clear whether or not it works with a low sample size." Of course, how would we change the study if it wasn't supposed to work in 100% of people? What if Sample Pharmaceuticals indicated that it only worked for 50% of people? Or 10%? Or 1%? Would it suffice to use 200 people each trial?

xaedes 9y ago

I had to look up what you meant by pre-registered studies. In case anyone else wonders: "Unlike traditional scientific publishing, in which manuscripts are peer reviewed only after studies have been completed, registered reports are reviewed before scientists collect data. If the scientific question and methods are deemed sound, the authors are then offered "in-principle acceptance" of their article, which virtually guarantees publication regardless of how the results turn out." https://www.theguardian.com/science/blog/2013/jun/05/trust-in-science-study-pre-registration https://www.theguardian.com/science/blog/2013/jun/05/trust-i...