4 ms·
What I'd like to know is if the "irreproducibility crisis" is really some combination of "sample size was too small" and "effect size was too small". When I we
by arafa 8y ago
What I'd like to know is if the "irreproducibility crisis" is really some combination of "sample size was too small" and "effect size was too small". When I went through a lot of these studies myself and saw the ones that don't reproduce, I saw this theme over and over. "P-hacking" is less of a concern to me when the effect is real and widespread.
It's so bad now that for any article/study, I look at the sample size first. If it's too small (especially < 100) or they don't say I just ignore it. And if you don't publish or give some estimate of your effect size I just think about it directionally but don't give it much weight mentally.
- hnuser1234 8y agoI'm not sure this captures the whole problem. Sure, huge sample sizes are great, but look at the current top comment by lisper - we have gained a boatload (that's a scientific term) of insight from single events. LIGO is currently rocking a sample size of 7 but I doubt you ignored it.
- arafa 8y agoOf course there are exceptions and what I said doesn't capture all cases (like with massive fixed costs or unique kinds of studies), you're right. I find it's a pretty good rule of thumb, though.
- amatic 8y agoHuge sample studies with high stat significance can have very low effect sizes, though. If an effect is shown on a small sample, it means it is a really big effect, and will likely be reproduced in a bigger study.
- arafa 8y agoIt's a combination of sample size and effect size. The larger one is, the more I'm willing to give on the other.