4 ms·

Can't you just pre-register for 20 studies of random correlations and still get one p>0.05?

by Iv 5y ago

funklute 5y ago
You can, but that's not necessarily problematic in itself (perhaps slightly depending on what your aim is). By definition, using a statistical significance threshold of 0.05 means that you will have 5% false positive rate. You always expect that _some_ results are erronous (either false positive or false negatives), and the point of using p-values is merely to control the rate at which this happens. That said, you can do something called multiple comparisons corrections, where you take into account the fact that you're doing multiple tests in the same go. In this case you'll end up being more conservative, and it's not the false positive rate you'll be controlling - instead you'll control the family-wise error rate, which basically has to do with whether _any_ of your positive results are false positives.
laGrenouille 5y ago
The idea is that preregistration is done publicly, so a reviewer of a paper could easily see that you have registered multiple analyses and reject the research based on this.
- magicalhippo 5y ago
  Can't it be compensated for[1] instead? [1]: https://en.wikipedia.org/wiki/Look-elsewhere_effect https://en.wikipedia.org/wiki/Look-elsewhere_effect
  - laGrenouille 5y ago
    Yes, that's entirely possible. My comment was made keeping in mind that the scientific norm is to have a single preregistration for a single experiment. If a researcher wants to test N hypothesis, using an appropriate p-value correction to account for this, usually the entire set of hypotheses and the correction itself would be contained in a single registration. And that would be perfectly fine. Submiting a large number of individual registrations for a single set of data is just not the norm and so it would be suspicious to begin with. (Okay, I will admit that on the order of 2-3 registrations for very different categories of study may not be as strange; but certainly 20 would signal something very odd).

Can't you just pre-register for 20 studies of random correlations and still get one p>0.05?

by Iv 5y ago

Can't you just pre-register for 20 studies of random correlations and still get one p>0.05?

You can, but that's not necessarily problematic in itself (perhaps slightly depending on what your aim is). By definition, using a statistical significance threshold of 0.05 means that you will have 5% false positive rate. You always expect that _some_ results are erronous (either false positive or false negatives), and the point of using p-values is merely to control the rate at which this happens. That said, you can do something called multiple comparisons corrections, where you take into account the fact that you're doing multiple tests in the same go. In this case you'll end up being more conservative, and it's not the false positive rate you'll be controlling - instead you'll control the family-wise error rate, which basically has to do with whether _any_ of your positive results are false positives.

laGrenouille 5y ago

The idea is that preregistration is done publicly, so a reviewer of a paper could easily see that you have registered multiple analyses and reject the research based on this.

magicalhippo 5y ago
Can't it be compensated for[1] instead? [1]: https://en.wikipedia.org/wiki/Look-elsewhere_effect https://en.wikipedia.org/wiki/Look-elsewhere_effect
- laGrenouille 5y ago
  Yes, that's entirely possible. My comment was made keeping in mind that the scientific norm is to have a single preregistration for a single experiment. If a researcher wants to test N hypothesis, using an appropriate p-value correction to account for this, usually the entire set of hypotheses and the correction itself would be contained in a single registration. And that would be perfectly fine. Submiting a large number of individual registrations for a single set of data is just not the norm and so it would be suspicious to begin with. (Okay, I will admit that on the order of 2-3 registrations for very different categories of study may not be as strange; but certainly 20 would signal something very odd).