4 ms·

Preregistration of studies [0]. Submit design and models (e.g., logistic regression with these covariates and these exclusion criteria) before collecting data.

Preregistration of studies [0]. Submit design and models (e.g., logistic regression with these covariates and these exclusion criteria) before collecting data. This is relatively easy to implement and hard to get around, but does need to be paired with more acceptance of the importance of null results in advancing science. [0] https://en.wikipedia.org/wiki/Preregistration_(science) https://en.wikipedia.org/wiki/Preregistration_(science)

Icathian 6y ago
This is the right answer. I worked in clinical trials for a while, and they're required to submit a Statistical Analysis Plan before the study is even allowed to proceed. After all the data are collected, you just run whatever was originally proposed and you get the answer you get. This really should be the standard for any meaningful science.
Iv 6y ago
Can't you just pre-register for 20 studies of random correlations and still get one p>0.05?
- funklute 6y ago
  You can, but that's not necessarily problematic in itself (perhaps slightly depending on what your aim is). By definition, using a statistical significance threshold of 0.05 means that you will have 5% false positive rate. You always expect that _some_ results are erronous (either false positive or false negatives), and the point of using p-values is merely to control the rate at which this happens. That said, you can do something called multiple comparisons corrections, where you take into account the fact that you're doing multiple tests in the same go. In this case you'll end up being more conservative, and it's not the false positive rate you'll be controlling - instead you'll control the family-wise error rate, which basically has to do with whether _any_ of your positive results are false positives.
- laGrenouille 6y ago
  The idea is that preregistration is done publicly, so a reviewer of a paper could easily see that you have registered multiple analyses and reject the research based on this.
  - magicalhippo 6y ago
    Can't it be compensated for[1] instead? [1]: https://en.wikipedia.org/wiki/Look-elsewhere_effect https://en.wikipedia.org/wiki/Look-elsewhere_effect
    - laGrenouille 6y ago
      Yes, that's entirely possible. My comment was made keeping in mind that the scientific norm is to have a single preregistration for a single experiment. If a researcher wants to test N hypothesis, using an appropriate p-value correction to account for this, usually the entire set of hypotheses and the correction itself would be contained in a single registration. And that would be perfectly fine. Submiting a large number of individual registrations for a single set of data is just not the norm and so it would be suspicious to begin with. (Okay, I will admit that on the order of 2-3 registrations for very different categories of study may not be as strange; but certainly 20 would signal something very odd).

Icathian 6y ago

This is the right answer. I worked in clinical trials for a while, and they're required to submit a Statistical Analysis Plan before the study is even allowed to proceed. After all the data are collected, you just run whatever was originally proposed and you get the answer you get. This really should be the standard for any meaningful science.

Iv 6y ago

Can't you just pre-register for 20 studies of random correlations and still get one p>0.05?

funklute 6y ago
You can, but that's not necessarily problematic in itself (perhaps slightly depending on what your aim is). By definition, using a statistical significance threshold of 0.05 means that you will have 5% false positive rate. You always expect that _some_ results are erronous (either false positive or false negatives), and the point of using p-values is merely to control the rate at which this happens. That said, you can do something called multiple comparisons corrections, where you take into account the fact that you're doing multiple tests in the same go. In this case you'll end up being more conservative, and it's not the false positive rate you'll be controlling - instead you'll control the family-wise error rate, which basically has to do with whether _any_ of your positive results are false positives.
laGrenouille 6y ago
The idea is that preregistration is done publicly, so a reviewer of a paper could easily see that you have registered multiple analyses and reject the research based on this.
- magicalhippo 6y ago
  Can't it be compensated for[1] instead? [1]: https://en.wikipedia.org/wiki/Look-elsewhere_effect https://en.wikipedia.org/wiki/Look-elsewhere_effect
  - laGrenouille 6y ago
    Yes, that's entirely possible. My comment was made keeping in mind that the scientific norm is to have a single preregistration for a single experiment. If a researcher wants to test N hypothesis, using an appropriate p-value correction to account for this, usually the entire set of hypotheses and the correction itself would be contained in a single registration. And that would be perfectly fine. Submiting a large number of individual registrations for a single set of data is just not the norm and so it would be suspicious to begin with. (Okay, I will admit that on the order of 2-3 registrations for very different categories of study may not be as strange; but certainly 20 would signal something very odd).