3 ms·
IMHO the simple problems with reproducibility (of statictical studies) can be solved with pre-registration, which should become a standard. The rest really is a
by stewbrew 7y ago
IMHO the simple problems with reproducibility (of statictical studies) can be solved with pre-registration, which should become a standard. The rest really is a system crisis.
Imagine: Scientist A uses (a statistical) method M to assert X. People praise A until scientist B uses method M and cannot assert X (which BTW usually doesn't imply X is wrong). People wait for scientist C to use method M and re-check assertion X. People then take the majority vote (or meta-analysis) for true.
That's the way much of science works today. It becomes a problem only in the following situations:
0. People hunt for statistically significant results.
1. X is mostly irrelevant to anyone who does not belong to the in-group of A, B, C, and peers. So nobody else really notices or cares about X.
2. There is no other way to observe any meaningful consequence of X other than by method M -- which eventually results in #1.
3. Method M is really expensive and complex, it's an almost impossible undertaking so that B or C most likely won't get a funding.
4. Everything is ok, actually, but the people who pay for the study don't care about A's methodological fineprint as long as the results play well with their other goals.
The presented list of "factors that build reproducibility" focuses on #0, for which pre-registration is a simple and clean solution. IMHO the list is much to narrow and focused on academic practice, though.