3 ms·
We both probably agree that parapsychology hasn't demonstrated that psychic phenomenon is real, and yet their various meta-analyses purport to show that it is.
by hndudette2 6y ago
We both probably agree that parapsychology hasn't demonstrated that psychic phenomenon is real, and yet their various meta-analyses purport to show that it is. The experimental design of the underlying studies is quite good, so it must be publication bias. I couldn't think of a better demonstration as to why preregistration is a critical necessity, even if it is difficult to get going.
All of your questions are very good ones however, and I agree with your conclusion that it isn't going to be a silver bullet.
Who trawls the (now huge) database
Only the authors of the meta-analyses, as part of their lit review. It's simply a criteria for inclusion into meta-analysis.
How do you move from very spotty preregistration to compulsory preregistration?
Eventually, if meta-analyses only look at studies that are preregistered, this would boost citation count for preregistered studies and therefore incent scientists to preregister since their careers are tied to citation count. How to arrive at this end state is a bit of a chicken and egg problem and requires a cultural shift, so it'll be very difficult. But once the end state is arrived at it should be sticky without needing to be compulsory, since career incentives are aligned.
How do you handle work on the margins of registerability?
Preregistration is mostly helpful for research questions that are amenable to statistical meta-analysis, i.e. those questions where the hypotheses tested across papers are sufficiently similar so the quantitative results can be statistically combined and statistical significance & clinical significance can be evaluated. I think preregistration is mostly needed for these questions, although it's probably helpful elsewhere too.