3 ms·

There are registration databases, but there are issues. Who trawls the (now huge) database to find the registered studies which never reported? In any case, th

by pmyteh 6y ago

There are registration databases, but there are issues. Who trawls the (now huge) database to find the registered studies which never reported? In any case, that could indicate a file drawer problem publication bias or simply that the funding disappeared or a researcher decided not to proceed. How do you move from very spotty preregistration to compulsory preregistration? It would probably take a coordinated push from all the major funders. And they haven't successfully moved to fully open access publication in over a decade of trying, despite the obvious financial gain to the non-publisher world of winning at that coordination game. How do you handle work on the margins of registerability? Not all scientific work is testing hypotheses, some is exploratory. Should a quantitative description be ruled out if it wasn't registered? If so, how are reasonable hypotheses for future work established? If not, you have to be extra vigilant to stop people smuggling phrases that imply confirmation into nominally exploratory work. And the other problem with the replication crisis is that even decently-planned, pre-registered, straightforward studies can not replicate. Sometimes for reasonable reasons (significance was actually by chance, some unanticipated confounder interfered) and sometimes for thoroughly blameworthy ones (rather than p-hacking you can just fudge the data, or make 'accidental' programming errors). So I'm all in favour of pre-registration, but it's not a magic bullet.

hndudette2 6y ago
We both probably agree that parapsychology hasn't demonstrated that psychic phenomenon is real, and yet their various meta-analyses purport to show that it is. The experimental design of the underlying studies is quite good, so it must be publication bias. I couldn't think of a better demonstration as to why preregistration is a critical necessity, even if it is difficult to get going. All of your questions are very good ones however, and I agree with your conclusion that it isn't going to be a silver bullet. Who trawls the (now huge) database Only the authors of the meta-analyses, as part of their lit review. It's simply a criteria for inclusion into meta-analysis. How do you move from very spotty preregistration to compulsory preregistration? Eventually, if meta-analyses only look at studies that are preregistered, this would boost citation count for preregistered studies and therefore incent scientists to preregister since their careers are tied to citation count. How to arrive at this end state is a bit of a chicken and egg problem and requires a cultural shift, so it'll be very difficult. But once the end state is arrived at it should be sticky without needing to be compulsory, since career incentives are aligned. How do you handle work on the margins of registerability? Preregistration is mostly helpful for research questions that are amenable to statistical meta-analysis, i.e. those questions where the hypotheses tested across papers are sufficiently similar so the quantitative results can be statistically combined and statistical significance & clinical significance can be evaluated. I think preregistration is mostly needed for these questions, although it's probably helpful elsewhere too.

by pmyteh 6y ago

hndudette2 6y ago

We both probably agree that parapsychology hasn't demonstrated that psychic phenomenon is real, and yet their various meta-analyses purport to show that it is. The experimental design of the underlying studies is quite good, so it must be publication bias. I couldn't think of a better demonstration as to why preregistration is a critical necessity, even if it is difficult to get going. All of your questions are very good ones however, and I agree with your conclusion that it isn't going to be a silver bullet. Who trawls the (now huge) database Only the authors of the meta-analyses, as part of their lit review. It's simply a criteria for inclusion into meta-analysis. How do you move from very spotty preregistration to compulsory preregistration? Eventually, if meta-analyses only look at studies that are preregistered, this would boost citation count for preregistered studies and therefore incent scientists to preregister since their careers are tied to citation count. How to arrive at this end state is a bit of a chicken and egg problem and requires a cultural shift, so it'll be very difficult. But once the end state is arrived at it should be sticky without needing to be compulsory, since career incentives are aligned. How do you handle work on the margins of registerability? Preregistration is mostly helpful for research questions that are amenable to statistical meta-analysis, i.e. those questions where the hypotheses tested across papers are sufficiently similar so the quantitative results can be statistically combined and statistical significance & clinical significance can be evaluated. I think preregistration is mostly needed for these questions, although it's probably helpful elsewhere too.