4 ms·

Regarding the replication crisis, can I ask why we don't yet have a central body to which scientists report their intended experimental design, sample size and

Regarding the replication crisis, can I ask why we don't yet have a central body to which scientists report their intended experimental design, sample size and research hypothesis before the experiment begins? Then we can eliminate publication bias due to null results not being published which is a big part of the replication crisis. The meta-analyses only survey those studies that reported this ahead of time and the scope of possible publication bias can be quantified.

pmyteh 6y ago
There are registration databases, but there are issues. Who trawls the (now huge) database to find the registered studies which never reported? In any case, that could indicate a file drawer problem publication bias or simply that the funding disappeared or a researcher decided not to proceed. How do you move from very spotty preregistration to compulsory preregistration? It would probably take a coordinated push from all the major funders. And they haven't successfully moved to fully open access publication in over a decade of trying, despite the obvious financial gain to the non-publisher world of winning at that coordination game. How do you handle work on the margins of registerability? Not all scientific work is testing hypotheses, some is exploratory. Should a quantitative description be ruled out if it wasn't registered? If so, how are reasonable hypotheses for future work established? If not, you have to be extra vigilant to stop people smuggling phrases that imply confirmation into nominally exploratory work. And the other problem with the replication crisis is that even decently-planned, pre-registered, straightforward studies can not replicate. Sometimes for reasonable reasons (significance was actually by chance, some unanticipated confounder interfered) and sometimes for thoroughly blameworthy ones (rather than p-hacking you can just fudge the data, or make 'accidental' programming errors). So I'm all in favour of pre-registration, but it's not a magic bullet.
- hndudette2 6y ago
  We both probably agree that parapsychology hasn't demonstrated that psychic phenomenon is real, and yet their various meta-analyses purport to show that it is. The experimental design of the underlying studies is quite good, so it must be publication bias. I couldn't think of a better demonstration as to why preregistration is a critical necessity, even if it is difficult to get going. All of your questions are very good ones however, and I agree with your conclusion that it isn't going to be a silver bullet. Who trawls the (now huge) database Only the authors of the meta-analyses, as part of their lit review. It's simply a criteria for inclusion into meta-analysis. How do you move from very spotty preregistration to compulsory preregistration? Eventually, if meta-analyses only look at studies that are preregistered, this would boost citation count for preregistered studies and therefore incent scientists to preregister since their careers are tied to citation count. How to arrive at this end state is a bit of a chicken and egg problem and requires a cultural shift, so it'll be very difficult. But once the end state is arrived at it should be sticky without needing to be compulsory, since career incentives are aligned. How do you handle work on the margins of registerability? Preregistration is mostly helpful for research questions that are amenable to statistical meta-analysis, i.e. those questions where the hypotheses tested across papers are sufficiently similar so the quantitative results can be statistically combined and statistical significance & clinical significance can be evaluated. I think preregistration is mostly needed for these questions, although it's probably helpful elsewhere too.
garden_hermit 6y ago
These kinds of things exist, though the rate that they are adopted varies widely. In Psychology, for instance, there was discussion back in 2015 about per-registration in the field [1]. Since then, tools and repositories have been created to help facilitate per-registration [2]. But again, they are not universally adopted. Some of this is probably generational—academia is quite conservative, and doesn't change quickly. A new generation will likely use these tools more. Other times, fields are more insular than others, and so such tools will take some time to diffuse into their community. In the meantime though, things have improved, even if only somewhat, across the sciences. [1] https://www.apa.org/science/about/psa/2015/08/pre-registration https://www.apa.org/science/about/psa/2015/08/pre-registrati... [2] https://www.cos.io/initiatives/prereg https://www.cos.io/initiatives/prereg

pmyteh 6y ago

There are registration databases, but there are issues. Who trawls the (now huge) database to find the registered studies which never reported? In any case, that could indicate a file drawer problem publication bias or simply that the funding disappeared or a researcher decided not to proceed. How do you move from very spotty preregistration to compulsory preregistration? It would probably take a coordinated push from all the major funders. And they haven't successfully moved to fully open access publication in over a decade of trying, despite the obvious financial gain to the non-publisher world of winning at that coordination game. How do you handle work on the margins of registerability? Not all scientific work is testing hypotheses, some is exploratory. Should a quantitative description be ruled out if it wasn't registered? If so, how are reasonable hypotheses for future work established? If not, you have to be extra vigilant to stop people smuggling phrases that imply confirmation into nominally exploratory work. And the other problem with the replication crisis is that even decently-planned, pre-registered, straightforward studies can not replicate. Sometimes for reasonable reasons (significance was actually by chance, some unanticipated confounder interfered) and sometimes for thoroughly blameworthy ones (rather than p-hacking you can just fudge the data, or make 'accidental' programming errors). So I'm all in favour of pre-registration, but it's not a magic bullet.

hndudette2 6y ago
We both probably agree that parapsychology hasn't demonstrated that psychic phenomenon is real, and yet their various meta-analyses purport to show that it is. The experimental design of the underlying studies is quite good, so it must be publication bias. I couldn't think of a better demonstration as to why preregistration is a critical necessity, even if it is difficult to get going. All of your questions are very good ones however, and I agree with your conclusion that it isn't going to be a silver bullet. Who trawls the (now huge) database Only the authors of the meta-analyses, as part of their lit review. It's simply a criteria for inclusion into meta-analysis. How do you move from very spotty preregistration to compulsory preregistration? Eventually, if meta-analyses only look at studies that are preregistered, this would boost citation count for preregistered studies and therefore incent scientists to preregister since their careers are tied to citation count. How to arrive at this end state is a bit of a chicken and egg problem and requires a cultural shift, so it'll be very difficult. But once the end state is arrived at it should be sticky without needing to be compulsory, since career incentives are aligned. How do you handle work on the margins of registerability? Preregistration is mostly helpful for research questions that are amenable to statistical meta-analysis, i.e. those questions where the hypotheses tested across papers are sufficiently similar so the quantitative results can be statistically combined and statistical significance & clinical significance can be evaluated. I think preregistration is mostly needed for these questions, although it's probably helpful elsewhere too.

garden_hermit 6y ago

These kinds of things exist, though the rate that they are adopted varies widely. In Psychology, for instance, there was discussion back in 2015 about per-registration in the field [1]. Since then, tools and repositories have been created to help facilitate per-registration [2]. But again, they are not universally adopted. Some of this is probably generational—academia is quite conservative, and doesn't change quickly. A new generation will likely use these tools more. Other times, fields are more insular than others, and so such tools will take some time to diffuse into their community. In the meantime though, things have improved, even if only somewhat, across the sciences. [1] https://www.apa.org/science/about/psa/2015/08/pre-registration https://www.apa.org/science/about/psa/2015/08/pre-registrati... [2] https://www.cos.io/initiatives/prereg https://www.cos.io/initiatives/prereg