4 ms·

Boy do I not agree with the substance of this article. I agree that yes, preregistration is not a panacea that will fix the reproducibility crisis or whatever.

Boy do I not agree with the substance of this article. I agree that yes, preregistration is not a panacea that will fix the reproducibility crisis or whatever. But the problem that it is solving is not to improve the quality of studies that get published. It only does that implicitly. What it does is allows us to factor in the unpublished studies. It gives us access to previously unpublishable null results. Would this have debunked the ESP research? No, that study would have gotten published because its results supported the expectations of the experimenters. BUT all the attempts to reproduce that would also have been pre-registered, and their lack of publishing (or the publishing of null results) would have effectively debunked the original.

jrochkind1 4y ago
The ESP research? Is a bigger story than some readers may realize. The "ESP guy" is J.B. Rhine. Not everyone knows/remembers that ESP was considered scientifically verified for a good couple 20th century decades, such that many people who considered themselves scientific thinkers felt they had to accept it's validity even though they had been suspicious. It was not just one study to be "debunked", it was literally a whole field (although granted one mostly controlled by Rhine, but that was not that unusual of scientific fields back then, not sure about now). Makes you think about the whole endeavor of science. On the other hand, it was eventually abandoned (not with a big debunking bang so much as just kind of forgotten). One thing it does is make me wonder what things today are accepted that won't be in the future. I'm not totally sure what the conclusions should be from the "ESP guy" thing. I think they include bigger questions about epistemology than simply "good research design" -- although on the other hand, research design was definitely a part of it, and pre-registration specifically actually specifically relevant to Rhine. One of the debunking claims of his critics is that he achieved his positive results by burying studies that didn't have the results he wanted (or even not being hygienic about what runs were part of what study), just keeping the ones with positive results. Pre-registration actually might have nipped him in the bud... (unless he was willing to be just plain fraudulent of course; there are few guards against that except discovering it; I think Rhine did not think he was being fraudulent, he was just conveniently sloppy in a way he didn't think mattered... to discover fraud you have to have a common understanding of what constitutes fraud, which pre-registration is part (just part) of developing I think)
- moyix 4y ago
  I suspect the "ESP guy" is actually meant to refer to Daryl Bem? https://slate.com/health-and-science/2017/06/daryl-bem-proved-esp-is-real-showed-science-is-broken.html https://slate.com/health-and-science/2017/06/daryl-bem-prove... https://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ https://slatestarcodex.com/2014/04/28/the-control-group-is-o...
  - jrochkind1 4y ago
    Aha, I hadn't been paying attention, thanks! I have known about Rhine since before Bem published his thing in 2010. It's interesting that neither of those articles on Bem mentions Rhine explicitly (at least the first one does link to him on "dating to the 1930s".) This has all happened before. And as far as we can tell Rhine did almost exactly the same thing as Bem, cherry-picked the "successful" runs, with statistically flawed results. But Rhine was actually a lot more succesful at convincing mainstream people of his results... but while by the 1950s or 1960s nobody believed them anymore, as far as I can tell there wasn't any conclusive single turning point where everyone by consensus agreed that they had been wrong, and why. It just sort of faded away. And isn't currently talked about much. It's sort of embaressing to scientists, I think, that so many believed in ESP from Rhine's experiments. As a result, it isn't part of general cultural knowledge, or scientific education, nor is an understanding of why Rhine wasn't right after all... and therefore, Bem is just a repeat. Or, more worrisome, the same mistakes could be made in more 'conventional' topics as well. [I don't think either Rhine or Bem were doing intentional fraud, I think they both believed themselves. It really does look like a repeat].
    - buescher 4y ago
      The same mistakes are made in all kinds of conventional research. I would go so far as to say they are institutionalized. That's the real reason ESP research is so embarassing. People had literally been talking about preregistration for decades before the replication crisis.
- buescher 4y ago
  > scientifically verified That's a bit of a strong statement, but not quite untrue. There was enough legitimate interest that there was funded parapsychology research at places like SRI into the eighties that was at least serious in intent if not in execution or outcome. I just checked and the academic Journal of Parapsychology is still published, and the AAAS has gracefully not kicked the Parapyschological Association out of affiliation yet. Yeah, the big debunking bang never happened, but the research always had a particular smell about it: smaller and smaller effects every year with greater and greater statistical significance. Preregistration helps with the optional stopping problem, but it doesn't completely solve it, and meta-analysis actively hurts. From my brief time reading the parapsychological literature and its critics for fun, I still look at any meta-analysis with extreme skepticism.
infogulch 4y ago
Preregistration doesn't solve the reproducibility crisis, but it's good progress on the "unpublished null results" sub-crisis which has a 'necessary but not sufficient' relationship to the former.
deleted 4y ago
[deleted]
- csours 4y ago
  Only if you do it "right" - and there are incentives to not do it right. And it's hard to agree on what right means. And what right means will change as preregistration becomes more common and people understand it better.

jrochkind1 4y ago

The ESP research? Is a bigger story than some readers may realize. The "ESP guy" is J.B. Rhine. Not everyone knows/remembers that ESP was considered scientifically verified for a good couple 20th century decades, such that many people who considered themselves scientific thinkers felt they had to accept it's validity even though they had been suspicious. It was not just one study to be "debunked", it was literally a whole field (although granted one mostly controlled by Rhine, but that was not that unusual of scientific fields back then, not sure about now). Makes you think about the whole endeavor of science. On the other hand, it was eventually abandoned (not with a big debunking bang so much as just kind of forgotten). One thing it does is make me wonder what things today are accepted that won't be in the future. I'm not totally sure what the conclusions should be from the "ESP guy" thing. I think they include bigger questions about epistemology than simply "good research design" -- although on the other hand, research design was definitely a part of it, and pre-registration specifically actually specifically relevant to Rhine. One of the debunking claims of his critics is that he achieved his positive results by burying studies that didn't have the results he wanted (or even not being hygienic about what runs were part of what study), just keeping the ones with positive results. Pre-registration actually might have nipped him in the bud... (unless he was willing to be just plain fraudulent of course; there are few guards against that except discovering it; I think Rhine did not think he was being fraudulent, he was just conveniently sloppy in a way he didn't think mattered... to discover fraud you have to have a common understanding of what constitutes fraud, which pre-registration is part (just part) of developing I think)

moyix 4y ago
I suspect the "ESP guy" is actually meant to refer to Daryl Bem? https://slate.com/health-and-science/2017/06/daryl-bem-proved-esp-is-real-showed-science-is-broken.html https://slate.com/health-and-science/2017/06/daryl-bem-prove... https://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ https://slatestarcodex.com/2014/04/28/the-control-group-is-o...
- jrochkind1 4y ago
  Aha, I hadn't been paying attention, thanks! I have known about Rhine since before Bem published his thing in 2010. It's interesting that neither of those articles on Bem mentions Rhine explicitly (at least the first one does link to him on "dating to the 1930s".) This has all happened before. And as far as we can tell Rhine did almost exactly the same thing as Bem, cherry-picked the "successful" runs, with statistically flawed results. But Rhine was actually a lot more succesful at convincing mainstream people of his results... but while by the 1950s or 1960s nobody believed them anymore, as far as I can tell there wasn't any conclusive single turning point where everyone by consensus agreed that they had been wrong, and why. It just sort of faded away. And isn't currently talked about much. It's sort of embaressing to scientists, I think, that so many believed in ESP from Rhine's experiments. As a result, it isn't part of general cultural knowledge, or scientific education, nor is an understanding of why Rhine wasn't right after all... and therefore, Bem is just a repeat. Or, more worrisome, the same mistakes could be made in more 'conventional' topics as well. [I don't think either Rhine or Bem were doing intentional fraud, I think they both believed themselves. It really does look like a repeat].
  - buescher 4y ago
    The same mistakes are made in all kinds of conventional research. I would go so far as to say they are institutionalized. That's the real reason ESP research is so embarassing. People had literally been talking about preregistration for decades before the replication crisis.
buescher 4y ago
> scientifically verified That's a bit of a strong statement, but not quite untrue. There was enough legitimate interest that there was funded parapsychology research at places like SRI into the eighties that was at least serious in intent if not in execution or outcome. I just checked and the academic Journal of Parapsychology is still published, and the AAAS has gracefully not kicked the Parapyschological Association out of affiliation yet. Yeah, the big debunking bang never happened, but the research always had a particular smell about it: smaller and smaller effects every year with greater and greater statistical significance. Preregistration helps with the optional stopping problem, but it doesn't completely solve it, and meta-analysis actively hurts. From my brief time reading the parapsychological literature and its critics for fun, I still look at any meta-analysis with extreme skepticism.

infogulch 4y ago

Preregistration doesn't solve the reproducibility crisis, but it's good progress on the "unpublished null results" sub-crisis which has a 'necessary but not sufficient' relationship to the former.

deleted 4y ago

[deleted]

csours 4y ago
Only if you do it "right" - and there are incentives to not do it right. And it's hard to agree on what right means. And what right means will change as preregistration becomes more common and people understand it better.