5 ms·

Require experimental analysis to be pre-registered? That is how it's done in particle physics, and it works well.

• mike_hearn 3y ago
  It helps a lot but rests on the assumption that the people with whom you're pre-registering will check that you did the same thing you claimed you'd do. And also, that they won't tolerate gamed pre-registrations e.g. pre-registrations so vague that they admit many different post-hoc analyses. Sometimes this is true. One example is when the people verifying the pre-registration work for a regulator. The introduction of pre-registration seems to be at least partly behind the collapse in pharma productivity: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0132382 https://journals.plos.org/plosone/article?id=10.1371/journal... 17 of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after 2000 (χ2=12.2,df= 1, p=0.0005). There has been no change in the proportion of trials that compared treatment to placebo versus active comparator. Industry co-sponsorship was unrelated to the probability of reporting a significant benefit. Pre-registration in clinicaltrials.gov was strongly associated with the trend toward null findings. But when journals do this the results seem to be worse. They aren't strongly incentivized to improve anything, so you can get studies that claim to be pre-registered but do something different to what the pre-registration said they'd do.