4 ms·

> We need to instate the rule that only preregistered studies can be published or at least used by the FDA as for decisions. That won't help, not by itself. P

by 08-15 8y ago

> We need to instate the rule that only preregistered studies can be published or at least used by the FDA as for decisions. That won't help, not by itself. Preregistration helps with "p hacking", the practice of moving the goal posts until some result become significant. A bigger problem remains. Medicine and Biology accepted the framework of Statistical Hypothesis Inference Testing, where a null hypothesis is rejected at some p-value, usually 0.05. Ignoring many faults of this framework (for example that the alternative hypothesis is not tested at all, that it is a bad caricature of the bad statistics R. A. Fisher introduced, that the numerical outcomes depend on the probability of events that didn't happen, etc.), the logic is that you limit the false positive rate to below the p-value threshold. Unfortunately, journals consistently reject submission, unless there is a significant p-value somewhere in there. ("Highly significant" is better, even though the term makes no sense.) So if plenty of researchers investigate some random nonsense where the null hypothesis is actually true, 100% of published studies will be spurious results, preregistration or not. To make progress, this kind of statistics has to go. Journals would have to change their policies, which probably means peer reviewers have to change theirs. I have no idea how to make that happen.

Bartweiss 8y ago
Preregistration does help with this issue a bit. It won't make published studies any better, but it still helps solve the file-drawer problem if the results are available somewhere. If you take a shotgun approach to getting significance out of null hypotheses, you presumably get 5% successes. (Sort of. Maybe. p-values really stink, as you point out.) But if 20 people all run variants of the same bunk study, preregistration means that instead of just seeing one success, we see one success on the cover of PNAS, and then at least have the option to look up the ~10 prior failures and decide not to accept the conclusion. (Well, one time in twenty it'll be the first try, but it's still progress.) Of course, that's still going to push researchers to change up their techniques for getting false positives. It rewards either covering unexplored domains (so there's no counterexample to your false positive), or exploiting other tricks to manipulate significance in preregistered work. I don't think existing journals have much hope of improving their practices, honestly. Between their ludicrous financial incentives and ideologues talking "methodological terrorism", that matter seems depressingly settled. But I wonder if there might be hope for creating a counterweight to this? What happens if Gelman and Khahneman get together to create the Very Prestigious Journal of Replications and the Somewhat Prestigious Journal of Rejected Hypotheses? The first one is neat because it rewards negative results, since they change the state of what's understood - at the very least it might encourage more people to start checking the most influential and most suspicious results. The second thing might be a harder sell, but it could reward comprehensive negative work on key topics (much like Alzheimer's research has heavily consisted of excluding possible pathways). Honestly, "create new prestigious journals from scratch" sounds far easier than "reform existing journals" or "tenure and reward professors for reasons other than publications".
- 08-15 8y ago
  > Honestly, "create new prestigious journals from scratch" sounds far easier... What used to be "prestigious" has come to mean "high impact factor", mostly because funding agencies needed an "objective" way to grant funding, and being run by bureaucrats, they picked impact factor as a measure of scientific success. For a Journal Of Replications to have impact, researchers would have to cite papers about replications. But that's not expected; you cite the original paper, not the one that replicates a result, and you never hear about the one that failed to reproduce a result. I don't see that approach working, but I'd like to see someone try it.
  - Bartweiss 8y ago
    > For a Journal Of Replications to have impact, researchers would have to cite papers about replications. Yep, this is definitely the biggest weakness for the idea. A journal about replications would probably end up with low impact factor, and wouldn't be able to grab a groundbreaking result to boost its prestige the way some narrow-topic journals have. My only thought for handling that was alluded to with "Gelman and Kahneman"; maintaining a tolerable reputation and impact factor would probably require leaning on big names and people who are devoted to doing replication work regardless of journal. Gelman, for instance, showed up in PNAS with references when he criticized that godawful "boarding and air rage" study. If we're willing to abuse the process even further, there might be room to publish methodology-of-replication papers in statistical or procedural journals and create further citations that way. It's not a very good answer, and I don't think it's going to free anyone from "publish or perish". The best I can really hope for is creating a bit of space for talented, skeptical researchers to add replications to their other work without needing to be tenured and entrenched first.
    - grigjd3 8y ago
      A big part of the problem is there are too many PhDs chasing too little funding. There's no room for negative results when the competition is that big.
      - 8y ago

by 08-15 8y ago

Bartweiss 8y ago

Preregistration does help with this issue a bit. It won't make published studies any better, but it still helps solve the file-drawer problem if the results are available somewhere. If you take a shotgun approach to getting significance out of null hypotheses, you presumably get 5% successes. (Sort of. Maybe. p-values really stink, as you point out.) But if 20 people all run variants of the same bunk study, preregistration means that instead of just seeing one success, we see one success on the cover of PNAS, and then at least have the option to look up the ~10 prior failures and decide not to accept the conclusion. (Well, one time in twenty it'll be the first try, but it's still progress.) Of course, that's still going to push researchers to change up their techniques for getting false positives. It rewards either covering unexplored domains (so there's no counterexample to your false positive), or exploiting other tricks to manipulate significance in preregistered work. I don't think existing journals have much hope of improving their practices, honestly. Between their ludicrous financial incentives and ideologues talking "methodological terrorism", that matter seems depressingly settled. But I wonder if there might be hope for creating a counterweight to this? What happens if Gelman and Khahneman get together to create the Very Prestigious Journal of Replications and the Somewhat Prestigious Journal of Rejected Hypotheses? The first one is neat because it rewards negative results, since they change the state of what's understood - at the very least it might encourage more people to start checking the most influential and most suspicious results. The second thing might be a harder sell, but it could reward comprehensive negative work on key topics (much like Alzheimer's research has heavily consisted of excluding possible pathways). Honestly, "create new prestigious journals from scratch" sounds far easier than "reform existing journals" or "tenure and reward professors for reasons other than publications".

08-15 8y ago
> Honestly, "create new prestigious journals from scratch" sounds far easier... What used to be "prestigious" has come to mean "high impact factor", mostly because funding agencies needed an "objective" way to grant funding, and being run by bureaucrats, they picked impact factor as a measure of scientific success. For a Journal Of Replications to have impact, researchers would have to cite papers about replications. But that's not expected; you cite the original paper, not the one that replicates a result, and you never hear about the one that failed to reproduce a result. I don't see that approach working, but I'd like to see someone try it.
- Bartweiss 8y ago
  > For a Journal Of Replications to have impact, researchers would have to cite papers about replications. Yep, this is definitely the biggest weakness for the idea. A journal about replications would probably end up with low impact factor, and wouldn't be able to grab a groundbreaking result to boost its prestige the way some narrow-topic journals have. My only thought for handling that was alluded to with "Gelman and Kahneman"; maintaining a tolerable reputation and impact factor would probably require leaning on big names and people who are devoted to doing replication work regardless of journal. Gelman, for instance, showed up in PNAS with references when he criticized that godawful "boarding and air rage" study. If we're willing to abuse the process even further, there might be room to publish methodology-of-replication papers in statistical or procedural journals and create further citations that way. It's not a very good answer, and I don't think it's going to free anyone from "publish or perish". The best I can really hope for is creating a bit of space for talented, skeptical researchers to add replications to their other work without needing to be tenured and entrenched first.
  - grigjd3 8y ago
    A big part of the problem is there are too many PhDs chasing too little funding. There's no room for negative results when the competition is that big.
    - 8y ago