4 ms·

>Researchers should embrace negative results instead of accentuating the positive The problem starts here. Most researchers would love to show their negative r

by carlmr 8y ago

>Researchers should embrace negative results instead of accentuating the positive The problem starts here. Most researchers would love to show their negative results, they're well aware of the problem. But they need to publish, they need grants, they need money which is outcome independent. They need an alternative reality where negative results and reproducibility studies make money. As it stands you get money for publishing praise, a lot of industrially sponsored medical research is just "9/10 doctors recommend x" advertising. We need to instate the rule that only preregistered studies can be published or at least used by the FDA as for decisions. Right now you can conduct 1000 studies until you find a handful which randomly show the result you want.

denzil_correa 8y ago
> The problem starts here. Most researchers would love to show their negative results There's another core issue which most people miss. In science, you need to publish results AND explain WHY. That last part of WHY is crucial. The amount of work required to prove a negative - or explaining WHY - is very very high and in many cases close to impossible.
- AstralStorm 8y ago
  Typically studies do not report negative but lack of null hypothesis rejection (means "we do not know if it does anything"). What we also need is effect sizes (risk reduction, number needed to treat). When a study significantly shows tiny effect size, it is much more damning.
08-15 8y ago
> We need to instate the rule that only preregistered studies can be published or at least used by the FDA as for decisions. That won't help, not by itself. Preregistration helps with "p hacking", the practice of moving the goal posts until some result become significant. A bigger problem remains. Medicine and Biology accepted the framework of Statistical Hypothesis Inference Testing, where a null hypothesis is rejected at some p-value, usually 0.05. Ignoring many faults of this framework (for example that the alternative hypothesis is not tested at all, that it is a bad caricature of the bad statistics R. A. Fisher introduced, that the numerical outcomes depend on the probability of events that didn't happen, etc.), the logic is that you limit the false positive rate to below the p-value threshold. Unfortunately, journals consistently reject submission, unless there is a significant p-value somewhere in there. ("Highly significant" is better, even though the term makes no sense.) So if plenty of researchers investigate some random nonsense where the null hypothesis is actually true, 100% of published studies will be spurious results, preregistration or not. To make progress, this kind of statistics has to go. Journals would have to change their policies, which probably means peer reviewers have to change theirs. I have no idea how to make that happen.
- Bartweiss 8y ago
  Preregistration does help with this issue a bit. It won't make published studies any better, but it still helps solve the file-drawer problem if the results are available somewhere. If you take a shotgun approach to getting significance out of null hypotheses, you presumably get 5% successes. (Sort of. Maybe. p-values really stink, as you point out.) But if 20 people all run variants of the same bunk study, preregistration means that instead of just seeing one success, we see one success on the cover of PNAS, and then at least have the option to look up the ~10 prior failures and decide not to accept the conclusion. (Well, one time in twenty it'll be the first try, but it's still progress.) Of course, that's still going to push researchers to change up their techniques for getting false positives. It rewards either covering unexplored domains (so there's no counterexample to your false positive), or exploiting other tricks to manipulate significance in preregistered work. I don't think existing journals have much hope of improving their practices, honestly. Between their ludicrous financial incentives and ideologues talking "methodological terrorism", that matter seems depressingly settled. But I wonder if there might be hope for creating a counterweight to this? What happens if Gelman and Khahneman get together to create the Very Prestigious Journal of Replications and the Somewhat Prestigious Journal of Rejected Hypotheses? The first one is neat because it rewards negative results, since they change the state of what's understood - at the very least it might encourage more people to start checking the most influential and most suspicious results. The second thing might be a harder sell, but it could reward comprehensive negative work on key topics (much like Alzheimer's research has heavily consisted of excluding possible pathways). Honestly, "create new prestigious journals from scratch" sounds far easier than "reform existing journals" or "tenure and reward professors for reasons other than publications".
  - 08-15 8y ago
    > Honestly, "create new prestigious journals from scratch" sounds far easier... What used to be "prestigious" has come to mean "high impact factor", mostly because funding agencies needed an "objective" way to grant funding, and being run by bureaucrats, they picked impact factor as a measure of scientific success. For a Journal Of Replications to have impact, researchers would have to cite papers about replications. But that's not expected; you cite the original paper, not the one that replicates a result, and you never hear about the one that failed to reproduce a result. I don't see that approach working, but I'd like to see someone try it.
snarf21 8y ago
Like all human systems, you get what you optimize for. Universities want paid grants for money and prestige, so the that's what the researchers go find. I've said this before on here but the solution to this problem is the universities like Harvard and foundations like RWJ which have near infinite money and don't need to operate like this. They need to push forward real research that is reproducible (maybe several times) and shows whatever the data really shows. The researchers who create reproducible studies get bonuses. Again, you get what you optimize for.
the6threplicant 8y ago
The number of discussions I have had with the staff in mathematics and computer science about the need for such journals would last years. Everyone pretty much wants them because they know the need of saying "hey I already did this and we got nowhere - try something else" is so crucial for science and mathematics.
maxxxxx 8y ago
"We need to instate the rule that only preregistered studies can be published or at least used by the FDA as for decisions." I like that idea a lot. And if the study finds something unexpected there should be a thorough explanation why they deviated from the original plan.

by carlmr 8y ago

denzil_correa 8y ago

> The problem starts here. Most researchers would love to show their negative results There's another core issue which most people miss. In science, you need to publish results AND explain WHY. That last part of WHY is crucial. The amount of work required to prove a negative - or explaining WHY - is very very high and in many cases close to impossible.

AstralStorm 8y ago
Typically studies do not report negative but lack of null hypothesis rejection (means "we do not know if it does anything"). What we also need is effect sizes (risk reduction, number needed to treat). When a study significantly shows tiny effect size, it is much more damning.

08-15 8y ago

> We need to instate the rule that only preregistered studies can be published or at least used by the FDA as for decisions. That won't help, not by itself. Preregistration helps with "p hacking", the practice of moving the goal posts until some result become significant. A bigger problem remains. Medicine and Biology accepted the framework of Statistical Hypothesis Inference Testing, where a null hypothesis is rejected at some p-value, usually 0.05. Ignoring many faults of this framework (for example that the alternative hypothesis is not tested at all, that it is a bad caricature of the bad statistics R. A. Fisher introduced, that the numerical outcomes depend on the probability of events that didn't happen, etc.), the logic is that you limit the false positive rate to below the p-value threshold. Unfortunately, journals consistently reject submission, unless there is a significant p-value somewhere in there. ("Highly significant" is better, even though the term makes no sense.) So if plenty of researchers investigate some random nonsense where the null hypothesis is actually true, 100% of published studies will be spurious results, preregistration or not. To make progress, this kind of statistics has to go. Journals would have to change their policies, which probably means peer reviewers have to change theirs. I have no idea how to make that happen.

Bartweiss 8y ago
Preregistration does help with this issue a bit. It won't make published studies any better, but it still helps solve the file-drawer problem if the results are available somewhere. If you take a shotgun approach to getting significance out of null hypotheses, you presumably get 5% successes. (Sort of. Maybe. p-values really stink, as you point out.) But if 20 people all run variants of the same bunk study, preregistration means that instead of just seeing one success, we see one success on the cover of PNAS, and then at least have the option to look up the ~10 prior failures and decide not to accept the conclusion. (Well, one time in twenty it'll be the first try, but it's still progress.) Of course, that's still going to push researchers to change up their techniques for getting false positives. It rewards either covering unexplored domains (so there's no counterexample to your false positive), or exploiting other tricks to manipulate significance in preregistered work. I don't think existing journals have much hope of improving their practices, honestly. Between their ludicrous financial incentives and ideologues talking "methodological terrorism", that matter seems depressingly settled. But I wonder if there might be hope for creating a counterweight to this? What happens if Gelman and Khahneman get together to create the Very Prestigious Journal of Replications and the Somewhat Prestigious Journal of Rejected Hypotheses? The first one is neat because it rewards negative results, since they change the state of what's understood - at the very least it might encourage more people to start checking the most influential and most suspicious results. The second thing might be a harder sell, but it could reward comprehensive negative work on key topics (much like Alzheimer's research has heavily consisted of excluding possible pathways). Honestly, "create new prestigious journals from scratch" sounds far easier than "reform existing journals" or "tenure and reward professors for reasons other than publications".
- 08-15 8y ago
  > Honestly, "create new prestigious journals from scratch" sounds far easier... What used to be "prestigious" has come to mean "high impact factor", mostly because funding agencies needed an "objective" way to grant funding, and being run by bureaucrats, they picked impact factor as a measure of scientific success. For a Journal Of Replications to have impact, researchers would have to cite papers about replications. But that's not expected; you cite the original paper, not the one that replicates a result, and you never hear about the one that failed to reproduce a result. I don't see that approach working, but I'd like to see someone try it.

snarf21 8y ago

Like all human systems, you get what you optimize for. Universities want paid grants for money and prestige, so the that's what the researchers go find. I've said this before on here but the solution to this problem is the universities like Harvard and foundations like RWJ which have near infinite money and don't need to operate like this. They need to push forward real research that is reproducible (maybe several times) and shows whatever the data really shows. The researchers who create reproducible studies get bonuses. Again, you get what you optimize for.

the6threplicant 8y ago

The number of discussions I have had with the staff in mathematics and computer science about the need for such journals would last years. Everyone pretty much wants them because they know the need of saying "hey I already did this and we got nowhere - try something else" is so crucial for science and mathematics.

maxxxxx 8y ago

"We need to instate the rule that only preregistered studies can be published or at least used by the FDA as for decisions." I like that idea a lot. And if the study finds something unexpected there should be a thorough explanation why they deviated from the original plan.