4 ms·

Publishing negative results is important to solving this issue.

atoav 5y ago
How about instead of applying to journals with a finished study, people would apply with a proposal of a study and then publish the results independent of how spectacular/boring they turn out? Journals would then have to select studies based on their design and based on the question and scientists could just do science.
- jononor 5y ago
  This is a good idea, and is known as "preregistration". https://en.wikipedia.org/wiki/Preregistration_(science) https://en.wikipedia.org/wiki/Preregistration_(science) Hopefully it will become more common practice.
  - FabHK 5y ago
    Well, it is preregistration plus a commitment by a journal to accept and publish the resulting paper (no matter the outcome of the study). That apparently is "Registered reports" (see same Wikipedia article), an idea I hadn't heard before. Edited to add (from Wikipedia): > Over 200 journals offer a registered reports option (Centre for Open Science, 2019),[33] and the number of journals that are adopting registered reports is approximately doubling each year (Chambers et al., 2019).[34] > Nature Human Behaviour has adopted the registered report format, as it “shift[s] the emphasis from the results of research to the questions that guide the research and the methods used to answer them”.[36] > European Journal of Personality defines this format: “In a registered report, authors create a study proposal that includes theoretical and empirical background, research questions/hypotheses, and pilot data (if available). Upon submission, this proposal will then be reviewed prior to data collection, and if accepted, the paper resulting from this peer-reviewed procedure will be published, regardless of the study outcomes.”[37]
smitty1e 5y ago
Incentivizing publishing negative results is a challenge.
- tux3 5y ago
  This has potential consequences I don't like. Once negative results are incentivized the same way as positive results, cheaters could have a perverse incentive to post fraudulent failures to replicate, too. I think I agree that we need much more transparency and scientific rigor, first. Post the data whenever possible, incentivize people for creating good study protocols that make it harder for any single person to fake the data without it being obvious. Once we have enough confidence in that, let's do preregistration, sure. And then negative results should be as incentivized as positive ones, absolutely. I worry about the order in which we want to do these things. If people are faking data, we need to make this specific step harder to hide from reviewers. If you incentivize people to fail to replicate each other without solving the lag between fraudulent data and retractation, you will get fraudulent negative results. That could cause a lot of confusion. If the replication crisis itself starts to be fraudulent, scientific credibility is further damaged, negative results will be less credible, and fraudsters will profit from the confusion.
  - regardsop 5y ago
    "If negative results are published like other information, people could have a incentive to replicate post fraudulent failures"? Um sounds worse, (OT:) to add something a simpified (strict stripped) example -hope practicable: You work at a company. Orders are comming in, wares are placed as 'reserved' till paid, than shipped. With a worker perspective, you check the emails with orders ('the customer whishes'), sending 'bills' and the wares if, or if not been paid - 'you've been paid'. Viewed by the company-manager your position may a) be valid oder come b) 'overvalued' (...) Stucked in this example, you talked to other 'employeees' during break and 'there was a time meeting on weekends', so you talked about work and there was a (special) kind of 'more transparency' than without peer review. ^^ Regards, hope this helps... (-:
    - tux3 5y ago
willis936 5y ago
I agree. The grad students agree. Now how do you convince a group obsessed with career advancement and holding all of the cards (PIs) to agree?
- detaro 5y ago
  By convincing the people deciding about their career advancement that negative results are worth something too. (no, that isn't easy, since it's the same people to a good degree, which is a problem with solving many problems in academia)
- unityByFreedom 5y ago
  Enough of these "falsified science" stories come out and people will become interested in stories debunking bad science. I subscribe to a podcast called "everything hertz" for this reason. https://everythinghertz.com/ https://everythinghertz.com/ https://twitter.com/jamesheathers https://twitter.com/jamesheathers https://twitter.com/dsquintana https://twitter.com/dsquintana
- MichaelZuo 5y ago
  Give groups larger grants for publishing negative results too? More money seems to be workable enough in the rest of the economy. Obviously it would be best if those at the top make the change and let it cascade down. But any private source of funds could also experiment with changing the grant model.

How about instead of applying to journals with a finished study, people would apply with a proposal of a study and then publish the results independent of how spectacular/boring they turn out? Journals would then have to select studies based on their design and based on the question and scientists could just do science.

jononor 5y ago
This is a good idea, and is known as "preregistration". https://en.wikipedia.org/wiki/Preregistration_(science) https://en.wikipedia.org/wiki/Preregistration_(science) Hopefully it will become more common practice.
- FabHK 5y ago
  Well, it is preregistration plus a commitment by a journal to accept and publish the resulting paper (no matter the outcome of the study). That apparently is "Registered reports" (see same Wikipedia article), an idea I hadn't heard before. Edited to add (from Wikipedia): > Over 200 journals offer a registered reports option (Centre for Open Science, 2019),[33] and the number of journals that are adopting registered reports is approximately doubling each year (Chambers et al., 2019).[34] > Nature Human Behaviour has adopted the registered report format, as it “shift[s] the emphasis from the results of research to the questions that guide the research and the methods used to answer them”.[36] > European Journal of Personality defines this format: “In a registered report, authors create a study proposal that includes theoretical and empirical background, research questions/hypotheses, and pilot data (if available). Upon submission, this proposal will then be reviewed prior to data collection, and if accepted, the paper resulting from this peer-reviewed procedure will be published, regardless of the study outcomes.”[37]

smitty1e 5y ago

Incentivizing publishing negative results is a challenge.

tux3 5y ago
This has potential consequences I don't like. Once negative results are incentivized the same way as positive results, cheaters could have a perverse incentive to post fraudulent failures to replicate, too. I think I agree that we need much more transparency and scientific rigor, first. Post the data whenever possible, incentivize people for creating good study protocols that make it harder for any single person to fake the data without it being obvious. Once we have enough confidence in that, let's do preregistration, sure. And then negative results should be as incentivized as positive ones, absolutely. I worry about the order in which we want to do these things. If people are faking data, we need to make this specific step harder to hide from reviewers. If you incentivize people to fail to replicate each other without solving the lag between fraudulent data and retractation, you will get fraudulent negative results. That could cause a lot of confusion. If the replication crisis itself starts to be fraudulent, scientific credibility is further damaged, negative results will be less credible, and fraudsters will profit from the confusion.
- regardsop 5y ago
  "If negative results are published like other information, people could have a incentive to replicate post fraudulent failures"? Um sounds worse, (OT:) to add something a simpified (strict stripped) example -hope practicable: You work at a company. Orders are comming in, wares are placed as 'reserved' till paid, than shipped. With a worker perspective, you check the emails with orders ('the customer whishes'), sending 'bills' and the wares if, or if not been paid - 'you've been paid'. Viewed by the company-manager your position may a) be valid oder come b) 'overvalued' (...) Stucked in this example, you talked to other 'employeees' during break and 'there was a time meeting on weekends', so you talked about work and there was a (special) kind of 'more transparency' than without peer review. ^^ Regards, hope this helps... (-:
  - tux3 5y ago

willis936 5y ago

I agree. The grad students agree. Now how do you convince a group obsessed with career advancement and holding all of the cards (PIs) to agree?

detaro 5y ago
By convincing the people deciding about their career advancement that negative results are worth something too. (no, that isn't easy, since it's the same people to a good degree, which is a problem with solving many problems in academia)
unityByFreedom 5y ago
Enough of these "falsified science" stories come out and people will become interested in stories debunking bad science. I subscribe to a podcast called "everything hertz" for this reason. https://everythinghertz.com/ https://everythinghertz.com/ https://twitter.com/jamesheathers https://twitter.com/jamesheathers https://twitter.com/dsquintana https://twitter.com/dsquintana
MichaelZuo 5y ago
Give groups larger grants for publishing negative results too? More money seems to be workable enough in the rest of the economy. Obviously it would be best if those at the top make the change and let it cascade down. But any private source of funds could also experiment with changing the grant model.