3 ms·

Publishing null results would be great, but I’m worried about how that could also be gamed. How do you distinguish a ‘real’ null result from one done in a slop

Publishing null results would be great, but I’m worried about how that could also be gamed. How do you distinguish a ‘real’ null result from one done in a sloppy study? Would people run shoddy experiments to get null results to undermine their rivals? Could somebody pump out dozens of null publications to pad their CV and screw up h-indexes?

vharuck 2y ago
Currently, the system can be gamed exactly as you say by publishing sloppy studies that falsely find a "real" effect. But editors and readers of the article will look at and judge the methodology section. Sloppy experiments risk not being printed or cited. >Would people run shoddy experiments to get null results to undermine their rivals? In this case, the rival would be very much inclined to recreate the "null" experiment. >Could somebody pump out dozens of null publications to pad their CV and screw up h-indexes? Possibly, but would null publications be cited as often? Also, who's going to keep funding a researcher that mostly publishes null results?[0] [0] Besides agenda-driven "think tanks". Which is worrying itself.
- parpfish 2y ago
  > Possibly, but would null publications be cited as often? Don’t underestimate an academics ability to cite ALL of their previous publications each time they publish
tefkah 2y ago
I run a journal where we publish both “real” null results and experiments were something practical went wrong, so i have a few thoughts: 1. Ideally peer review would catch this. A badly setup study should be critiqued in peer review. Forcing scientists to first publish their methods before doing the experiment also helps, as it validates the experimental setup before hand. I also think it’s worth publishing studies where a null result was reached due to some error in experimental setup or other factors, as long as it’s presented as such and reflected upon. This can still be valuable information for future experiments. Offering scientists social capital for that (an “official” publication, citations) might also incentivize scientists to publish the results as is, rather than making it appear as a “true” null result, or even as a non-null one (eg through p hacking). 2. While obviously possible, given the amount of effort scientists have to go through to raise funding for an experiment nowadays, i find it highly unlikely that people would go through this effort. 3. This is already possible and a problem. This is a problem of academic misconduct, has very little to do with null results. The current publishing system is of course already set up to be gamed, so I understand your worries. But null results should be published, as they are just science. Even if someone were to “game” the system by publishing a ton of null results, those publications should be held to the same level of scrutiny as any other publication. If someone is extremely prolific in replicating existing studies and comes up with a ton of null results, that should be lauded and those papers should be published, no? I do believe the entire idea of a researchers output only being recognized by being allowed to be published in a journal is terrible and should be abolished, but baby steps I guess.

vharuck 2y ago

Currently, the system can be gamed exactly as you say by publishing sloppy studies that falsely find a "real" effect. But editors and readers of the article will look at and judge the methodology section. Sloppy experiments risk not being printed or cited. >Would people run shoddy experiments to get null results to undermine their rivals? In this case, the rival would be very much inclined to recreate the "null" experiment. >Could somebody pump out dozens of null publications to pad their CV and screw up h-indexes? Possibly, but would null publications be cited as often? Also, who's going to keep funding a researcher that mostly publishes null results?[0] [0] Besides agenda-driven "think tanks". Which is worrying itself.

parpfish 2y ago
> Possibly, but would null publications be cited as often? Don’t underestimate an academics ability to cite ALL of their previous publications each time they publish

tefkah 2y ago

I run a journal where we publish both “real” null results and experiments were something practical went wrong, so i have a few thoughts: 1. Ideally peer review would catch this. A badly setup study should be critiqued in peer review. Forcing scientists to first publish their methods before doing the experiment also helps, as it validates the experimental setup before hand. I also think it’s worth publishing studies where a null result was reached due to some error in experimental setup or other factors, as long as it’s presented as such and reflected upon. This can still be valuable information for future experiments. Offering scientists social capital for that (an “official” publication, citations) might also incentivize scientists to publish the results as is, rather than making it appear as a “true” null result, or even as a non-null one (eg through p hacking). 2. While obviously possible, given the amount of effort scientists have to go through to raise funding for an experiment nowadays, i find it highly unlikely that people would go through this effort. 3. This is already possible and a problem. This is a problem of academic misconduct, has very little to do with null results. The current publishing system is of course already set up to be gamed, so I understand your worries. But null results should be published, as they are just science. Even if someone were to “game” the system by publishing a ton of null results, those publications should be held to the same level of scrutiny as any other publication. If someone is extremely prolific in replicating existing studies and comes up with a ton of null results, that should be lauded and those papers should be published, no? I do believe the entire idea of a researchers output only being recognized by being allowed to be published in a journal is terrible and should be abolished, but baby steps I guess.