4 ms·

Providing _places_ to publish the result is only part of the problem. The other part is incentivizing scientists to do so. And similarly, Nature itself is respo

Providing _places_ to publish the result is only part of the problem. The other part is incentivizing scientists to do so. And similarly, Nature itself is responding to incentives. The core problem is that scientists themselves, the individuals, do not mostly display much interest (in a revealed preference kind of way) for null results. If scientists were interested and wanted to read articles about null results, then either journals like Nature would do so, or the numerous examples of journals like yours that have come and gone over the years would have been more succesful and widespread. Because of this revealed lack of interest, high tier journals don't tend to take of them (correctly responding to the lack of demand), and journals like your that specifically target these kinds of articles A) struggle to succeed and B) remain relatively "low impact", which means that the professional rewards to publishing in them are not very high, which means that the return on effort of publishing such a work is lower. Don't get me wrong, the scientific community could do a lot more to combat this issue, but the core problem is that right now, the "market" is just following the incentives, and the incentives show that, despite what the stream of articles like this one over the past few decades is that most scientists don't seem to actually have an interest in reading null-result papers.

dleeftink 2y ago
What if to each publication of a non-null result, academics are given the opportunity to publish their nulls as well, if only as a appendix or better, a counterpublication to their main conclusions? I don't buy the argument that papers need be of max-n length, now that documents and journals can be easily stored and distributed. I would love something like Living Papers [0][1] to take off, where the null an non-non results could be compared interactively on similar footing. [0]: https://github.com/uwdata/living-papers https://github.com/uwdata/living-papers [1]: https://idl.uw.edu/living-papers-template/ https://idl.uw.edu/living-papers-template/
- bluGill 2y ago
  A null result may be a dead end and so there is no related paper worth publishing it in. A null result should be published right away in a searchable place, but probably isn't worth a lot of effort in general. I tried X, it didn't work, here is the raw data.
  - dleeftink 2y ago
    That's my thought exactly--not a related paper but simply providing additional room for discussing the less shiny bits of the same experiment. Even if the whole thing is a null, the setup, instruments, dependencies and what methods worked/didn't work is worth describing by itself.
    - acchow 2y ago
      All of that - the setup, the instruments, dependencies, methods - should be pre-submitted to the journal before the experimental results arrive. The journal should be the one that uses the data from the experiment and runs your pre-submitted program over the data to produce a result. Papers need to be published backwards.
      - bluGill 2y ago
        Right now you don't even know who will publish you paper until all that is done. Your experiment might be try some promising molecule/drug in a petri dish, and see what happens, if the results are amazing you will get in a different journal than if the results are something happens but the control molecule/drug is better.
        acchow 2y ago
        Right, I'm saying journals should be picking which to publish before the results arrive. They should even publish it before the results arrive. And then publish again after.
      - bumby 2y ago
        I agree that in an idealized way, this would be much better. But what do you do about going through all this process and ending up with a bad reviewer?* In those cases, how would you handle re-submitting to a different journal without looking like you're creating those artifacts after-the-fact to suit your outcome? Would the pre-submittals need to be handled by some third party? * the current process still has a lot of luck in terms of getting assigned referees. Sometimes you just plain get a bad reviewer who just can't be bothered to read the submission carefully and is quick to reject it. I would hate to see a system that only allows for a single shot at publication
c-linkage 2y ago
In the old days, Science Weekly[1] used to print 4-5 paragraph summaries of published research in a three-column layout. The magazine was dense with information across a huge number of topics. And in the very old days, newspapers used to publish in tabular form local election results and sports games. I feel that Nature could dedicate one to two pages of one paragraph summaries of null results with links to the published papers. It's amazingly easy to skim such pages to find interesting interesting things! [1] I think that was the name; I canceled my subscription when they changed to a Scientific American wannabe. I was looking for breadth not depth! I could always get the original paper if I wanted more information.
kkylin 2y ago
I agree incentivization is definitely a big part of the problem, but I think in general a bigger issue is that as a society we tend to reward people who are the first to arrive at a non-null result. This is as true in science as much as in any other area of human endeavor.
borski 2y ago
From the article: “A 2022 survey of scientists in France, for instance, found that 75% were willing to publish null results they had produced, but only 12.5% were able to do so.”
- EvgeniyZh 2y ago
  The question is how many of them are willing to review and read these publications. Of course as an individual scientist (not me, but someone who does experiments), I'd love to capitalize on my work, even if it is unsuccessful (in the sense of null result), by publishing it. But do I, and scientific community in general, care about null results? I'd say mostly no. Null results, if universally published, would overwhelm already overwhelmed publication system. If you think it will be helpful to others to know about specific failure, put it in a blogpost or even on arxiv. Or talk about it at conference (for CS, workshop). Also, if we use publications as a measure of scientists success, and we do, is a scientist with a lot of null results really successful?
  - caddemon 2y ago
    Obviously most scientists are not going to be interested in null results from adjacent subfields, but when it comes to specific questions of interest it is absolutely useful to know what has been tried before and how it was done/what was observed. I know a lab that had documentation not only on their own historical null results but also various anecdotes from colleagues' labs about specific papers that were difficult to replicate, reagents that were often problematic, etc. That is a non-ideal way for the scientific community at large to maintain such info. Trying to go through traditional peer review process is probably also non-ideal for this type of work though, for reasons you cited. We need to be willing to look at publication as something more broadly defined in order to incentivize the creation of and contribution to that sort of knowledge base. It shouldn't be implemented as a normal journal just meant for null results - there's really no need for this sort of thing to be peer reviewed specifically at the prepub stage. But it should still count as a meaningful type of scientific contribution.

dleeftink 2y ago

What if to each publication of a non-null result, academics are given the opportunity to publish their nulls as well, if only as a appendix or better, a counterpublication to their main conclusions? I don't buy the argument that papers need be of max-n length, now that documents and journals can be easily stored and distributed. I would love something like Living Papers [0][1] to take off, where the null an non-non results could be compared interactively on similar footing. [0]: https://github.com/uwdata/living-papers https://github.com/uwdata/living-papers [1]: https://idl.uw.edu/living-papers-template/ https://idl.uw.edu/living-papers-template/

bluGill 2y ago
A null result may be a dead end and so there is no related paper worth publishing it in. A null result should be published right away in a searchable place, but probably isn't worth a lot of effort in general. I tried X, it didn't work, here is the raw data.
- dleeftink 2y ago
  That's my thought exactly--not a related paper but simply providing additional room for discussing the less shiny bits of the same experiment. Even if the whole thing is a null, the setup, instruments, dependencies and what methods worked/didn't work is worth describing by itself.
  - acchow 2y ago
    All of that - the setup, the instruments, dependencies, methods - should be pre-submitted to the journal before the experimental results arrive. The journal should be the one that uses the data from the experiment and runs your pre-submitted program over the data to produce a result. Papers need to be published backwards.
    - bluGill 2y ago
      Right now you don't even know who will publish you paper until all that is done. Your experiment might be try some promising molecule/drug in a petri dish, and see what happens, if the results are amazing you will get in a different journal than if the results are something happens but the control molecule/drug is better.
      - acchow 2y ago
        Right, I'm saying journals should be picking which to publish before the results arrive. They should even publish it before the results arrive. And then publish again after.
    - bumby 2y ago
      I agree that in an idealized way, this would be much better. But what do you do about going through all this process and ending up with a bad reviewer?* In those cases, how would you handle re-submitting to a different journal without looking like you're creating those artifacts after-the-fact to suit your outcome? Would the pre-submittals need to be handled by some third party? * the current process still has a lot of luck in terms of getting assigned referees. Sometimes you just plain get a bad reviewer who just can't be bothered to read the submission carefully and is quick to reject it. I would hate to see a system that only allows for a single shot at publication

c-linkage 2y ago

In the old days, Science Weekly[1] used to print 4-5 paragraph summaries of published research in a three-column layout. The magazine was dense with information across a huge number of topics. And in the very old days, newspapers used to publish in tabular form local election results and sports games. I feel that Nature could dedicate one to two pages of one paragraph summaries of null results with links to the published papers. It's amazingly easy to skim such pages to find interesting interesting things! [1] I think that was the name; I canceled my subscription when they changed to a Scientific American wannabe. I was looking for breadth not depth! I could always get the original paper if I wanted more information.

kkylin 2y ago

I agree incentivization is definitely a big part of the problem, but I think in general a bigger issue is that as a society we tend to reward people who are the first to arrive at a non-null result. This is as true in science as much as in any other area of human endeavor.

borski 2y ago

From the article: “A 2022 survey of scientists in France, for instance, found that 75% were willing to publish null results they had produced, but only 12.5% were able to do so.”

EvgeniyZh 2y ago
The question is how many of them are willing to review and read these publications. Of course as an individual scientist (not me, but someone who does experiments), I'd love to capitalize on my work, even if it is unsuccessful (in the sense of null result), by publishing it. But do I, and scientific community in general, care about null results? I'd say mostly no. Null results, if universally published, would overwhelm already overwhelmed publication system. If you think it will be helpful to others to know about specific failure, put it in a blogpost or even on arxiv. Or talk about it at conference (for CS, workshop). Also, if we use publications as a measure of scientists success, and we do, is a scientist with a lot of null results really successful?
- caddemon 2y ago
  Obviously most scientists are not going to be interested in null results from adjacent subfields, but when it comes to specific questions of interest it is absolutely useful to know what has been tried before and how it was done/what was observed. I know a lab that had documentation not only on their own historical null results but also various anecdotes from colleagues' labs about specific papers that were difficult to replicate, reagents that were often problematic, etc. That is a non-ideal way for the scientific community at large to maintain such info. Trying to go through traditional peer review process is probably also non-ideal for this type of work though, for reasons you cited. We need to be willing to look at publication as something more broadly defined in order to incentivize the creation of and contribution to that sort of knowledge base. It shouldn't be implemented as a normal journal just meant for null results - there's really no need for this sort of thing to be peer reviewed specifically at the prepub stage. But it should still count as a meaningful type of scientific contribution.