4 ms·

You could publish it in the Journal of Trial and Error (https://journal.trialanderror.org https://journal.trialanderror.org), which I created with a number of c

• MostlyStable 2y ago
  Providing _places_ to publish the result is only part of the problem. The other part is incentivizing scientists to do so. And similarly, Nature itself is responding to incentives. The core problem is that scientists themselves, the individuals, do not mostly display much interest (in a revealed preference kind of way) for null results. If scientists were interested and wanted to read articles about null results, then either journals like Nature would do so, or the numerous examples of journals like yours that have come and gone over the years would have been more succesful and widespread. Because of this revealed lack of interest, high tier journals don't tend to take of them (correctly responding to the lack of demand), and journals like your that specifically target these kinds of articles A) struggle to succeed and B) remain relatively "low impact", which means that the professional rewards to publishing in them are not very high, which means that the return on effort of publishing such a work is lower. Don't get me wrong, the scientific community could do a lot more to combat this issue, but the core problem is that right now, the "market" is just following the incentives, and the incentives show that, despite what the stream of articles like this one over the past few decades is that most scientists don't seem to actually have an interest in reading null-result papers.
  • dleeftink 2y ago
    What if to each publication of a non-null result, academics are given the opportunity to publish their nulls as well, if only as a appendix or better, a counterpublication to their main conclusions? I don't buy the argument that papers need be of max-n length, now that documents and journals can be easily stored and distributed. I would love something like Living Papers [0][1] to take off, where the null an non-non results could be compared interactively on similar footing. [0]: https://github.com/uwdata/living-papers https://github.com/uwdata/living-papers [1]: https://idl.uw.edu/living-papers-template/ https://idl.uw.edu/living-papers-template/
    • bluGill 2y ago
      A null result may be a dead end and so there is no related paper worth publishing it in. A null result should be published right away in a searchable place, but probably isn't worth a lot of effort in general. I tried X, it didn't work, here is the raw data.
      • dleeftink 2y ago
        That's my thought exactly--not a related paper but simply providing additional room for discussing the less shiny bits of the same experiment. Even if the whole thing is a null, the setup, instruments, dependencies and what methods worked/didn't work is worth describing by itself.
        • acchow 2y ago
          All of that - the setup, the instruments, dependencies, methods - should be pre-submitted to the journal before the experimental results arrive. The journal should be the one that uses the data from the experiment and runs your pre-submitted program over the data to produce a result. Papers need to be published backwards.
          • bluGill 2y ago
            Right now you don't even know who will publish you paper until all that is done. Your experiment might be try some promising molecule/drug in a petri dish, and see what happens, if the results are amazing you will get in a different journal than if the results are something happens but the control molecule/drug is better.
• Waterluvian 2y ago
  Omg I love this. For like 20 years I’ve joked about “The Journal of Null Results and Failed Experimenrs” and it looks like you and your friends are actually doing it. There’s so much to learn from these cases.
• greenavocado 2y ago
  If you publish null results you accelerate the development of competing hypotheses by your competition. It's best to make sure they waste as much time as possible so you can maintain an edge and your reputation. /s
  • tefkah 2y ago
    so trueeee! science is ofc a zero-sum game, and we should stifle competition and collaboration as much as possible. how else will i get that 3rd citation on my phd thesis? /s
• bumby 2y ago
  Years ago, I came across SURE: Series of Unsurprising Results in Economics with the goal of publishing good, but statistically insignificant, research. https://blogs.canterbury.ac.nz/surejournal/ https://blogs.canterbury.ac.nz/surejournal/
  • jraph 2y ago
    I thought "statistically insignificant" meant we couldn't conclude anything. So I was surprised. [1] says: > In statistical hypothesis testing,[1][2] a result has statistical significance when a result at least as "extreme" would be very infrequent if the null hypothesis were true So I understand this journal publishes results for which a hypothesis was tested, found to give insignificant results, which would rule out the hypothesis assuming the research was correctly conducted, without biases in the methodology, with a big enough sample, etc. Which would be worthy to know but no journal usually takes this research because it doesn't make the headlines (which yes, I've always found was a shame). Do I get this right? [1] https://en.wikipedia.org/wiki/Statistical_significance https://en.wikipedia.org/wiki/Statistical_significance
    • bumby 2y ago
      Yes, statistical insignificance doesn't "prove" the null hypothesis, it just fails to reject it. It's a subtle, but sometime misunderstood distinction. It's a measure of how big the effect size is and how often you'd expect to see it just by chance rather than due to the variables you're measuring. If it's a really extreme difference, we expect it to happen less often just by chance alone than if it's a really miniscule difference. >Which would be worthy to know but no journal usually takes this research because it doesn't make the headlines. That's usually correct, which gives rise to all kinds of issues like the article talks about. It can result in a lot of wasted time (when you're conducting what you "think" is a new experiment, but it's been done many times but unpublished because it doesn't provide statistically significant results). It provides little incentive for replication, which can lead to stronger conclusions about the results than may be warranted, etc.
      • foldr 2y ago
• greentxt 2y ago
  Good idea but bad name. Error implies mistakes, which will deter people.
  • tefkah 2y ago
    Disagree, as we also publish “failed” research, where authors reflect on their experiment, such that others may learn from it, and that the others may still gain something useful academically from this (citations, a publication). One of our goal is to change the perception of and culture around failure in academia. Research/science is not just a steady upward trend of progress, it comes with a lot of trial and error. Academics’s success and job prospects however depend mostly on them publishing in high impact journals, which in turn only publish “interesting” aka positive results, which creates this very toxic publish-or-perish culture. Having an experiment fail is a natural part of doing science, but academic institutions punish you for not producing positive results. By providing a place to publish these failed experiments, at least it provides some relief for this problem. This is not real change however, that needs to happen at a much higher level, but that is one we do are not able to impact. Ideally our journal will not be necessary in the future, as we detail in our opening editorial: https://doi.org/10.36850/ed1 https://doi.org/10.36850/ed1
    • greentxt 2y ago
      I see. I guess I see part of the problem you're trying to solve as reflected in the language of "failed" and "error" as opposed to framing say non-replication of a prior false positive as a "correction" or "additional evidence" (against a prior false positive). It may not matter to everyone but some funders might wince as research they've funded as being "failure" or "error." Just something to consider.