14 ms·

You got a null result. Will anyone publish it?

256 points by sohkamyung 2y ago

transcriptase 2y ago
The irony of this appearing on the Nature site… when their own editors routinely reject even remarkable results that go on to become highly cited seminal papers in “lesser” journals.
xpe 2y ago
> found that 75% were willing to publish null results they had produced, but only 12.5% were able to do so What are the corresponding statistics for researchers that find positive results? Closer to 100% are willing to publish? And how many succeed?
glial 2y ago
The article hints at this, but not publishing null results (at least in a database - somewhere!) goes hand-in-hand with the replication crisis. An experimental outcome is always a single sample from a distribution of outcomes that you would obtain if you repeated the experiment many times. Choosing to only publish the most extreme positive values means that when the experiment is replicated, "regression to the mean" makes it very likely that the measured effect will be weaker, and possibly not statistically significant. This is not an evidence of scientific fraud -- rather, it is a predictable outcome of a publishing incentive scheme that rewards hype and novelty over robust science. I've said it before but it bears repeating - replicating published results, and adding the findings to a database, should be a standard part of PhD training programs.
- setopt 2y ago
  Relevant XKCD: https://xkcd.com/882/ https://xkcd.com/882/ Do 20 experiments with a p<5% criterion, and it’s likely that one will be a false positive. Only publish positive results, and someone will eventually publish a false positive result without fraud.
  - etrautmann 2y ago
    Very few studies report a single statistical test as the sole conclusion. Most papers should assess some outcome in multiple ways using complementary data, multiple analyses etc. not always of course, but there are lots of ways of making sure your conclusions a robust without relying on a single analysis result.
    - kerkeslager 2y ago
      That doesn't fix the problem at all. No matter how many statistical tests you run on a sample, you can't get around the fact that the sample may not be representative of the population or the underlying phenomenon. You need different samples. There isn't a statistical trick that gets around this. For example: let's say there's a cancer with 20% survival rate. You test a treatment with 25 experimental and 25 control patients, 40% in the experimental group survive[1]. You can analyze this with a bunch of statistical methods. You can ask different questions about the patients, focusing on well-being rather than simple carcinogenic remission. But ultimately, the thing that happened in this study is that 50% got better and no fiddling with numbers or changing the questions you ask is going to change that underlying phenomenon. You can check for blood markers of cancer: you get 50% have no blood markers. You ask them questions about how they feel: you get 50% feel better. You body scan the area where tumors were: you get 50% no longer have tumors. You have only tested one phenomenon in one sample, and that essentially amounts to 5 people getting better. [1] I know this is not how cancer treatment studies work exactly, this is a simplified hypothetical.
      - etrautmann 2y ago
        That’s not what I’m saying. Obviously you can’t run multiple tests on one set of data. If you have a hypothesis, test it on multiple data sets in multiple ways and find supporting evidence rather than finding a single p=.05 and writing a story around it.
        kerkeslager 2y ago
        Okay, if that's what you're saying, then I don't understand why it sounds like you're disagreeing with the poster you're responding to.
- mistermann 2y ago
  > This is not an evidence of scientific fraud... In its scriptures/philosophy, science describes extremely thorough and sound principles and guidelines...but in on the ground practice (by scientists, which are a part of "science"), they are often not achieved[1]. However, this distinction is not only not advertised broadly and without aversion, it is usually (in my experience) not mentioned at all, if not outright denied using persuasive rhetorical language (like, for example, when an object level instance of not achieving it is pointed to in the wild, such as in forum conversations). This may not be fraud (that requires intent I think?), but it achieves the same end: misinforming people. I absolutely agree with your database idea, and if science would like me to take them seriously (something near how seriously they take themselves) they'd also have to go much further. [1] Not unlike in religion, a competing metaphysical framework (model of reality) to science.
  - 3np 2y ago
    > Not unlike in religion, a competing metaphysical framework (model of reality) to science. No. Correlation fallacy.
    - mistermann 2y ago
      Fallacy fallacy. Naive Realism fallacy.
  - kerkeslager 2y ago
    This sort of comment is why I think a lot of philosophy is just communicating poorly to make yourself sound smart. In your footnote, for example, you translated your philosophy-speak into English (metaphysical framework -> model of reality). Why not just say that? Your entire comment goes into "philosophy mode" and communicates a few very simple ideas in overcomplicated language. Science and religion are pretty poorly understood as competing models of reality. Religion originates when people make up answers to other people's questions to gain social standing, and religion continues due to (among other things) anchoring bias--the bias people have toward continuing to believe what they already believe. While religion does result in those people having a model of reality, there is no attempt being made at any point to relate the model to reality. When religious people and scientists disagree, it's not because the religious person is trying to model reality differently--the religious person isn't even trying to model reality--it's because the religious person is biased in favor of their existing belief. You said: > In its scriptures/philosophy, science describes extremely thorough and sound principles and guidelines...but in on the ground practice (by scientists, which are a part of "science"), they are often not achieved[1]. This is presented as some sort of gotcha, but it's not: few scientists will claim that science is being practiced perfectly or even well. Outside of a few areas such as particle physics, we're quite aware that our ability to practice scientific ideals is hampered by funding, publication incentives, availability of test subjects in human studies, data privacy, etc. And we're aware that this means that our conclusions need to be understood as probabilities rather than 100%-confidence facts. There are certainly some people who treat scientific conclusions with religious absolute confidence, but doing that is fundamentally against scientific principles. The accusation you are leveling against science would be better targeted toward people: generally science journalists and the science-illiterate public rather than scientists themselves. The entire reproducibility crisis is scientists using science to show that our practice of science is too imperfect to result in high-confidence conclusions. Religious people jumping on the replication crisis because they think it disproves science is rich. The replication crisis isn't a disproof of science, it's an application of science. The reason we know that there's a replication crisis is because scientists asked "How confident can we be in the conclusions of existing studies?" and applied science to answer that question. If you really think science is invalid, then you can't use science to prove that. And the fact remains that any confidence in conclusions at all is more than religion has to offer, because again, religion isn't trying to model reality--the fact that religion produces a model of reality is merely an unfortunate side-effect.
- hcks 2y ago
  “We thought academia was not soul crushing enough so from now on you will additionally spend 10 hours a week replicating dumb papers from 1993”
  - bumby 2y ago
    Snark aside, replication is a cornerstone of science. If someone doesn't want to be involved in science because they think it's soul-crushing, perhaps academia isn't the right place for them.
- matthewdgreen 2y ago
  There is a major issue of limited resources to replicate results, in terms of both time and funding. For example: I would assume that most important results are replicated. As a concrete example, if someone identifies a medication that (in one small trial) shows a statistically significant effect in curing some serious medical condition, then this will drive further replication attempts. On the other hand, if someone publishes a study showing that holding a pen in your mouth makes you 1% likelier to do well on the PSATs, this study will probably languish without replication for a decade — because honestly who cares? It’s basically a curiosity. I can’t help but notice that many of the headline results that characterize the “replication crisis” were small-effect-size social science experiments that fundamentally weren’t that important outside of popular science news. I’m not saying that our current allocation of resources is optimal. I am pointing out that our resources are finite and “replicate everything” is not even a remotely practical allocation of those resources.
  - nostrademons 2y ago
    > I would assume that most important results are replicated. GP is pointing out that the incentive structure makes this an invalid assumption. If publications reward hype and novelty when deciding what to publish, then there is no point spending your limited resources replicating other peoples' results, they won't get published anyway. And experiments that give a null result won't be published anyway. What's left are one-off results that showed something surprising simply by chance and don't replicate...but then, we generally will never know that they don't replicate, because the replication experiment is not novel, has a low chance of being published, and hence isn't worth spending limited resources on. Basically the publication process introduces selection bias into the types of research that are even attempted, which then filters down into the conclusions we take from it. A cornerstone of the scientific method is random sampling, but as long as the results that get disseminated are chosen by a non-random process, it introduces bias.
  - freestyle24147 2y ago
    > For example: I would assume that most important results are replicated. The example you provide is solely your assumption? Seems pretty odd to provide a baseless assumption as an "example".
- YeGoblynQueenne 2y ago
  >> I've said it before but it bears repeating - replicating published results, and adding the findings to a database, should be a standard part of PhD training programs. Wait, why should PhD students do that work? That just sounds like pushing more grunt work to the lower rung of the academic hierarchy. Nope. If you want people to do that kind of work that is important to everyone but is not directly conducive to promoting one's research career then the solution is simple: pay them.
  - bumby 2y ago
    >why should PhD students do that work? I think there is some reasonable argument that replicating research is the first step to learning how to do good research on your own. In an ideal world, PhD students should probably be trying to replicate similar work anyway and applying existing approaches own pet problem. In practice, many gloss over this because they are narrowly focused on doing something "new" so it can get published.
  - glial 2y ago
    PhD students are in training, and replicating a published result is a great training exercise. PhD students ARE paid. But this work won't be prioritized by their PI unless it's also a requirement of the program.
    - YeGoblynQueenne 2y ago
      >> PhD students are in training, and replicating a published result is a great training exercise. I just got my PhD last July and that sounds like boring drudgery rather than "good exercise". Good exercise is to have a student write their own paper, dig up their own references, formulate their own experimental hypotheses, run their own experiments, write up their own results etc. Re-doing someone else's possibly badly-done work is grunt work that should not be forced upon anyone. >> PhD students ARE paid. Haha. Good one XD
    - anticensor 2y ago
      Not all doctoral students are paid as part of a research position.
- deleted 2y ago
  [deleted]
- aeternum 2y ago
  It's kind of amazing that we discovered the scientific method, used it to invent the transistor and bring the information revolution. Yet we still pool scientific results using only the printing press. It's like we unlocked the tech tree but then got so caught up in chasing citations and peer review that we forgot to use the new tech we invented.
  - glial 2y ago
    Yes, so-called "social technology" sometimes doesn't feel very advanced.
- Gormo 2y ago
  Richard Feynman was complaining about exactly this phenomenon fifty years ago: https://sites.cs.ucsb.edu/~ravenben/cargocult.html https://sites.cs.ucsb.edu/~ravenben/cargocult.html
- naasking 2y ago
  > This is not an evidence of scientific fraud -- rather, it is a predictable outcome of a publishing incentive scheme that rewards hype and novelty over robust science. And knowing all this, this behavy should be considered borderline scientific fraud at this point.
calibas 2y ago
The process for publishing, how a study becomes "legitimate" science, is not very scientific. Same with the process for getting funding for studies.
- nick238 2y ago
  My thesis committee chair bristled when I said I hated the marketing aspect of academic science and I implied that he was a very good marketer because he played the game so well. After I kept making comparison after comparison, he didn't have much to say in response. I would say he begrudgingly accepted my view, but I don't think accepted it at all, he just couldn't refute any point I made.
deleted 2y ago
[deleted]
cactusfrog 2y ago
Just write a preprint or a blog post
- zug_zug 2y ago
  A blog post won't be indexed on google scholar for other academics to reference though. Maybe a preprint would be? In my opinion the goal is to get a record of the information and the dataset out there.
  - tokai 2y ago
    >A blog post won't be indexed on google scholar It will if its on an edu domain.
- cydodon 2y ago
  A preprint will still give less value to the negative result. No peer review (being a broken system or not) and not being published in a "proper" journal will make it less likely that the results will be recognised / accepted. The whole point is, that a negative result can have as much value as a positive one...
  - smcin 2y ago
    Why was this comment flagged, it's reasonable?
- vsuperpower2021 2y ago
  The whole point of publishing studies is to be able to brag about how many impressions you have, which is good for your career. Who is going to care that your blog got views?
mlhpdx 2y ago
This seems like a tough problem to solve, but given the article states 75% of researchers are _willing_ to publish the null results at least that's something to build on. Making publishing compulsory could lead to other, worse problems given humans are involved. I realize as I'm writing this that I don't really understand what "publishing" means. It's more than just making the paper available, right? Is there a formal definition, or just a colloquial one in science?
- michaelt 2y ago
  > I realize as I'm writing this that I don't really understand what "publishing" means. 1. Fully gather and analyse the data, no stopping early when you realise it isn't working 2. Write the paper, read those background papers you hadn't got to yet so you can cite them, chase down references for things you know from memory. 3. Realise there's a gap in your table because you tested A, B and D at three levels each but C you only tested at the low and high level, not at the medium level. Go set up your test equipment again to fill in the blank space. 4. Run the paper by your collaborators and your boss, all of whom will feel obliged to suggest at least some improvements, which you'll make. 5. Choose a journal, apply the journal's template and style, send it in. 6. Wait for as much as several months for peer review. 7. The first peer reviewer suggests you retest with a slightly different protocol for cleaning your equipment before the test. You do so. 8. The second peer reviewer replies suggesting you test combinations of A, B, C and D, not just one at a time....
- eyeundersand 2y ago
  From my experience, there's no definition as such but having your study "published" implies that it went through a peer-review process featuring at least two qualified referees and an editor. The implication being that the claims from the study are valid as reference for future studies, to varying extent depending on the quality of the journal etc.
- elashri 2y ago
  It does not work as standard anyway. The publishing practices are different from field to field. Also journals have different policies and practices. So it is hard to get a real representative definition other than making paper available. In which case arxiv will be a publishing mechanism that does not provide editor, peer-reviewed and does not cost money.
parpfish 2y ago
Publishing null results would be great, but I’m worried about how that could also be gamed. How do you distinguish a ‘real’ null result from one done in a sloppy study? Would people run shoddy experiments to get null results to undermine their rivals? Could somebody pump out dozens of null publications to pad their CV and screw up h-indexes?
- vharuck 2y ago
  Currently, the system can be gamed exactly as you say by publishing sloppy studies that falsely find a "real" effect. But editors and readers of the article will look at and judge the methodology section. Sloppy experiments risk not being printed or cited. >Would people run shoddy experiments to get null results to undermine their rivals? In this case, the rival would be very much inclined to recreate the "null" experiment. >Could somebody pump out dozens of null publications to pad their CV and screw up h-indexes? Possibly, but would null publications be cited as often? Also, who's going to keep funding a researcher that mostly publishes null results?[0] [0] Besides agenda-driven "think tanks". Which is worrying itself.
  - parpfish 2y ago
    > Possibly, but would null publications be cited as often? Don’t underestimate an academics ability to cite ALL of their previous publications each time they publish
- tefkah 2y ago
  I run a journal where we publish both “real” null results and experiments were something practical went wrong, so i have a few thoughts: 1. Ideally peer review would catch this. A badly setup study should be critiqued in peer review. Forcing scientists to first publish their methods before doing the experiment also helps, as it validates the experimental setup before hand. I also think it’s worth publishing studies where a null result was reached due to some error in experimental setup or other factors, as long as it’s presented as such and reflected upon. This can still be valuable information for future experiments. Offering scientists social capital for that (an “official” publication, citations) might also incentivize scientists to publish the results as is, rather than making it appear as a “true” null result, or even as a non-null one (eg through p hacking). 2. While obviously possible, given the amount of effort scientists have to go through to raise funding for an experiment nowadays, i find it highly unlikely that people would go through this effort. 3. This is already possible and a problem. This is a problem of academic misconduct, has very little to do with null results. The current publishing system is of course already set up to be gamed, so I understand your worries. But null results should be published, as they are just science. Even if someone were to “game” the system by publishing a ton of null results, those publications should be held to the same level of scrutiny as any other publication. If someone is extremely prolific in replicating existing studies and comes up with a ton of null results, that should be lauded and those papers should be published, no? I do believe the entire idea of a researchers output only being recognized by being allowed to be published in a journal is terrible and should be abolished, but baby steps I guess.
proof_by_vibes 2y ago
I recall reading someone who proposed the need for what they dubbed "meta-science," and I think it's clear that this concept is becoming more needed as time goes on. Our publishing process, and the incentives therein, are obviously faulty and we are aware of it. We can do the math: I believe it's time we do away with playing speculative games with science.
- shishy 2y ago
  "Science of Science" is a good book / overview of this field!
shishy 2y ago
Research should require pre-registration like clinical trials so others have visibility into failed outcomes
- elashri 2y ago
  Who would track this registration and would it require approvals now?. Then what if you changed your current research because you have personal reasons, change of plans, didn't see it fitting....etc. How would you handle these situations? And why are you introducing MITMs.
  - _flux 2y ago
    In basic level, maybe some publications that print these papers would take the registrations, as a precondition to publishing them? It's not like it couldn't be gamed, but maybe it would incentivize people to also publish null results.
    - elashri 2y ago
      >incentivize people to also publish null results This will hardly achieve this goal as you basically make things harder, and now you are introducing more overhead. The main reason why people don't like publishing null results is that it hurts them in funding applications. The current system works with mentality, we shouldn't fund someone who don't get positive results. It is better to allocate this somewhere else. Most of the problems with research can be tracked down to funding issues and practices. But these are political issues, so people try to argue about other things because it is easy.
      - shishy 2y ago
        All good points- I was imagining that from a funder's perspective, knowing null results is actually important to guaranteeing future positive results / being strategic about what is worth funding in (versus what is destined to fail because others failed and they just didn't know). Not sure how it would work in practice; might be bumpy but certainly seems worthwhile / not impossible.
hcks 2y ago
The introductory example is quite illuminating. No theory behind, just a random hypothesis tested (by comparing the preferences of 10 fishes from populations separated from as little as 50 meters, what effect size did the authors expect?), with claims of generalisation (climate change ok bad?? Or climate change bad bad??)
jjmarr 2y ago
Would recommend people interested in this start following "metascience". https://en.wikipedia.org/wiki/Metascience https://en.wikipedia.org/wiki/Metascience And read "Why Most Published Research Findings Are False". https://en.wikipedia.org/wiki/Why_Most_Published_Research_Findings_Are_False https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...
- sunshinesnacks 2y ago
  There’s a bit of irony in the second reference, as the author ended up with some controversial work related to COVID-19. He co-authored a study that was widely cited to downplay severity of the pandemic, but was also heavily criticized for poor methodology (and later I think firmly found to be very wrong). He also published a paper with personal attacks of a grad student that had disagreed with him, which is probably not in the spirit of encouraging constructive science.
  - roland_nilsson 2y ago
    So he proved his point! :)
tefkah 2y ago
You could publish it in the Journal of Trial and Error (https://journal.trialanderror.org https://journal.trialanderror.org), which I created with a number of colleagues a couple years ago! Our editor-in-chief was interviewed for this related Nature article a couple months ago (https://www.nature.com/articles/d41586-024-01389-7 https://www.nature.com/articles/d41586-024-01389-7). While it’s easy pickings, it’s still always worth pointing out the hypocrisy of Nature publishing pieces like this, given that they are key drivers of this phenomenon by rarely publishing null results in their mainline journals. They are have extremely little incentive to change anything about the way scientific publishing works, as they are currently profiting the most from the existing structures, so them publishing something like this always leaves a bit of a sour taste.
- MostlyStable 2y ago
  Providing _places_ to publish the result is only part of the problem. The other part is incentivizing scientists to do so. And similarly, Nature itself is responding to incentives. The core problem is that scientists themselves, the individuals, do not mostly display much interest (in a revealed preference kind of way) for null results. If scientists were interested and wanted to read articles about null results, then either journals like Nature would do so, or the numerous examples of journals like yours that have come and gone over the years would have been more succesful and widespread. Because of this revealed lack of interest, high tier journals don't tend to take of them (correctly responding to the lack of demand), and journals like your that specifically target these kinds of articles A) struggle to succeed and B) remain relatively "low impact", which means that the professional rewards to publishing in them are not very high, which means that the return on effort of publishing such a work is lower. Don't get me wrong, the scientific community could do a lot more to combat this issue, but the core problem is that right now, the "market" is just following the incentives, and the incentives show that, despite what the stream of articles like this one over the past few decades is that most scientists don't seem to actually have an interest in reading null-result papers.
  - dleeftink 2y ago
    What if to each publication of a non-null result, academics are given the opportunity to publish their nulls as well, if only as a appendix or better, a counterpublication to their main conclusions? I don't buy the argument that papers need be of max-n length, now that documents and journals can be easily stored and distributed. I would love something like Living Papers [0][1] to take off, where the null an non-non results could be compared interactively on similar footing. [0]: https://github.com/uwdata/living-papers https://github.com/uwdata/living-papers [1]: https://idl.uw.edu/living-papers-template/ https://idl.uw.edu/living-papers-template/
EricE 2y ago
Remember all the cries of "the science is settled"! Yeah, that's not science - it's the exact opposite of science. This is the perfect example of why reasoned skepticism is more necessary than ever. Blind trust in any institution is a recipe for disaster.
__MatrixMan__ 2y ago
I wish we would disentangle publication from endorsement. Making the bits available and saying that they're useful are different things. Your null result could contain data which is relevant to some other inquiry. All results should be published, some should be celebrated.
- Brian_K_White 2y ago
  The example null correlation sure sounds as significant as any correlation.
- tmalsburg2 2y ago
  Publication servers like arXiv + overlay journals. I‘d love that.
- KeplerBoy 2y ago
  This goes both ways. Some people publish fantastic papers without data nor code. Sometimes annoying, other times a complete waste of everyone's time.
  - __MatrixMan__ 2y ago
    Does it? I mean there's a lot of trash on the internet that just doesn't get looked at. If you waste your time reading it, that's on you.
    - __MatrixMan__ 2y ago
      I just reread this and realized how much of a jerk I'm coming off as (too late to delete it). My point is just that if you're reading something because it was published then publication and endorsement must not be sufficiently decoupled. In a world where they are decoupled, you'd only end up reading it because somebody you trust endorsed it (no reason for that to coincide with the publisher) or because you went searching for it despite its lack of endorsement--in which case you probably know what you're doing.
yzydserd 2y ago
It’s a shame to see no mention of https://opentrials.net/ https://opentrials.net/ in the article.
SeanLuke 2y ago
I think the fundamental problem is that for every claim yielding a positive result there are many more, perhaps infinitely more, related claims yielding negative results. Positive result claim: the sun comes up in the morning. Negative result claims: the sun moves sideways in the morning. The sun was always there. The sun peeks up in the morning and immediately goes back down. And so on. Positive result claim: aspirin is an effective pain reliever. Negative result claims: eating sawdust is an effective pain reliever. Snorting water is an effective pain reliever. Crystal Healing is an effective pain reliever. Etc. Because there are so many negative results, it's trivial to construct an experiment which produces one. So why should that be published? Negative results should be published when people in the community are asking that question, or have a wrong belief in the answer (hence the replication crisis). But if nobody cares about the question, it's hard to argue for why a given negative result would be preferred over any other negative result for purposes of publication.
- dfgtyu65r 2y ago
  None of these are negative results in the sense of being a 'null' hypothesis? In the language of hypothesis testing you have your null and alternative hypotheses. So for alternative hypothesis that the sun comes up in the morning, the null hypothesis would simply be that the sun does not come up in the morning. Each of the negative results, reads to me like a separate 'alternative' hypothesis.
  - SeanLuke 2y ago
    Sure they are. So let's say I claim that the sun goes in a circle in the sky in the morning. The null hypothesis is that it doesn't do that. Perform experiment. Null hypothesis wins. Write up paper! This is a negative result. The point is that for every result where the alternative hypothesis wins, there are a massive, if not infinite, number of results where the null hypothesis will win. Are these publishable?
    - nick238 2y ago
      The idea is that some null hypotheses being true is actually interesting because it challenges an assumed belief. From the first paragraph of the article, the immediate feedback from the postdoc's supervisor was 'you did it wrong [because everyone knows that fish do like warmer water]'. > It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so.
mettamage 2y ago
Is it an idea to publish null results in appendix when a sexy result will be published? Kinda like a Thomas Edison thing. How many ways are there to not make a lightbulb, included with the ways that do make it possible to create one
bowsamic 2y ago
Researchers are expected to publish papers at such a frequency today that spending time writing a paper for a null result would be considered a bad career move
bandrami 2y ago
There is a psychology journal specifically for null hypothesis results: https://www.jasnh.com/ https://www.jasnh.com/
ketanmaheshwari 2y ago
Relevant, I run a workshop for negative results: https://error-workshop.org/ https://error-workshop.org/
fharding 2y ago
In cryptology there’s something called CFail, which is a bit like this. https://www.cfail.org/call-for-papers https://www.cfail.org/call-for-papers
deleted 2y ago
[deleted]
gtmitchell 2y ago
As someone whose early scientific career was destroyed by null results, no. No one will publish your negative results. Unless you win the lottery and stumble across a once-in-a-generation negative result (e.g. the Michelson–Morley experiment), any time you spend working on research that yields negative results is essentially wasted. This article completely glosses over the fact that to publish a typical negative result, you need to have progressed your scientific career to the point where you are able to do so. To get there, you need piles of publications, and since publishing positive results is vastly easier than publishing negative ones, everyone is incentivized to not waste time on the negative ones. You either publish or you perish, after all. Simply put, within the current framework of how people actually become scientists and do research, there is no way to solve the 'file drawer' problem. You might see an occasional graduate student find something unusual enough to publish, or an already-tenured professor with enough freedom to spend the time submitting their manuscript to 20 different journals, but the vast majority of scientists are going to drop any research avenue that doesn't immediately yield positive results.
gzer0 2y ago
The main issue with publishing research is the cost. Some time ago, I worked in a research lab studying Lupus. Our results were negative, and my initial inclination was not to publish them. However, my Principal Investigator (PI) emphasized that all results, whether positive or negative, should be published. Fortunately, we had the funds to do so. At that time, publishing in a reputable journal cost $2,300. Not everyone is so fortunate. This lesson has stuck with me, as I have seen or heard from different labs where, unfortunately, they couldn't afford to publish their findings.
nick238 2y ago
What's the last null result Nature, Science, and Cell published?
SubiculumCode 2y ago
Perhaps my most prominent paper was a null result. If the question is important enough, the work to test it non-trivial, it will find an audience and a likely a reputable journal. What is the value in reporting a null result other than reducing file drawer effects? Well one, even though the null does not tell us whether the effect is absent, it does help suggest bounds on how large an effect could be if it did exist. In my case, a particularly large effects were observed in cross-sectional comparisons (different people at different ages), but our research showed that longitudinal changes within individuals were generally negligible, suggesting systematic bias in the cross-sectional sampling.
ok123456 2y ago
Why not self-publish it on Arvix?
bachmeier 2y ago
The problem is that peer review is primarily focused on results. Peer review should be done up to but not including the results. Provide motivation, explain your methodology, explain how it will resolve issues in the literature, but don't say anything about your results. Papers should be conditionally accepted, subject to confirmation that the results you report are the results of the proposal that went through peer review.
joemazerino 2y ago
Crowdstrike will.
analogwzrd 2y ago
Some others have mentioned this in their comments and I agree that once you succeed in getting a non-null result, publishing the null results (all the things you tried that didn't work) could be included as appendices or something. Also, just because you get a null result doesn't mean that nothing was learned, that something new (and unexpected) wasn't stumbled on, or that some innovation didn't happen. There are tiers of publications and journals. Even if you get a null result and you're not going to get it accepted in Nature, it's very possible that you can get a conference paper (sometimes peer reviewed) out of something that was learned.
lokimedes 2y ago
We published a load of null results in particle physics. Simply go to arxiv.org and look for papers beginnig with “search for…” that would be a null result. Well technically a sigma<3 result.
diffxx 2y ago
True story: I wrote my ph.d. thesis on a special case of a general problem. After about one year of work, I realized that the approach would never work for the general problem for intractable reasons. But I also really wanted to finish my ph.d. within 5 years, so I spent the next two years refining the work enough to be able to write a dissertation on it and ignored the fact that it would never really work for what it was intended. I did do some interesting work and learned a lot, but I couldn't really bring myself to try and publish the results (beyond my thesis) because I very clearly had not made an advance in the field. Of course, I do think it would have been useful to publish why I thought that essentially the entire field of inquiry was a dead end, but that would not have made me very popular with my collaborators or others in the field and it wouldn't likely have ingratiated me with anyone else.
- okwhateverdude 2y ago
  Great, so the economic {dis|in}centives are keeping a bunch of people fed but doing useless study of a dead end and even kept you from pointing out the emperor has no clothes (assuming your epiphany is valid). Is social conformity holding back science?
jimmar 2y ago
Not all null results are created equal. To get your null results published, the null result must shed light on some phenomena. And of course, the study must be sound. E.g., everybody takes for granted that X causes Y, but in a well-designed study, X did not in fact cause Y, which reveals an error in what we assumed was true.
jenders 2y ago
Crowdstrike will
sandraruby512 2y ago
[dead]
roland_nilsson 2y ago
I think this problem is at least in part due to the hypothesis testing concept itself. Classical hypothesis testing is asymmetric: there is a "null" hypothesis, which is typically the uninteresting/useless case, and an "alternative" hypothesis, which is the one you would like to be true. Critically, you cannot determine if the data _supports_ the null hypothesis, only if the data _rejects_ it (and supports the alternative). A so-called "null result" occurs when data is not sufficient to reject the null hypothesis; then you can't tell if you actually have a useful finding (for example, that there is no major difference between species A and B) or a failed experiment (data was so bad/noisy that we cannot conclude anything). And so you end up with the unfortunate situation where you either succeed in proving your favorite hypothesis and get your degree / promotion / tenure, or you have nothing. This happens because hypothesis testing conflates effect size (how big is the difference between A and B) with uncertainty about that effect size (significance/reproducibility). Confidence intervals are more useful IMHO, as they help untangle these two aspects, for example showing that the difference between A and B is small _and_ reproducible. Bayesian analysis is also a major improvement, as it allows examining both the "null" and "alternative" hypotheses on equal terms, as well as reasoning about our prior beliefs / biases. Unfortunately many areas of science are still stuck with statistical methods from the early 1900's.
nikau 2y ago
Crowdstrike will

256 points by sohkamyung 2y ago

The irony of this appearing on the Nature site… when their own editors routinely reject even remarkable results that go on to become highly cited seminal papers in “lesser” journals.

xpe 2y ago

> found that 75% were willing to publish null results they had produced, but only 12.5% were able to do so What are the corresponding statistics for researchers that find positive results? Closer to 100% are willing to publish? And how many succeed?

glial 2y ago

The article hints at this, but not publishing null results (at least in a database - somewhere!) goes hand-in-hand with the replication crisis. An experimental outcome is always a single sample from a distribution of outcomes that you would obtain if you repeated the experiment many times. Choosing to only publish the most extreme positive values means that when the experiment is replicated, "regression to the mean" makes it very likely that the measured effect will be weaker, and possibly not statistically significant. This is not an evidence of scientific fraud -- rather, it is a predictable outcome of a publishing incentive scheme that rewards hype and novelty over robust science. I've said it before but it bears repeating - replicating published results, and adding the findings to a database, should be a standard part of PhD training programs.

setopt 2y ago
Relevant XKCD: https://xkcd.com/882/ https://xkcd.com/882/ Do 20 experiments with a p<5% criterion, and it’s likely that one will be a false positive. Only publish positive results, and someone will eventually publish a false positive result without fraud.
- etrautmann 2y ago
  Very few studies report a single statistical test as the sole conclusion. Most papers should assess some outcome in multiple ways using complementary data, multiple analyses etc. not always of course, but there are lots of ways of making sure your conclusions a robust without relying on a single analysis result.
  - kerkeslager 2y ago
    That doesn't fix the problem at all. No matter how many statistical tests you run on a sample, you can't get around the fact that the sample may not be representative of the population or the underlying phenomenon. You need different samples. There isn't a statistical trick that gets around this. For example: let's say there's a cancer with 20% survival rate. You test a treatment with 25 experimental and 25 control patients, 40% in the experimental group survive[1]. You can analyze this with a bunch of statistical methods. You can ask different questions about the patients, focusing on well-being rather than simple carcinogenic remission. But ultimately, the thing that happened in this study is that 50% got better and no fiddling with numbers or changing the questions you ask is going to change that underlying phenomenon. You can check for blood markers of cancer: you get 50% have no blood markers. You ask them questions about how they feel: you get 50% feel better. You body scan the area where tumors were: you get 50% no longer have tumors. You have only tested one phenomenon in one sample, and that essentially amounts to 5 people getting better. [1] I know this is not how cancer treatment studies work exactly, this is a simplified hypothetical.
    - etrautmann 2y ago
      That’s not what I’m saying. Obviously you can’t run multiple tests on one set of data. If you have a hypothesis, test it on multiple data sets in multiple ways and find supporting evidence rather than finding a single p=.05 and writing a story around it.
      - kerkeslager 2y ago
        Okay, if that's what you're saying, then I don't understand why it sounds like you're disagreeing with the poster you're responding to.
mistermann 2y ago
> This is not an evidence of scientific fraud... In its scriptures/philosophy, science describes extremely thorough and sound principles and guidelines...but in on the ground practice (by scientists, which are a part of "science"), they are often not achieved[1]. However, this distinction is not only not advertised broadly and without aversion, it is usually (in my experience) not mentioned at all, if not outright denied using persuasive rhetorical language (like, for example, when an object level instance of not achieving it is pointed to in the wild, such as in forum conversations). This may not be fraud (that requires intent I think?), but it achieves the same end: misinforming people. I absolutely agree with your database idea, and if science would like me to take them seriously (something near how seriously they take themselves) they'd also have to go much further. [1] Not unlike in religion, a competing metaphysical framework (model of reality) to science.
- 3np 2y ago
  > Not unlike in religion, a competing metaphysical framework (model of reality) to science. No. Correlation fallacy.
  - mistermann 2y ago
    Fallacy fallacy. Naive Realism fallacy.
- kerkeslager 2y ago
  This sort of comment is why I think a lot of philosophy is just communicating poorly to make yourself sound smart. In your footnote, for example, you translated your philosophy-speak into English (metaphysical framework -> model of reality). Why not just say that? Your entire comment goes into "philosophy mode" and communicates a few very simple ideas in overcomplicated language. Science and religion are pretty poorly understood as competing models of reality. Religion originates when people make up answers to other people's questions to gain social standing, and religion continues due to (among other things) anchoring bias--the bias people have toward continuing to believe what they already believe. While religion does result in those people having a model of reality, there is no attempt being made at any point to relate the model to reality. When religious people and scientists disagree, it's not because the religious person is trying to model reality differently--the religious person isn't even trying to model reality--it's because the religious person is biased in favor of their existing belief. You said: > In its scriptures/philosophy, science describes extremely thorough and sound principles and guidelines...but in on the ground practice (by scientists, which are a part of "science"), they are often not achieved[1]. This is presented as some sort of gotcha, but it's not: few scientists will claim that science is being practiced perfectly or even well. Outside of a few areas such as particle physics, we're quite aware that our ability to practice scientific ideals is hampered by funding, publication incentives, availability of test subjects in human studies, data privacy, etc. And we're aware that this means that our conclusions need to be understood as probabilities rather than 100%-confidence facts. There are certainly some people who treat scientific conclusions with religious absolute confidence, but doing that is fundamentally against scientific principles. The accusation you are leveling against science would be better targeted toward people: generally science journalists and the science-illiterate public rather than scientists themselves. The entire reproducibility crisis is scientists using science to show that our practice of science is too imperfect to result in high-confidence conclusions. Religious people jumping on the replication crisis because they think it disproves science is rich. The replication crisis isn't a disproof of science, it's an application of science. The reason we know that there's a replication crisis is because scientists asked "How confident can we be in the conclusions of existing studies?" and applied science to answer that question. If you really think science is invalid, then you can't use science to prove that. And the fact remains that any confidence in conclusions at all is more than religion has to offer, because again, religion isn't trying to model reality--the fact that religion produces a model of reality is merely an unfortunate side-effect.
hcks 2y ago
“We thought academia was not soul crushing enough so from now on you will additionally spend 10 hours a week replicating dumb papers from 1993”
- bumby 2y ago
  Snark aside, replication is a cornerstone of science. If someone doesn't want to be involved in science because they think it's soul-crushing, perhaps academia isn't the right place for them.
matthewdgreen 2y ago
There is a major issue of limited resources to replicate results, in terms of both time and funding. For example: I would assume that most important results are replicated. As a concrete example, if someone identifies a medication that (in one small trial) shows a statistically significant effect in curing some serious medical condition, then this will drive further replication attempts. On the other hand, if someone publishes a study showing that holding a pen in your mouth makes you 1% likelier to do well on the PSATs, this study will probably languish without replication for a decade — because honestly who cares? It’s basically a curiosity. I can’t help but notice that many of the headline results that characterize the “replication crisis” were small-effect-size social science experiments that fundamentally weren’t that important outside of popular science news. I’m not saying that our current allocation of resources is optimal. I am pointing out that our resources are finite and “replicate everything” is not even a remotely practical allocation of those resources.
- nostrademons 2y ago
  > I would assume that most important results are replicated. GP is pointing out that the incentive structure makes this an invalid assumption. If publications reward hype and novelty when deciding what to publish, then there is no point spending your limited resources replicating other peoples' results, they won't get published anyway. And experiments that give a null result won't be published anyway. What's left are one-off results that showed something surprising simply by chance and don't replicate...but then, we generally will never know that they don't replicate, because the replication experiment is not novel, has a low chance of being published, and hence isn't worth spending limited resources on. Basically the publication process introduces selection bias into the types of research that are even attempted, which then filters down into the conclusions we take from it. A cornerstone of the scientific method is random sampling, but as long as the results that get disseminated are chosen by a non-random process, it introduces bias.
- freestyle24147 2y ago
  > For example: I would assume that most important results are replicated. The example you provide is solely your assumption? Seems pretty odd to provide a baseless assumption as an "example".
YeGoblynQueenne 2y ago
>> I've said it before but it bears repeating - replicating published results, and adding the findings to a database, should be a standard part of PhD training programs. Wait, why should PhD students do that work? That just sounds like pushing more grunt work to the lower rung of the academic hierarchy. Nope. If you want people to do that kind of work that is important to everyone but is not directly conducive to promoting one's research career then the solution is simple: pay them.
- bumby 2y ago
  >why should PhD students do that work? I think there is some reasonable argument that replicating research is the first step to learning how to do good research on your own. In an ideal world, PhD students should probably be trying to replicate similar work anyway and applying existing approaches own pet problem. In practice, many gloss over this because they are narrowly focused on doing something "new" so it can get published.
- glial 2y ago
  PhD students are in training, and replicating a published result is a great training exercise. PhD students ARE paid. But this work won't be prioritized by their PI unless it's also a requirement of the program.
  - YeGoblynQueenne 2y ago
    >> PhD students are in training, and replicating a published result is a great training exercise. I just got my PhD last July and that sounds like boring drudgery rather than "good exercise". Good exercise is to have a student write their own paper, dig up their own references, formulate their own experimental hypotheses, run their own experiments, write up their own results etc. Re-doing someone else's possibly badly-done work is grunt work that should not be forced upon anyone. >> PhD students ARE paid. Haha. Good one XD
  - anticensor 2y ago
    Not all doctoral students are paid as part of a research position.
deleted 2y ago
[deleted]
aeternum 2y ago
It's kind of amazing that we discovered the scientific method, used it to invent the transistor and bring the information revolution. Yet we still pool scientific results using only the printing press. It's like we unlocked the tech tree but then got so caught up in chasing citations and peer review that we forgot to use the new tech we invented.
- glial 2y ago
  Yes, so-called "social technology" sometimes doesn't feel very advanced.
Gormo 2y ago
Richard Feynman was complaining about exactly this phenomenon fifty years ago: https://sites.cs.ucsb.edu/~ravenben/cargocult.html https://sites.cs.ucsb.edu/~ravenben/cargocult.html
naasking 2y ago
> This is not an evidence of scientific fraud -- rather, it is a predictable outcome of a publishing incentive scheme that rewards hype and novelty over robust science. And knowing all this, this behavy should be considered borderline scientific fraud at this point.

calibas 2y ago

The process for publishing, how a study becomes "legitimate" science, is not very scientific. Same with the process for getting funding for studies.

nick238 2y ago
My thesis committee chair bristled when I said I hated the marketing aspect of academic science and I implied that he was a very good marketer because he played the game so well. After I kept making comparison after comparison, he didn't have much to say in response. I would say he begrudgingly accepted my view, but I don't think accepted it at all, he just couldn't refute any point I made.

deleted 2y ago

[deleted]

cactusfrog 2y ago

Just write a preprint or a blog post

zug_zug 2y ago
A blog post won't be indexed on google scholar for other academics to reference though. Maybe a preprint would be? In my opinion the goal is to get a record of the information and the dataset out there.
- tokai 2y ago
  >A blog post won't be indexed on google scholar It will if its on an edu domain.
cydodon 2y ago
A preprint will still give less value to the negative result. No peer review (being a broken system or not) and not being published in a "proper" journal will make it less likely that the results will be recognised / accepted. The whole point is, that a negative result can have as much value as a positive one...
- smcin 2y ago
  Why was this comment flagged, it's reasonable?
vsuperpower2021 2y ago
The whole point of publishing studies is to be able to brag about how many impressions you have, which is good for your career. Who is going to care that your blog got views?

mlhpdx 2y ago

This seems like a tough problem to solve, but given the article states 75% of researchers are _willing_ to publish the null results at least that's something to build on. Making publishing compulsory could lead to other, worse problems given humans are involved. I realize as I'm writing this that I don't really understand what "publishing" means. It's more than just making the paper available, right? Is there a formal definition, or just a colloquial one in science?

michaelt 2y ago
> I realize as I'm writing this that I don't really understand what "publishing" means. 1. Fully gather and analyse the data, no stopping early when you realise it isn't working 2. Write the paper, read those background papers you hadn't got to yet so you can cite them, chase down references for things you know from memory. 3. Realise there's a gap in your table because you tested A, B and D at three levels each but C you only tested at the low and high level, not at the medium level. Go set up your test equipment again to fill in the blank space. 4. Run the paper by your collaborators and your boss, all of whom will feel obliged to suggest at least some improvements, which you'll make. 5. Choose a journal, apply the journal's template and style, send it in. 6. Wait for as much as several months for peer review. 7. The first peer reviewer suggests you retest with a slightly different protocol for cleaning your equipment before the test. You do so. 8. The second peer reviewer replies suggesting you test combinations of A, B, C and D, not just one at a time....
eyeundersand 2y ago
From my experience, there's no definition as such but having your study "published" implies that it went through a peer-review process featuring at least two qualified referees and an editor. The implication being that the claims from the study are valid as reference for future studies, to varying extent depending on the quality of the journal etc.
elashri 2y ago
It does not work as standard anyway. The publishing practices are different from field to field. Also journals have different policies and practices. So it is hard to get a real representative definition other than making paper available. In which case arxiv will be a publishing mechanism that does not provide editor, peer-reviewed and does not cost money.

parpfish 2y ago

Publishing null results would be great, but I’m worried about how that could also be gamed. How do you distinguish a ‘real’ null result from one done in a sloppy study? Would people run shoddy experiments to get null results to undermine their rivals? Could somebody pump out dozens of null publications to pad their CV and screw up h-indexes?

vharuck 2y ago
Currently, the system can be gamed exactly as you say by publishing sloppy studies that falsely find a "real" effect. But editors and readers of the article will look at and judge the methodology section. Sloppy experiments risk not being printed or cited. >Would people run shoddy experiments to get null results to undermine their rivals? In this case, the rival would be very much inclined to recreate the "null" experiment. >Could somebody pump out dozens of null publications to pad their CV and screw up h-indexes? Possibly, but would null publications be cited as often? Also, who's going to keep funding a researcher that mostly publishes null results?[0] [0] Besides agenda-driven "think tanks". Which is worrying itself.
- parpfish 2y ago
  > Possibly, but would null publications be cited as often? Don’t underestimate an academics ability to cite ALL of their previous publications each time they publish
tefkah 2y ago
I run a journal where we publish both “real” null results and experiments were something practical went wrong, so i have a few thoughts: 1. Ideally peer review would catch this. A badly setup study should be critiqued in peer review. Forcing scientists to first publish their methods before doing the experiment also helps, as it validates the experimental setup before hand. I also think it’s worth publishing studies where a null result was reached due to some error in experimental setup or other factors, as long as it’s presented as such and reflected upon. This can still be valuable information for future experiments. Offering scientists social capital for that (an “official” publication, citations) might also incentivize scientists to publish the results as is, rather than making it appear as a “true” null result, or even as a non-null one (eg through p hacking). 2. While obviously possible, given the amount of effort scientists have to go through to raise funding for an experiment nowadays, i find it highly unlikely that people would go through this effort. 3. This is already possible and a problem. This is a problem of academic misconduct, has very little to do with null results. The current publishing system is of course already set up to be gamed, so I understand your worries. But null results should be published, as they are just science. Even if someone were to “game” the system by publishing a ton of null results, those publications should be held to the same level of scrutiny as any other publication. If someone is extremely prolific in replicating existing studies and comes up with a ton of null results, that should be lauded and those papers should be published, no? I do believe the entire idea of a researchers output only being recognized by being allowed to be published in a journal is terrible and should be abolished, but baby steps I guess.

proof_by_vibes 2y ago

I recall reading someone who proposed the need for what they dubbed "meta-science," and I think it's clear that this concept is becoming more needed as time goes on. Our publishing process, and the incentives therein, are obviously faulty and we are aware of it. We can do the math: I believe it's time we do away with playing speculative games with science.

shishy 2y ago
"Science of Science" is a good book / overview of this field!

shishy 2y ago

Research should require pre-registration like clinical trials so others have visibility into failed outcomes

elashri 2y ago
Who would track this registration and would it require approvals now?. Then what if you changed your current research because you have personal reasons, change of plans, didn't see it fitting....etc. How would you handle these situations? And why are you introducing MITMs.
- _flux 2y ago
  In basic level, maybe some publications that print these papers would take the registrations, as a precondition to publishing them? It's not like it couldn't be gamed, but maybe it would incentivize people to also publish null results.
  - elashri 2y ago
    >incentivize people to also publish null results This will hardly achieve this goal as you basically make things harder, and now you are introducing more overhead. The main reason why people don't like publishing null results is that it hurts them in funding applications. The current system works with mentality, we shouldn't fund someone who don't get positive results. It is better to allocate this somewhere else. Most of the problems with research can be tracked down to funding issues and practices. But these are political issues, so people try to argue about other things because it is easy.
    - shishy 2y ago
      All good points- I was imagining that from a funder's perspective, knowing null results is actually important to guaranteeing future positive results / being strategic about what is worth funding in (versus what is destined to fail because others failed and they just didn't know). Not sure how it would work in practice; might be bumpy but certainly seems worthwhile / not impossible.

hcks 2y ago

The introductory example is quite illuminating. No theory behind, just a random hypothesis tested (by comparing the preferences of 10 fishes from populations separated from as little as 50 meters, what effect size did the authors expect?), with claims of generalisation (climate change ok bad?? Or climate change bad bad??)

jjmarr 2y ago

Would recommend people interested in this start following "metascience". https://en.wikipedia.org/wiki/Metascience https://en.wikipedia.org/wiki/Metascience And read "Why Most Published Research Findings Are False". https://en.wikipedia.org/wiki/Why_Most_Published_Research_Findings_Are_False https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...

sunshinesnacks 2y ago
There’s a bit of irony in the second reference, as the author ended up with some controversial work related to COVID-19. He co-authored a study that was widely cited to downplay severity of the pandemic, but was also heavily criticized for poor methodology (and later I think firmly found to be very wrong). He also published a paper with personal attacks of a grad student that had disagreed with him, which is probably not in the spirit of encouraging constructive science.
- roland_nilsson 2y ago
  So he proved his point! :)

tefkah 2y ago

You could publish it in the Journal of Trial and Error (https://journal.trialanderror.org https://journal.trialanderror.org), which I created with a number of colleagues a couple years ago! Our editor-in-chief was interviewed for this related Nature article a couple months ago (https://www.nature.com/articles/d41586-024-01389-7 https://www.nature.com/articles/d41586-024-01389-7). While it’s easy pickings, it’s still always worth pointing out the hypocrisy of Nature publishing pieces like this, given that they are key drivers of this phenomenon by rarely publishing null results in their mainline journals. They are have extremely little incentive to change anything about the way scientific publishing works, as they are currently profiting the most from the existing structures, so them publishing something like this always leaves a bit of a sour taste.

MostlyStable 2y ago
Providing _places_ to publish the result is only part of the problem. The other part is incentivizing scientists to do so. And similarly, Nature itself is responding to incentives. The core problem is that scientists themselves, the individuals, do not mostly display much interest (in a revealed preference kind of way) for null results. If scientists were interested and wanted to read articles about null results, then either journals like Nature would do so, or the numerous examples of journals like yours that have come and gone over the years would have been more succesful and widespread. Because of this revealed lack of interest, high tier journals don't tend to take of them (correctly responding to the lack of demand), and journals like your that specifically target these kinds of articles A) struggle to succeed and B) remain relatively "low impact", which means that the professional rewards to publishing in them are not very high, which means that the return on effort of publishing such a work is lower. Don't get me wrong, the scientific community could do a lot more to combat this issue, but the core problem is that right now, the "market" is just following the incentives, and the incentives show that, despite what the stream of articles like this one over the past few decades is that most scientists don't seem to actually have an interest in reading null-result papers.
- dleeftink 2y ago
  What if to each publication of a non-null result, academics are given the opportunity to publish their nulls as well, if only as a appendix or better, a counterpublication to their main conclusions? I don't buy the argument that papers need be of max-n length, now that documents and journals can be easily stored and distributed. I would love something like Living Papers [0][1] to take off, where the null an non-non results could be compared interactively on similar footing. [0]: https://github.com/uwdata/living-papers https://github.com/uwdata/living-papers [1]: https://idl.uw.edu/living-papers-template/ https://idl.uw.edu/living-papers-template/

EricE 2y ago

Remember all the cries of "the science is settled"! Yeah, that's not science - it's the exact opposite of science. This is the perfect example of why reasoned skepticism is more necessary than ever. Blind trust in any institution is a recipe for disaster.

__MatrixMan__ 2y ago

I wish we would disentangle publication from endorsement. Making the bits available and saying that they're useful are different things. Your null result could contain data which is relevant to some other inquiry. All results should be published, some should be celebrated.

Brian_K_White 2y ago
The example null correlation sure sounds as significant as any correlation.
tmalsburg2 2y ago
Publication servers like arXiv + overlay journals. I‘d love that.
KeplerBoy 2y ago
This goes both ways. Some people publish fantastic papers without data nor code. Sometimes annoying, other times a complete waste of everyone's time.
- __MatrixMan__ 2y ago
  Does it? I mean there's a lot of trash on the internet that just doesn't get looked at. If you waste your time reading it, that's on you.
  - __MatrixMan__ 2y ago
    I just reread this and realized how much of a jerk I'm coming off as (too late to delete it). My point is just that if you're reading something because it was published then publication and endorsement must not be sufficiently decoupled. In a world where they are decoupled, you'd only end up reading it because somebody you trust endorsed it (no reason for that to coincide with the publisher) or because you went searching for it despite its lack of endorsement--in which case you probably know what you're doing.

yzydserd 2y ago

It’s a shame to see no mention of https://opentrials.net/ https://opentrials.net/ in the article.

SeanLuke 2y ago

I think the fundamental problem is that for every claim yielding a positive result there are many more, perhaps infinitely more, related claims yielding negative results. Positive result claim: the sun comes up in the morning. Negative result claims: the sun moves sideways in the morning. The sun was always there. The sun peeks up in the morning and immediately goes back down. And so on. Positive result claim: aspirin is an effective pain reliever. Negative result claims: eating sawdust is an effective pain reliever. Snorting water is an effective pain reliever. Crystal Healing is an effective pain reliever. Etc. Because there are so many negative results, it's trivial to construct an experiment which produces one. So why should that be published? Negative results should be published when people in the community are asking that question, or have a wrong belief in the answer (hence the replication crisis). But if nobody cares about the question, it's hard to argue for why a given negative result would be preferred over any other negative result for purposes of publication.

dfgtyu65r 2y ago
None of these are negative results in the sense of being a 'null' hypothesis? In the language of hypothesis testing you have your null and alternative hypotheses. So for alternative hypothesis that the sun comes up in the morning, the null hypothesis would simply be that the sun does not come up in the morning. Each of the negative results, reads to me like a separate 'alternative' hypothesis.
- SeanLuke 2y ago
  Sure they are. So let's say I claim that the sun goes in a circle in the sky in the morning. The null hypothesis is that it doesn't do that. Perform experiment. Null hypothesis wins. Write up paper! This is a negative result. The point is that for every result where the alternative hypothesis wins, there are a massive, if not infinite, number of results where the null hypothesis will win. Are these publishable?
  - nick238 2y ago
    The idea is that some null hypotheses being true is actually interesting because it challenges an assumed belief. From the first paragraph of the article, the immediate feedback from the postdoc's supervisor was 'you did it wrong [because everyone knows that fish do like warmer water]'. > It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so.

mettamage 2y ago

Is it an idea to publish null results in appendix when a sexy result will be published? Kinda like a Thomas Edison thing. How many ways are there to not make a lightbulb, included with the ways that do make it possible to create one

bowsamic 2y ago

Researchers are expected to publish papers at such a frequency today that spending time writing a paper for a null result would be considered a bad career move

bandrami 2y ago

There is a psychology journal specifically for null hypothesis results: https://www.jasnh.com/ https://www.jasnh.com/

ketanmaheshwari 2y ago

Relevant, I run a workshop for negative results: https://error-workshop.org/ https://error-workshop.org/

fharding 2y ago

In cryptology there’s something called CFail, which is a bit like this. https://www.cfail.org/call-for-papers https://www.cfail.org/call-for-papers

deleted 2y ago

[deleted]

gtmitchell 2y ago

As someone whose early scientific career was destroyed by null results, no. No one will publish your negative results. Unless you win the lottery and stumble across a once-in-a-generation negative result (e.g. the Michelson–Morley experiment), any time you spend working on research that yields negative results is essentially wasted. This article completely glosses over the fact that to publish a typical negative result, you need to have progressed your scientific career to the point where you are able to do so. To get there, you need piles of publications, and since publishing positive results is vastly easier than publishing negative ones, everyone is incentivized to not waste time on the negative ones. You either publish or you perish, after all. Simply put, within the current framework of how people actually become scientists and do research, there is no way to solve the 'file drawer' problem. You might see an occasional graduate student find something unusual enough to publish, or an already-tenured professor with enough freedom to spend the time submitting their manuscript to 20 different journals, but the vast majority of scientists are going to drop any research avenue that doesn't immediately yield positive results.

gzer0 2y ago

The main issue with publishing research is the cost. Some time ago, I worked in a research lab studying Lupus. Our results were negative, and my initial inclination was not to publish them. However, my Principal Investigator (PI) emphasized that all results, whether positive or negative, should be published. Fortunately, we had the funds to do so. At that time, publishing in a reputable journal cost $2,300. Not everyone is so fortunate. This lesson has stuck with me, as I have seen or heard from different labs where, unfortunately, they couldn't afford to publish their findings.

nick238 2y ago

What's the last null result Nature, Science, and Cell published?

SubiculumCode 2y ago

Perhaps my most prominent paper was a null result. If the question is important enough, the work to test it non-trivial, it will find an audience and a likely a reputable journal. What is the value in reporting a null result other than reducing file drawer effects? Well one, even though the null does not tell us whether the effect is absent, it does help suggest bounds on how large an effect could be if it did exist. In my case, a particularly large effects were observed in cross-sectional comparisons (different people at different ages), but our research showed that longitudinal changes within individuals were generally negligible, suggesting systematic bias in the cross-sectional sampling.

ok123456 2y ago

Why not self-publish it on Arvix?

bachmeier 2y ago

The problem is that peer review is primarily focused on results. Peer review should be done up to but not including the results. Provide motivation, explain your methodology, explain how it will resolve issues in the literature, but don't say anything about your results. Papers should be conditionally accepted, subject to confirmation that the results you report are the results of the proposal that went through peer review.

joemazerino 2y ago

Crowdstrike will.

analogwzrd 2y ago

Some others have mentioned this in their comments and I agree that once you succeed in getting a non-null result, publishing the null results (all the things you tried that didn't work) could be included as appendices or something. Also, just because you get a null result doesn't mean that nothing was learned, that something new (and unexpected) wasn't stumbled on, or that some innovation didn't happen. There are tiers of publications and journals. Even if you get a null result and you're not going to get it accepted in Nature, it's very possible that you can get a conference paper (sometimes peer reviewed) out of something that was learned.

lokimedes 2y ago

We published a load of null results in particle physics. Simply go to arxiv.org and look for papers beginnig with “search for…” that would be a null result. Well technically a sigma<3 result.

diffxx 2y ago

True story: I wrote my ph.d. thesis on a special case of a general problem. After about one year of work, I realized that the approach would never work for the general problem for intractable reasons. But I also really wanted to finish my ph.d. within 5 years, so I spent the next two years refining the work enough to be able to write a dissertation on it and ignored the fact that it would never really work for what it was intended. I did do some interesting work and learned a lot, but I couldn't really bring myself to try and publish the results (beyond my thesis) because I very clearly had not made an advance in the field. Of course, I do think it would have been useful to publish why I thought that essentially the entire field of inquiry was a dead end, but that would not have made me very popular with my collaborators or others in the field and it wouldn't likely have ingratiated me with anyone else.

okwhateverdude 2y ago
Great, so the economic {dis|in}centives are keeping a bunch of people fed but doing useless study of a dead end and even kept you from pointing out the emperor has no clothes (assuming your epiphany is valid). Is social conformity holding back science?

jimmar 2y ago

Not all null results are created equal. To get your null results published, the null result must shed light on some phenomena. And of course, the study must be sound. E.g., everybody takes for granted that X causes Y, but in a well-designed study, X did not in fact cause Y, which reveals an error in what we assumed was true.

jenders 2y ago

Crowdstrike will

sandraruby512 2y ago

[dead]

roland_nilsson 2y ago

I think this problem is at least in part due to the hypothesis testing concept itself. Classical hypothesis testing is asymmetric: there is a "null" hypothesis, which is typically the uninteresting/useless case, and an "alternative" hypothesis, which is the one you would like to be true. Critically, you cannot determine if the data _supports_ the null hypothesis, only if the data _rejects_ it (and supports the alternative). A so-called "null result" occurs when data is not sufficient to reject the null hypothesis; then you can't tell if you actually have a useful finding (for example, that there is no major difference between species A and B) or a failed experiment (data was so bad/noisy that we cannot conclude anything). And so you end up with the unfortunate situation where you either succeed in proving your favorite hypothesis and get your degree / promotion / tenure, or you have nothing. This happens because hypothesis testing conflates effect size (how big is the difference between A and B) with uncertainty about that effect size (significance/reproducibility). Confidence intervals are more useful IMHO, as they help untangle these two aspects, for example showing that the difference between A and B is small _and_ reproducible. Bayesian analysis is also a major improvement, as it allows examining both the "null" and "alternative" hypotheses on equal terms, as well as reasoning about our prior beliefs / biases. Unfortunately many areas of science are still stuck with statistical methods from the early 1900's.

nikau 2y ago

Crowdstrike will