18 ms·
New academic journal only publishes 'unsurprising' research rejected by others
- eloff 6y agoGood, science needs more of this. A negative result is a result, and if you don't publish it, then you end up skewing the distribution of results. Which can lead to people (and meta studies) drawing incorrect conclusions.
- DrBazza 6y agoBen Goldacre talks about this in Bad Science. Well worth a read.
- onurcel 6y agoI like the idea but I don't really get why they accept "statistically insignificant" results. What I expect from a scientific paper is to prove (even emprically) something. For example, if a paper claims something like "we show that using method X instead of Y doesn't improve the results", it won't get published on most journals... except this one, which is awesome, but the paper still has to prove that claim.
- acchow 6y agoIf the only goal is p 0.05 and there's many groups around the world trying the same experiment, we should hope that they all publish their non-results otherwise by chance someone is going to have a significant result just by luck.
- bboy13 6y agoWhen you open the field to reward research itself rather than p-hacking, everyone benefits. It's like a flea market. Bargain bin discounts, junk to many, valuable to some, but it shifts the economics of the whole system and lets specialty niche researchers not have to reinvent the wheel. Also, knowing that a method or study was statistically insignificant is statistically significant for future study design. Thanks!
- walty 6y agoIf you only publish the positive results, you miss the context of the unsurprising results. For example, in https://xkcd.com/882/ https://xkcd.com/882/ , if the results from all the experiments in the comic were published, the public would know that the "Green Jellybeans cause acne" result is probably a fluke.
- progval 6y agoDoesn't "statistically insignificant" mean papers which conclude something like "we find no correlation between X and Y"?
- kelnos 6y agoI think information like that is perhaps useful, at least to avoid wasting time repeating the same research, but publishing that does not prove that there's no correlation between X and Y, only that the things tested didn't show one.
- BenoitEssiambre 6y agoNo! It means the data is too noisy to make any conclusion.
- ellis-bell 6y agonot necessarily. it could. but if x and y are truly uncorrelated variables that are observed with unlimited precision, then you still would not reject the null that they are uncorelated
- BenoitEssiambre 6y ago"Truly uncorrelated" is a problematic concept. Most of the time, any two things within each other's light cones, have at least tiny direct or indirect effects on each other. These effects are usually not that tiny in experimental settings because of imperfect experimental tools and procedures.
- gus_massa 6y agoLike the speed of sound and the height of the table where the experiment is. Note that the pressure of the air changes with height. For a few inches the change is negligible and very difficult (impossible?) to measure, but the effect must be out there. I had once a problem with the temperature of the room, you usually ignore it, but it has a bigger effect like a 5% variation.
- dguest 6y agoYou can't ever prove that switching to X doesn't change the result. All you can really hope for is putting smaller limits on the amount by which something could be improved. For one dumb example: if you wanted to see if rats can fly, you might observe 100 rats and see that none of them fly away. But this doesn't prove anything: in reality even if 3% of rats could fly you'd get this result about 5% of the time. You've really shown with 95% confidence is that less than 3% of rats fly.
- collyw 6y agoIt should stop other groups wasting their time on the same research, no?
- timwaagh 6y agothis is old news from last year, also there seems to be only one issue with one article, which means this project is stillborn. https://ir.canterbury.ac.nz/handle/10092/14932/browse?type=dateissued https://ir.canterbury.ac.nz/handle/10092/14932/browse?type=d...
- owenshen24 6y agoOof, that sucks to hear. Thanks for letting us know it's not been updated.
- whatever1 6y agoWe also need a journal to publish methods that failed. I did so much work during my PhD that was dead end and is not documented.
- xncl 6y agoChemistry is so bad for this. That and how everything is behind a paywall that only multinationals and universities have access too.
- auraham 6y agoI support this. Although a journal may not accept that kind of work, you can still publish it in your blog.
- hprotagonist 6y agothere was a journal of negative results in biomedicine, but i don't think they've published since 2017.
- DonCopal 6y agohttps://www.journalnetwork.org/journals/international-journal-of-negative-and-null-results https://www.journalnetwork.org/journals/international-journa...
- MattGaiser 6y agoIs it normal for journals to charge a fee for publishing?
- 60secz 6y agoNull Hypothesis quarterly needs to be on every coffee table.
- nippoo 6y agoRelated (but more general): the Journal of Articles in Support of the Null Hypothesis, https://www.jasnh.com https://www.jasnh.com
- michaelmior 6y agoThis reminds me of the Failed Aspirations in Database Systems workshop at VLDB back in 2017. I don't think it turned out to be quite what I was personally hoping it to be, but the idea is great. https://fads.ws/ https://fads.ws/
- kraetzin 6y agoHalf of my PhD thesis is considered "unpublishable" because, after doing the work, my supervisors felt it's actually "unsurprising" that it didn't work out. We took methods that had been exploited to improve on previous results for over a decade to their logical extreme, and found that this method no longer leads to improvements. After doing the work it seems obvious. A paper on the subject would almost be considered uninteresting, and a high ranking journal would ignore it (which is why it's considered "unpublishable"). However, nobody has published this information, and it would help others to not make the same mistake. I wonder how many times similar "mistakes" have been made by PhD students across disciplines.
- owenshen24 6y agoThat sucks to hear. Null results are important, as you say, if only to dissuade others from doing the same. See also the "file-drawer problem" (https://en.wikipedia.org/wiki/Publication_bias https://en.wikipedia.org/wiki/Publication_bias). Also, with regards to the incentives in the field and the lack of null results, there's always Ioannidis's classic work (https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo...).
- kraetzin 6y agoThose are some good reads, and have absolutely been my experience. It's depressing how many publications that I've come across don't provide the whole story, and are probably false. I've found that looking at what a paper doesn't report can be far more important that what they claim.
- mycall 6y agoI wonder how many novel techniques could come out of these types of reports if they were actually analyzed by ML or NLP.
- owenshen24 6y ago
- klysm 6y agoThe problem is researchers aren't going to want/be able to spend the time to properly document negative/unsurprising results. The financial incentives in place don't support it despite its incredible value.
- e2021 6y agoBut isn't the reason they don't want to write it up because they know they won't get any credit for it? That is exactly what this is trying to remedy
- klysm 6y agoI'm referring to grants and how people get funding.
- throwaway590007 6y agoIt takes time to write a paper, even for negative results, and moreover journals charge a hefty fee.
- gus_massa 6y agoThere are many method to evaluate the work. Sometimes it is just the count the number of published papers, or that dividing by the number of authors, or some weight if you are the first or the last author. Number of citations, h-index, ... About the journal, there is the impact factor and many somewhat arbitrary ranks... A paper in a totally obscure journal that is not cited by other papers, has the same weight than a blog post.
- throwawayiionqz 6y agoThe challenge of modern academia in certain fields is not to publish but to be read by others. Everyone is so busy publishing that very few papers get decent readership (retweets and citations happen mostly without reading the substance). I would rather have an upper limit on the number of papers one can publish in a year than more avenues to publish unsubstantial findings.
- catsarebetter 6y agoI can see this journal being 10x-20x the size of normal journals
- refurb 6y agoA very good point. My experience was that maybe 5% of the work I did made it into a publication. If I wrote up everything that didn't work, my PhD would have been 10 years instead of 5.
- catsarebetter 6y agoI sympathize, your work that was never published was probably of equal caliber to your published work, and just as valid.
- ay 6y ago“We kind of want to fill the void and publish results that are the opposite of that —unsurprising, weaker, statistically insignificant, not conclusive and so on.” These are at least 5 Distinct categories, hope they make a distinction...
- choxi 6y agoWhy don’t research groups just publish to their own websites or directories like arxiv? What’s the role of an academic journal in 2020? Honest question, I’d love to see more blogging from hard science academics but I’m wondering if there’s a reason why that’s challenging or if it’s just academic culture. We should have a Substack/OnlyFans for scientists.
- icebergwarrior 6y agoWhile we're at it, why not make public whoever audited or referred the paper? Why does that need to be secret?
- apsec112 6y agoPeople might retaliate against a colleague who gave a bad review.
- zapzupnz 6y agoIndeed. I've seen some mightily acerbic rebuttals to other researchers' articles published as articles; I dread to think what a rebuttal to a review might look like. I'm sure someone will say "well, maybe the acerbicness is the problem"; perhaps so, but I welcome the rigorous honesty with which some academics willingly write.
- bluenose69 6y agoTrue, and just imagine if the reviewer were a PhD candidate who would be seeking a job in a few years.
- sinkasapa 6y agoIt might be that the person whoes paper you're reviewing will some day be able to influence whether you'll be hired.
- smabie 6y agoWe should have less transparency, not more. Right now the peer review process is single-blind, but it really should be double-blind. Even if someone doesn't have a PhD or work at a research institution, they should be able to publish good science. Right now, that just isn't possible. And the opposite problem is also true: if you're a big shot in your field, you'll be able to find at least one journal that will publish whatever crap you submitted, regardless of the quality.
- austinl 6y agoI'd also love to see a journal that agreed to publish work before any results are known. Researchers would submit hypotheses and methodology for review, and the journal would publish the results after the experiments were conducted, regardless of their outcome. It would still incentivize interesting hypotheses, but wouldn't lead to results-biased publications.
- bonoboTP 6y agoYou'd do the work, then sign up, wait a bit (while working on your next project) then you submit the results. This is what happens with many grants anyway. You propose to research the stuff that you already understand, you get funded and can finally work on new stuff you don't understand well enough to get funded for.
- laGrenouille 6y agoThis is a great idea and already being done (somewhat) in some of fields of psychology through pre-registration [0, 1, 2] [0] https://en.wikipedia.org/wiki/Preregistration https://en.wikipedia.org/wiki/Preregistration [1] https://www.psychologicalscience.org/publications/psychological_science/preregistration https://www.psychologicalscience.org/publications/psychologi... [2] https://www.psychologicalscience.org/observer/preregistration-becoming-the-norm-in-psychological-science https://www.psychologicalscience.org/observer/preregistratio...
- occamschainsaw 6y agoThere is a workshop[1] at NeurIPS this year (2020) for experimenting with this model. I hope it is adopted more widely, especially in the ML community to disincentivize the +1-2% performance increase in SOTA papers. [1] https://nips.cc/Conferences/2020/Schedule?showEvent=16158 https://nips.cc/Conferences/2020/Schedule?showEvent=16158
- chunkyks 6y agoI just gave a briefing today where I work, that had two conclusion slides: On the first, "None of this matters", and on the second "I've no idea". One the one hand, that's pretty nihilistic. On the other hand, it's cool that I was able to explore it and come to the conclusion that everything is fine and some aspects of the past are unknowable.
- anigbrowl 6y agoLove the idea but wish there was a better writeup. Transcripts of radio interviews seem stilted and miss the tone context that can indicate whether something was a joke or a casual aside. I think the journal itself could be quite valuable and hope it succeeds. Perhaps this model can be generalized to other fields too.
- plaidfuji 6y agoThere's a name for that: it's called a "low-level journal"
- rossdavidh 6y agoSo, it's a year old, and only one article? Now that is surprising.
- jarmybarmy 6y agoAn unsurprising result for sure.
- nrev 6y agoI initially read this headline as an Onion-article style joke. That said, I don’t disagree with some of the ideas set forth here. I mean, I think this same sort of thing is the reason I regularly find myself reading journal articles and thinking that the title is so misleading/essentially just scholarly clickbait titles. From my perspective, however, I would be really interested to see a journal do something like this but instead with ambivalence about the use of particular stat models. (i.e. I definitely feel that there is still a religiosity in how some reviewers/journals/even disciplines have singular loyalty to p-values, but this may merely be my own perception or anecdotal experience).
- gverrilla 6y ago20th century+ science = pyramid scheme
- deleted 6y ago[deleted]
- quelltext 6y ago> Nick and Andrew did a similar experiment, they found no effects whatsoever. > And so they send this to a journal and the response they got was, well, you know, you probably wouldn't expect an effect because this is not fairly novel information to the students. So we are not going to publish this because it's not really surprising. I don't quite understand that example. If every one else in their papers indeed suggested that telling students of the cost should solve the issue, and everyone believed them, then how can it be claimed that the experiment results are not surprising. The whole point was to disprove what apparently was blindly accepted because of common sense; being then told that in fact the opposite is common sense seems like a slap in the face. I didn't read the paper, so maybe I'm missing something, but I'm not sure it's actually a good example for that journal. Had they done experiment in isolation with no preexisting notion or consensus of what might solve the graduation issue, maybe.
- ineedasername 6y agoBuilding off the example in the article about college students, I can attest to this: For years it's been required for colleges to be upfront and disclose the total costs, and financial literacy initiative at orientations & introductory courses have pushed the issue explaining the additional costs of taking longer than 4 years to graduate. The results have been indiscernible from prior to this.
- kkylin 6y agoReminds me of this: https://en.wikipedia.org/wiki/Rejecta_Mathematica https://en.wikipedia.org/wiki/Rejecta_Mathematica which sadly did not last.
- User23 6y agoPerhaps they can publish Dijkstra's little gem[1] on the Pythagorean theorem? He explicitly asks what journal would publish such a triviality. However it's a really neat little bit of reasoning and I quite enjoyed it. [1] https://www.cs.utexas.edu/users/EWD/transcriptions/EWD09xx/EWD975.html https://www.cs.utexas.edu/users/EWD/transcriptions/EWD09xx/E...
- a9h74j 6y agoI haven't seen anyone mentioning this scary thing, true when I was at a top-10 university: No significant-enough publication, no Ph.D., even after years of all-in work and an accepted dissertation are done. Things might have changed, but this is up there with foreign language requirement in terms of fine print nobody reads going in.
- nurettin 6y agoWhy isn't "verbal intervention for university students didn't make them graduate any faster at all" an interesting result?
- hoseja 6y agoI wonder what is the actual, hidden, incentive for academia to stay this dysfunctional. It can't just be Hanlon's razor, right?
- quixoticelixer- 6y agoHuh, this is my university
- mjfl 6y agoThe problem with null results and boring results is that there are too many of them.
- punnerud 6y agoIf every SW-developera had the same mindset they would have build a lot from scratch every time
- throwaway590007 6y agoThat journal already exists lol, it's called PlosONE.
- anticensor 6y agoPLOS accepts positive results as well.
- throwaway4747l 6y agoTo the detriment of the authors who should know better than publish there if they have something interesting
- nxpnsv 6y agoThere is only one article.
- Vinnl 6y agoThe root problem is the reward system in academia. Even today, when the relevant players are discussing how to improve it, the question is always phrased as needing to find better ways to reward excellence. If you want to reward excellence, you're by definition looking for the exceptional, the surprising. It might sound good, and surprising discoveries are necessary, but they're not the only thing that's good for science. We need ways to recognise more than just exciting results; researchers should also be rewarded for robust contributions to the body of human knowledge. (This is also why Plaudit.pub, a project I volunteer for, allows researchers to explicitly recognise robust or exciting work as separate qualities.)
- remus 6y ago> If you want to reward excellence, you're by definition looking for the exceptional, the surprising. It might sound good, and surprising discoveries are necessary, but they're not the only thing that's good for science. I think this depends a lot on how you define 'excellence'. I agree that currently 'excellence' == novel and interesting findings, but we could interpret it to mean 'excellent science' in the broader sense, where showing that another study fails to replicate, or getting a negative result, are equally important parts of the scientific process.
- cjfd 6y agoIt is an improvement over the current situation but the question remains what these journals are good for besides gobbling up money that should go to research instead. There are preprint servers. If they can create a comment system where scientists can leave comments what are journals good for anymore?
- MaxBarraclough 6y agoAs a non-academic: this journal will presumably end up with a low impact factor, right? So it might help with the problem of negative results not getting published at all, but it doesn't address the 'academic penalty' for getting negative results, right? Seems to me that preregistration [0] is the real answer here. If all journals went with preregistration, there'd be no need for a journal like this, right? [0] https://plos.org/open-science/preregistration/ https://plos.org/open-science/preregistration/
- realradicalwash 6y agothis is nice, and a clear step forward. but more is needed to fix a semi-broken system. because publication bias is one thing, but there is also something like a _reception bias_. there are already null result journals and workshops. it is increasingly possible to publish negative results / unsurprising results. still, imv those results are less likely get cited. because negative results are often messy that can be hard to understand. and people don't want to pick up a paper and then not understand. so it's no surprise that positive results, certainly the way they are framed, often neatly come together. the 'stories' they tell you are often easy to grasp mono-causal stories. it's almost like you read them and you feel good about yourself, because you feel like you suddenly understand a difficult problem. and then, which paper will you cite in your own work? the paper you understood, that has that catchy title? or the one that you struggled through, the one that painted a picture of a far more complex, messy reality? this kind of reception bias will be very hard to fix. it takes more critical editors, reviewers, and readers to fix this.
- Udik 6y agoClimate change science is in dire need of a journal like this.
- williesleg 6y agoNice! Get some Hydroxychloroquine research done, it's cheap and effective according to the doctors! But not big pharma!
- barrenko 6y agoI still shudder when I think of the fact Tim Ferris mentioned, that trouble with thesis advisors is among leading cause of suicide in young men. Had similar trouble myself.
- asah 6y agoIs there a journal that publishes "beautiful experiments" ?
- markstos 6y agoFor those involved in QA, the benefits of boring work are unsurprising.