10 ms·
What happens when you try to publish a failure to replicate in 2015/2016
- arcticfox 10y agoSo broken. I'm not involved in academia, so the most I can contribute are upvotes here and there, and giving respect to those who push against the current.
- Waterluvian 10y agoThere needs to be a journal called "Failures" that focuses exclusively on failed experiments and unreproducible results. I'm far more interested in learning why something may have gone wrong. Maybe it's just me but I feel it would surprise many people with its popularity.
- apathy 10y agoIt's just you.
- jessriedel 10y agoSee bandrami's comment https://news.ycombinator.com/item?id=11999413 https://news.ycombinator.com/item?id=11999413 as well as this: > The biotech company Amgen Inc. and prominent biochemist Bruce Alberts have created a new online journal that aims to lift the curtain on often hidden results in biomedicine: failed efforts to confirm other groups’ published papers. Amgen is seeding the publication with reports on its own futile attempts to replicate three studies in diabetes and neurodegenerative disease and hopes other companies will follow suit. > The contradictory results—along with successful confirmations—will be published by F1000Research, an open-access, online-only publisher. Its new “Preclinical Reproducibility and Robustness channel,” launched today, will allow both companies and academic scientists to share their replications so that others will be less likely to waste time following up on flawed findings, says Sasha Kamb, senior vice president for research at Amgen in Thousand Oaks, California. http://www.sciencemag.org/news/2016/02/if-you-fail-reproduce-another-scientist-s-results-journal-wants-know http://www.sciencemag.org/news/2016/02/if-you-fail-reproduce...
- deleted 10y ago[deleted]
- mattkrause 10y agoI'm not sure this would work as a journal. Who would read this? I skim Nature, Science, and some journals in my particular subfield every week/month because I am sure I will find something relevant to my research or generally interesting. With the exception of some hilarious or spectacular ones, I don't think Failures would be so engaging, so the only way I would find an article in Failures is if I specifically set out to search for it. I might do that (though searching by technique is hard), but even if I found something, read it, and decided it was helpful, what would I do with it? For better or worse, citations are the currency of academia and academic publishing. Where would I cite a paper that specifically warned me away from a possible project? "Dear Nature, I was going to study XZY via ABC, but [1] indicates that's a dead end." So, with no regular readers and no citations, Failures becomes more like a database than a journal. There currently aren't any good incentives to submit things to databases, so Failures becomes a....failure.
- startling 10y ago"There has been some promising research with alternative technique X, but that has shown to be unreliable. [1] Because of this and [other factors], we have decided on technique Y."
- sevenless 10y agoAcademia is broken. If we'd had the internet before we got around to creating academic science, our system of disseminating knowledge would look nothing like the journal/peer review system and probably a lot more like Github.
- danbmil99 10y agoSeems to me this issue is getting to the point where it could become an existential threat to the credibility of science in general. Note how climate-change deniers have recently used these sorts of arguments to challenge the consensus - is it really so far-fetched to argue that perhaps climate scientists are as biased as researchers in areas such as medicine and linguistics? The pay-wall, blind peer review process seems broken beyond repair. There needs to be a better, robust method to publish every relevant study that is not utter crank, and get some sort of crowd-sourced consensus from researchers with credible reputations.
- fchopin 10y ago> and get some sort of crowd-sourced consensus from researchers with credible reputations. Would be nice, but where would the money come from? Even if you live in a country with high taxes that fund ridiculous projects, they are still often accountable to the public. Would the public want to fund new research or try to replicate or disprove other studies? The only way to make it work would be to build-in a certain percentage of funding for such a purpose in reaction to some gross oversight, so you could get the public behind it, or perhaps get some wealthy donors behind the cause, but I think you'll have a difficult time. The better option might be having research students be required to replicate at least one study that hasn't been replicated in addition to doing their own research. That way you get some free labor!
- apathy 10y agoDepends. I would like to think that "the public" would notice all those silly cancer patients showing up to clinic (don't they know that cancer is cured twice a week at every major hype factory, err, university?) but, no. Pie in the sky it is, then! If only a field existed that attempted to quantify the expected variation from a given size and design of experiment. That field would be of great use to major nation states. So much so, you might call it statistics. The practitioners would probably be big party poopers, always referring to weird concepts like "regression to the mean", "power", "effect size" and other unexciting yet critical details. I imagine there would be funny videos on YouTube about these people, and "real" researchers would happily ignore them while they jerk each other off in P01 study sections. Anyways, let's keep science the way it is. There's no way the process or its results could possibly be improved!
- apathy 10y agoPut it on arXiv or f1000 for fucks sake. Who actually believes psychology papers anyways? The vast majority are fishing expeditions as best i can tell. When the field starts enforcing minimal standards (as expected for, say, clinical trials, or even genetics studies nowadays) maybe someone will give a shit. Until then people like this guy who actually seek the truth will be ostracized.
- danso 10y agoThe field is pressured to adopt and enforce standards when practitioners speak out and get noticed by the establishment. Much of getting noticed, for better or worse, involves getting published in a traditional journal.
- apathy 10y agoAs opposed to actually doing science. This is the root of the problem. By the time I finished grad school I had been cited over 5000 times; that didn't make me a better scientist. Actually implementing clinical trials and supervising junior researchers, that did. (Oddly, so did reviewing others' work) Traditional journals are an artifact of the past. They interfere with the proper assignment of priority for discovery and their opaque review processes give some truly terrible work the stamp of "peer review". I submit some work to them, as requested by my colleagues, but if it were all my decision I'd never again send a paper to a journal that doesn't allow preprints (or considers it "prior publication") because that process is truly ass backwards. Fortunately NIH and NSF have started to notice the same. It's only a matter of time now.
- nostrebored 10y agoThe government believes psychology research and actively legislates against it.
- deleted 10y ago[deleted]
- laughinghan 10y ago
- bandrami 10y agoThere's a psychology journal[1] dedicated to only publishing null-hypothesis results. [1] http://www.jasnh.com/about.html http://www.jasnh.com/about.html
- munin 10y agothere is also the journal of universal rejection: http://www.universalrejection.org/ http://www.universalrejection.org/
- bandrami 10y agoThis has made my day; thank you.
- argonaut 10y agoA journal dedicated to negative results is IMO not the solution here. Academics must publish in reputable, prestigious journals in order to advance their career (a publication in a non-prestigious journal basically counts for nothing). A journal dedicated to negative results is never going to be a prestigious, impactful, or widely-disseminated venue.
- flukus 10y agoWhy is there no prestige associated with getting rid of rubbish? As a programmer the best feeling in the world is deleting code, reducing the cognitive load of the overall system without sacrificing it effectiveness. Does the academic world have the concept of technical debt?
- argonaut 10y agoOne flaw in your analogy is that if all you do is actually just delete code (you don't replace the code), that's not usually valuable. It is easier to publish null results than it is to publish legitimate positive results (of course easiest of all is to publish bogus positive results). It doesn't require creativity or brilliance to publish null results, just competence (competence enough to replicate). Of course, it is still valuable, so a thoroughly analyzed and argued negative result should be publishable in prestigious venues - but this is more a reflection of the value to the community than the actual novelty/brilliance of the work. Put another way: a university is not going to hire you if all you've published are null results. They want leaders who are going to pioneer/discover/invent/theorize new fields.
- skosuri 10y ago1. There are many places this could have been published without an importance review, eg PLoS ONE. 2. I think anyone interested in the replication problem needs to read this piece [1] by Peter Walter. As he put it: "It is much harder to replicate than to declare failure.". [1] http://www.ascb.org/on-reproducibility-and-clocks/ http://www.ascb.org/on-reproducibility-and-clocks/
- tmalsburg2 10y agoUnfortunately, the importance of the journal you publish in is decisive for whether or not you get a job in academia. It's fucked up but people like the author of the blog post are not in the position to change that.
- laughinghan 10y agoRe: 1. As stated in the concluding paragraph, she put it on researchgate.
- argonaut 10y agoShe's talking about point 1) when she talks about open access journals (plos one is an open access journal).
- mattkrause 10y agoAlternately, it would have worked really well as a "Registered Report." Cortex now offers a format where the paper is submitted in two stages. One first submits the introduction/background, methods, and proposed analysis, before collecting data. If the reviews are favorable, the paper is accepted, regardless of how the results turn out. There is a second round of reviews for the final manuscript to assess the quality of the data and ensure that the pre-determined analysis plan was followed. More here: http://cdn.elsevier.com/promis_misc/PROMIS%20pub_idt_CORTEX%20Guidelines_RR_29_04_2013.pdf http://cdn.elsevier.com/promis_misc/PROMIS%20pub_idt_CORTEX%... I've had a hard time selling collaborators on it ("TWO ROUNDS OF REVIEW?! Isn't one slow and painful enough?!"), but I really like this idea. It rewards interesting hypotheses and good experimental design, but avoids the ask-a-stupid-question-get-a-stupid-answer issue with null results.
- cpncrunch 10y agoPLoS one specifically says they will publish "Studies reporting negative results".
- apathy 10y agoOr save yourself the APC and put it on arXiv. Even Eisen (who was one of the founders) suggests this.
- cpncrunch 10y agoIts not peer reviewed so wont be found by researchers searching in pubmed or whatever. Thats the main point of publishing...so reviews will find the negative data.
- argonaut 10y agoPLoS one is an open access journal, which she talks about at the end.
- cpncrunch 10y agoI know. I was adding information.
- harry8 10y agoAny journal that refuses to publish a failure to replicate research they originally published without proper reasoning should be closed down. That journal should have such a reputational black mark next to it that nobody would want to publish there and anyone who already had should be at the door with pitchforks and torchers for tarnishing the researchers' reputations. If it's was important enough to publish research saying "here's something" then it's important enough to publish properly done research showing "actually, probably it's nothing." By definition. Or it's not science, it's fking marketing and the journal should be treated with the same scientific reverence we reserve for pepsi-cola advertisements from the 1990s.
- cleeus 10y agoYes, but there is just not a way to regulate journals other then by market forces - which seem to fail through information asymmetry about the journal quality and maybe some monopoly inducing positive feedback effects. Maybe law could break the information asymmetry between sellers and buyers of the journals by adding new clauses to the copyrighted works of state-sponsored scientists...
- harry8 10y agoHow about if say, Harvard, Stanford, MIT & Caltech made a pronouncement that from now none of their academics should submit research for publication in journal X on the grounds that journal X lacks commitment to being a scientific journal and it would diminish their reputations as academic institutions to have any association with that journal. Then canceled their libraries subscriptions to that journal in public, demanded a full refund and generally made a fuss in the newspapers. The NYT would print the fuss. I think that would change the dynamic of Scientific Journals pretty quickly. Pick one of the worst, get the evidence together, ring your opposite number at a couple of places that have similar numbers of nobel prize winners and go right at the rubbish journals with the wrecking ball having the reputation of being a place where the world's best research is being conducted.
- Someone 10y agoI somewhat disagree. They still should have the right/plight to vet the failure to replicate on quality. Without that, everyone and his dog would have publications in Nature and Science tomorrow.
- faux_intellect 10y agoAt this point, I think Google Scholar should step in and just put a replications section beside every scientific publication. People should be able to quickly and easily know how many times a study has been attempted to replicate and, of those attempted, how many times it has actually successfully replicated. It's unfortunate that replications aren't taken more seriously these days, but it also doesn't help that, when there are actual replications, you have to scour the internet for them rather than having them readily available to you.
- kriro 10y agoI think that's a pretty great idea actually. If anyone here has the reach to get this on their table please do reach out. It would be even greater if they'd use some replication metric in their ranking algorithm for the papers. The danger however is that this would just lead to only me-too studies being accepted and failed replications still being rejected. I'd also love if they'd add (possibly Google hosted) repositories where the data/scripts etc. that belong to the paper can be uploaded and archived for all eternity.
- mwambua 10y agoI saw something like this a while ago... Codalab (http://codalab.org/ http://codalab.org/). It's run by Percy Liang (https://cs.stanford.edu/~pliang/ https://cs.stanford.edu/~pliang/). It lets you create executable worksheets to go with your papers; and people can validate both the code and data used in your experiments.
- tehwalrus 10y agoThe paper was a meta-analysis of nine unpublished studies. While the issue of non-publication of negative results is extremely important, this should not be the test case for it. I agree with other comments that the lack of nul-results in journals is a massive showstopper for epistemology and confidence in scientific work. In my experience during my PhD, academics were unwilling to submit nuls, they wanted to find the actual answer and publish that instead - which leads to delays of years, leaving well known and obvious errors in existing papers unchallenged in public, and potentially leading to that scientist never even submitting the nul result.
- rtpg 10y agoWould it be possible to take the studies and actually develop them into the analysis? > the authors focus on a bunch of unpublished studies from a dissertation and a colleague who is not even an author of the present paper. There is no way of knowing whether these unpublished experiments meet the standards to be published in high-quality journals.” The way to know if the experiments meet the standards is to show it in the same paper, I guess?
- mattkrause 10y agoThat's what I would do. I think that isn't actually an unreasonable complaint: the current paper has very few details about the paradigm used in those experiments. There is a really brief table, along with some comments embedded in R output, but what the paper really needs is something that looks like a methods section from an experimental psycholinguistics paper. Something like this: Experiment 1: Nineteen native English speakers (demographics here) performed a lexical decision task. Three string of letters was presented on a computer monitor (viewing conditions here). Subjects were asked to report, via a button press, whether the middle string was a valid English word (left button) or psuedoword (right button). Although they had up to 5 sec to respond, subjects were instructed to respond as quickly as possible and the median reaction time was N.NN seconds. The original experiment manipulated (some irrelevant condition), which is not analyzed here. Instead, we calculated... Experiment 2: As above, except the subject pool consisted of native German speakers and the stimuli were either German words or German psuedowords. Experiment 3: Same as experiment 1, except that (irrelevant condition #2) was also manipulated instead of (irrelevant condition #1). This might not fit in the main paper--even online-only journals can be weird about page limits--but it would be fine in an appendix.
- arviewer 10y agoThere should be a Nulled Science Magazine!
- TorKlingberg 10y agoThere are dedicated journals for negative results actually. * https://en.wikipedia.org/wiki/Journal_of_Negative_Results_in_Biomedicine https://en.wikipedia.org/wiki/Journal_of_Negative_Results_in... * http://www.pnrjournal.com/ http://www.pnrjournal.com/ * http://www.jinr.org/ http://www.jinr.org/ * http://www.psychfiledrawer.org/journal_of_negative_results.php http://www.psychfiledrawer.org/journal_of_negative_results.p...
- schlipity 10y agoForgive me for not being an academic, so maybe this question is moot. Why isn't there a place that links to a given paper so that discussion about the paper can be centralized? It could also contain links to papers that link to that paper, among them would/could be the failure to replicate information, adding to the discussion. And I don't really mean a topical "this is what's new" site, I mean a historical "This is the paper, and this is what people have said about it." sort of site. This seems like a fairly elementary idea. The only seeming difficult bits I see are: a) Getting (legal?) access to these papers. b) Dealing with a large number of papers (millions?). c) Authenticating users to keep the discussion level high. d) Moderating the discussion in a way that doesn't piss off academia (impossible?). e) Keeping the number of these sites (competition, if you will) low so that the discussion is not fractured between them. It would seem like one of the "Information wants to be free" sites that host the papers that everyone shares with each other would be a great place to start something like this.
- jacobolus 10y agoIn general the way academics have a “discussion about the paper” is by writing another paper in response and citing the original. So what you’re looking for is the citation graph, and you can find it e.g. on Google scholar. Search for the paper you are interested in, and then click the link which lists all the papers which cite that one. You’ll probably need to be on a university campus or ask your anonymous internet friends for help to get around paywalls.
- gritzko 10y agoThat only gives you academic critique and follow-ups. Sometimes, some practitioner is pissed to the degree he/she posts an opinion online. That is extremely valuable, but never gets into the citation graph. For example... http://bramcohen.livejournal.com/20140.html?thread=214956 http://bramcohen.livejournal.com/20140.html?thread=214956 > I'd comment on academic papers more, but generally they're so bad that evaluating them does little more than go over epistemological problems with their methodology, and is honestly a waste of time.
- anonymfus 10y ago
- guard-of-terra 10y agoSome of those reviews are good materials for http://shitmyreviewerssay.tumblr.com/ http://shitmyreviewerssay.tumblr.com/
- jknoepfler 10y agoThe narrative would be more persuasive if it incorporated a story of how the paper evolved meaningfully in response to peer criticism. The question lingering in my mind after reading this is whether and how the paper was substantially revised (in light of reviewer feedback) between rejections. I'm sure it was (it has to have been, right?), but we don't get that feeling from the blog post. The author(s?) should have received a large amount of very good feedback between rejections from well-meaning peers in their scientific community. I don't recall reading about incorporating any of that feedback into subsequent revisions of the paper. The term "meta-analysis" probably should have been dropped after the first (pointed) rejection, for example, and the paper should probably have been broken down into two or three smaller papers rather than submitted as a 'meta-analysis' of unpublished work. This is not to say that peer feedback wasn't taken seriously. I don't know that at all. But if the goal is to persuade a skeptical audience that academic publishing is broken, the author should articulate how they followed best practices in response to rejection letters from peer-reviewed journals. The alternative is to sound arrogant and self-defeating, which I'm sure was not the intent!
- deleted 10y ago[deleted]
- mcguire 10y agoThis seems to be the original work in question: https://www.researchgate.net/publication/8098564_Reading_Acquisition_Developmental_Dyslexia_and_Skilled_Reading_Across_Languages_A_Psycholinguistic_Grain_Size_Theory https://www.researchgate.net/publication/8098564_Reading_Acq...
- LanceH 10y agoThere should be a failure to replicate journal. The standards committee would should be all about rigor so that just getting published there would be a demonstration of technique and ability if not headlines.