5 ms·
Strategies for making reproducible research the norm
- alanbernstein 3y ago> An alternative approach is to encourage students tnduct replication studies, evidence synthesis, or meta-research as part of graduate theses. Why not require it?
- deleted 3y ago[deleted]
- SubiculumCode 3y ago[flagged]
- rowyourboat 3y agoIsn't the main problem that academics are measured by the number of publications they publish, and reproductions of existing studies aren't published by the main journals, thus there is little incentive to try and reproduce findings? I never thought this was a problem of ability.
- reignbeaux 3y agoUnfortunately this is only part of the problem. Even studies on ML that use public datasets, which are the kinds of studies that when code is shared should be very easy to reproduce, are often surprisingly hard to repeat. Sometimes only parts of the code are published, the code has a lot of bugs (who knows why? Added intentionally?), the code is very badly documented, or the exact libraries are not specified properly. And this is in a field where everything is based on code, where in principle reproducibility is easy. Go into materials science or chemistry and try to synthesize something following a published paper and you get all sorts of problems. Different equipment, different temperature, not all steps documented, ... Reproducing experimental findings can take you months.
- mitthrowaway2 3y agoPage limits certainly don't help!
- sigtstp 3y agoIt still largely comes down to incentives from what I've seen. A lot of times all anyone (from the researcher to the reviewer) cares about is the paper. Journals don't check that code actually works, and a lot of researchers don't spend time on preparing their code. They feel there's no need, since they now got a new article on their CV. It's true that they may not have the skills and experience to produce good code they can share (depending on the area), but often 1) there's no time to prep code since they've got 3 other projects going on and a crazy work pace 2) the code is seen as something incidental and secondary - what matters to them is the figures and results 3) some groups want to milk a topic for a few papers so they're guarding their code and data. Luckily at least plenty of journals demand access to data or even making it public.
- staunton 3y agoIn fact, there's even more incentives for researchers to make reproducing their work as hard as possible. For example, what if someone tried to reproduce it and found contradictory results? In both cases (reproducer made mistake, original made mistake) it's additional hassle that the original authors can basically only suffer and never gain.
- edgyquant 3y agoThis is just you confirming that tons of research is essentially fraudulent. If it can be contradicted it absolutely should be, that is how fields progress and weed out bad ideas.
- geokon 3y agoIf you fail to reproduce some important research then I think that would absolutely get published. (see the recent superconductor drama) So if you feel some impactful work is suspicious .. I think disproving it would absolutely be incentivised If you show its actually correct.. Well then usually it's not that hard to push the envelope a bit further and say something new. That happens all the time
- ttymck 3y agoIt seems to me, instead of funding a new college or traditional research institute, some benefactor ought to fund a "research reproduction institute", dedicated to identifying and reproducing suspicious publications.
- sharma-arjun 3y agoYes, but in the vast majority of cases, it's hard to tell just by reading a paper if there's been dishonesty somewhere in the pipeline. Also, the LK-99 example is an exception, not the norm–the chances of receiving significant attention for a replication study are near zero in almost all other cases.
- geokon 3y agoI just don't think it's really relevant. If the research is impactful, then it'll be replicated (at least in part) when the next person tries to build on the results. If it doesn't replicate then they'll probably end up discovering something new/different - and that'll lead to it's own paper. Even in the ideal world, you effectively almost never end up with a replication paper. Either it replicates and you add on your own novel research. Or it doesn't replicate and you discover something new You can in theory end up with a super dull null result that disproves someone else's claim. But even then, when you set out on the project you're aim is to add something new on top of what's been already done. This happens all the time
- matusp 3y agoAnother issue is that making things reproducible costs you time and that is exactly what most researchers do not have. For example, many ML papers have code that is just a barely working Jupyter notebook. To make it reproducible you would have to create a reproducible environment, package the data, and prepare scripts that would rerun all the experiments you have done. That can take several weeks, but it will not increase the chance of acceptance for your paper at all.
- BenoitP 3y agoNo. Several weeks is the time it takes to learn and master Docker. About two hours is the cumulative time one must cater to the Dockerfile for a 3 weeks project. But it requires institution insisting on reproducibility, and fostering best practices to make it even easier for the researchers to be compliant. I get it that reproducibility can be quite hard for biology. But ML cannot be taken as an example of a hard problem.
- matusp 3y agoI agree that docker is great. But docker solves only one of the problems mentioned above (env) and even that solution does not work for some teams that run their experiments in GPU clusters where docker is not supported.
- sideshowb 3y agoPerhaps the core issue is that academia excels at being a textbook case of goodhart's law. If/when reproducibility became a target then the academic system would/will likely make an equally bad mess as it has with its current targets.
- tangjurine 3y agoMaybe the cause is funding sources that fund researchers publishing too often, and not funding other researchers to double check their work
- sundarurfriend 3y agoMore precisely, making things reproducible after the fact costs you significant time - there are tools for reproducible setups that take maybe an hour (at most) to setup upfront, after which it takes very little effort to do your work within that framework and keep things reproducible (for eg. Julia has DrWatson, DataDeps, etc., I'd be surprised if Python doesn't have equivalents). The problem is knowing upfront which of your work would need to be reproducible, or having the discipline to do all your hacking starting from such reproducible setups.
- somenameforme 3y agoYou're also leaving out the biggest issue. Journals generally don't want to produce negative results. If you spend researching [shocking possibility] and it turns out that [shocking possibility] isn't true, you're not getting published. It motivates everything from HARKing [1] to outright data manipulation. By contrast if negative results were seen as valuable, then none of this is an issue. On the other hand, it really is the case that there's just not much of any value in learning that [shocking possibility] is, as everybody would naturally expect, indeed not the case. And filling up limited journal space with such discoveries would seem to be counter-productive, at best. And when you have limited space/funding for researchers, one guy who keeps proving everything everybody knows to be false, to be false, is always going to be perceived as less valuable than one making [shocking discovery] [... which ends up being proven false years later]. [1] - https://en.wikipedia.org/wiki/HARKing https://en.wikipedia.org/wiki/HARKing
- QuesnayJr 3y agoIt can also be hard to judge whether replication failed because the result is bogus or replication failed because the replication team is themselves incompetent.
- edgyquant 3y agoDid they follow the exact same steps claimed to result in something? Did it result in that thing? The repetition team being incompetent sounds like a cop out. The researcher did a bad job and it’s on them to explain better etc in that case. No excuses, if it can’t be reproduced it isn’t taken seriously no exceptions
- crabbone 3y agoSort of. Yes and no. There has to be a metric to assess researcher's performance. Otherwise we won't know what research is worthwhile. When the rules of the game are known, players will find their way to cheat, or at least bend the rules to their advantage. So, for example, suppose negative results become as valuable: well, they are easier to produce. They are also less valuable as stepping stones for further research. Given that, you'd still need to have a metric that compares publishing positive results to negative results. Even if you declare them to be equally important, the shared understanding will be that they aren't. And one would be more important than the other. And here were are back to square one. There are some minor things that can be done in the near future. For example, results produced with code must come with the code that produced these results. A lot of research bodies resist this because they want to commercialize their code, or their code may inadvertently contain organization's secrets and therefore needs more auditing... but, in the end of the day, it needs to be made clear that this is a necessary and unavoidable price to pay. Data sharing is even more problematic. Beside confidentiality concerns, data is always a bargaining chip in the game of getting collaborators (and grants). Should it be made public, it loses its value to those who collected it. Right now, the trend is: if you managed to collect a worthwhile dataset, then you'll cover yourself foot to head with NDAs, contracts of all kinds etc, and will sit on it, exploiting it for a series of research. And if anyone wants to do research on the same subject, you will only invite them if they bring grants or equipment etc. But you cannot really verify results w/o having the data available. Even if you have the code. --- It's really sad to see how research is doing wrt' programming in part because of the above, but I don't think the programs outlined in OP will have a noticeable effect. They don't paint a convincing picture in terms of incentives, i.e. they don't answer the question why would researches want to do any of that RepRes and OS training. Even in computationally-heavy research today you often find that all the computation work is outsourced by the researchers and they themselves have no clue what their code is doing. Above were all sorts of arguments for why the current (or yours) approaches are ineffective. But I don't claim to know what needs to be done.
- fsckboy 3y agoit's not entirely important (theoretically) that research be reproducible, but it would be important that the attempt at reproduction is made in a timely way. In many cases today, it's not so easy to set up the experiments in the first place to attempt it.
- naasking 3y ago> it's not entirely important (theoretically) that research be reproducible Well that's confusing, because reproducibility is a core tenet of science. How else could you tell whether your results were a statistical fluke or due to some flaw in the study design?
- oddmiral 3y ago12. Create a journal for reproduced papers only.
- Qem 3y agohttp://rescience.github.io/ http://rescience.github.io/
- seydor 3y agothe root of the issue is trust in published science. if people focus too much on reproducibility, then you will get people trying to game the system to publish reproducible but still misleading/cherrypicked science
- RandomLensman 3y agoThe important thing to change would be that the consumers of research need to care about reproducibility. Currently, that is only the case in very few instances and there reproducibility is typically checked quickly.
- robertlagrant 3y agoIdea: create a journal that is GitHub based. It generates any data plots/tables through CI, which means you need to have your data available and consumable, and containerised code. Its PR reviewers are scientists doing peer review. Then you get public review and comment, reproducible environments, and open data.
- fleischhauf 3y agoI like the idea a lot. I don't expect a lot of feedback/reviews from other scientists, there are just not that many and they are likely busy with something else
- rsfern 3y agoYou could still have assigned reviewers like in a conventional journal, which maybe if done right could stimulate broader discussion JOSS has a review process like this, but I don’t think I’ve heard of any journals focused on primary scientific results using this kind of GitHub based review process https://joss.readthedocs.io/en/latest/submitting.html#the-review-process https://joss.readthedocs.io/en/latest/submitting.html#the-re...
- robertlagrant 3y agoThanks! I was thinking an actual journal that would have a subscription. I wonder if it could also get people to sponsor replication studies, and publish them linked alongside.
- smokel 3y agoI was recently interested in road traffic speed prediction. The problem with academic research into predicting speeds is not as much the lack of reproducibility, but the lack of measures that make sense. Hundreds of papers compare a short-term prediction (say 30 minutes ahead) using mean average error, or similar measures. This completely ignores the fact that vehicle speeds are mostly constant and only vary significantly during rush hours or other disruptions, at which time the process of breakdown is rather chaotic. From what I gather this will not change easily, because it is considerable academic fun to improve on a number for existing benchmarks, instead of redefining the game. I'd love to help academics in this field who need some guidance on practical applications!
- daoboy 3y agoIt seems remiss to leave out the publish or perish dynamic that requires anyone pursuing a career in academia to churn out publications with little to no regard for their quality. Especially in light of the evidence in recent years that peer review is a poor filter for fraud. Deincentivising an endless stream of trash publications will both cut down on the BS and narrow the field of research one might try to reproduce. I don't have strong feelings about the optimal number of papers that should be published, but my sense is that we are running an order of magnitude above that number.
- edgyquant 3y agoWhy not just require independent reproduction to even publish?
- mikrl 3y agoThat would also incentivize the paper writers to be exacting in describing their methodology and lead to higher quality papers. If a paper needed to clear the bars of - can be reproduced by reading the paper and/or a few meetings with the original team - data is independently reproducible to within the usual statistical parameters It would presumably increase the signal:noise of research.
- BurningFrog 3y agoI like this. Requiring reproduction is several bridges to far. But requiring reproducibility seems complete reasonable!
- swayvil 3y agoSo you just take their word for it? Pretend that reading about it is equivalent to seeing it?
- anonymouskimmer 3y agoMany scientists reading an article will seek to extend the work in the article, not duplicate it. If the extension fails then they may spend the effort to replicate the original results. Even reproduction doesn't solve the problem of theoretical errors: https://news.ycombinator.com/item?id=36230450 https://news.ycombinator.com/item?id=36230450
- civilized 3y agoI don't know if we will get a healthy research community by attempting to police unhealthy communities out of existence. This will only lead to research activity being further stifled by useless bureaucracy that fails to restrain most bad research while slowing down what actually matters, quality research. Instead, I think we need to focus our attention on outstanding communities and how to distinguish them from less outstanding ones. We don't need to be threatened by the mere existence of fake journals that publish articles with "counterfeit consciousness" in them. No one actually reads that stuff. It only exists to feed badly-managed communities with terrible incentives. And the solution to such bad communities is to create good ones and showcase their work. Not waste our time obsessing over the fear that somewhere, someone is not a great researcher. Poor research is the norm and should be ignored. Good research is the exception that should be recognized and nurtured.
- naasking 3y agoI'm a little confused by your post. This paper is about increasing training about reproducible research and open science at all levels of education, with the goal that awareness of them will increase participation and reduce the incidence of pitfalls that lead to faulty research. Is this not exactly showing people how to conduct "good research that should be recognized and nurtured"? Is this really any different than requiring training in proper statistical methods?
- civilized 3y agoNevermind, it looks like I got mixed up responding to a different comment or article. Sounds dumb but it has happened to me a couple times over the years...
- analog31 3y agoHere's a thought: Teach it in elementary school, so that it becomes a habit, and benefits those who are not training to become professional scientists. It doesn't even need statistics or "code" since reproducible science predates those things. If it becomes a habit, and a natural expectation, then teaching the specifics pertaining to any specific field at an advanced level should be easier. And maybe it would encourage the public to demand better science reporting. Also, mark the topics in the introductory textbooks that are not based on reproducible research, so that students know the actual state of the field that they're studying. I do industrial R&D, and my stuff doesn't get published at all, but I benefit from doing work that is at least "open and reproducible" within my organization. It actually improves the quality of my work.
- corethree 3y agoNo. Teaching it in elementary school doesn't do shit. Everybody knows this stuff. The reason why a lot of people don't make reproducible research is the same reason why communism tends not to work. In order to make it work you need to put incentives in place. For example. No journal publishes a paper unless the work has been reproduced by someone from a different institution.
- loldk 3y ago[dead]
- brightstep 3y agoWe need a publicly funded body whose purpose is to issue grants for firms to reproduce scientific findings. A scientific reproduction corps. Extra can be awarded for finding flawed/fraudulent research. This would create a community of organizations that counter-balance industry incentives and instill trust in our scientific process.