3 ms·
I think this is a great idea at least for fields where reproducibility is comparatively easy to achieve, e.g. computer science: Include your data, code and run
by ThePhysicist 8y ago
I think this is a great idea at least for fields where reproducibility is comparatively easy to achieve, e.g. computer science: Include your data, code and run environment (using a suitable technology) with the paper to give reviewers a chance to "play" with your experiments (if they actually have the time for this).
In other fields such as experimental physics or biology it might be more difficult to achieve the same effect though, as experiments are quite hard to repeat in general: For example, in my former field (experimental quantum computing), building the setup and fabricating the sample required for a given experiment could take years of effort, making it almost impossible to "just" reproduce someone's work for the sake of verification.
That said, in experimental quantum physics the exciting results tended to get replicated within a couple of years by different teams anyway, not because these teams wanted to verify the results but rather because they wanted to build their own experiments on top of them (I imagine this is similar in other fields). Another natural way of exchanging knowledge and improving reproducibility was via the exchange of PostDocs and PhD students: If you do good work in one group, another group will usually be very eager to give you a position so that you can help them to set up the experiments there as well. I'd even argue that this is one of the main mode of knowledge dissemination in experimental science today, as most research papers are just extremely hard to reproduce without the specific -and often not encoded- knowledge of the individuals that ran the experiments.
I'm not sure though if it is practical to document everything in a research paper in such a way so that any person can reproduce a given experiment, as many of the techniques are very specialized and a lot of the equipment in the labs (at least in physics) like sample holders, electronics, chip design templates and fabrication recipes is custom-built, so documenting down to the last detail would take years of effort.
That's why (IMHO) written PhD theses are so important, because that's kind of the only place where you can write 200-400 pages about your work, and where you can include minute details such as your chip fabrication recipe, a description of your custom-built measurement software and a detailed summary of the experimental techniques used in your work. In that sense, PhD theses are probably more important to reproducibility than short papers.
- dekhn 8y agoYour description of experimental quantum physics exactly matches the work done in state of the art biology. It basically takes 10-20 years of training and excellent brains to reproduce your competitor's paper in a way that lets you build on top of it. Many people who complain that papers aren't reproducible just don't have the skill to move the state of the art, because it's become so esoteric and challenging to run the critical experiments. I used to be a "everything must be replicable by even the simplest of people" person, but I changed to "for progress to be made there must be <X> competent people who can reproduce challenging experiments and run new ones".
- BoiledCabbage 8y ago> because it's become so esoteric and challenging to run the critical experiments. Maybe more people would be able to run the experiments if papers detailed how to do so. The knowledge would no longer be esoteric.
- pas 8y agoIt's completely on and off the walls bollocks with sriracha sauce on top and a good kick to the aforementioned balls that we don't have enough space (and time) for writing down the important bits of experiments.