4 ms·
Thanks! That's awesome. I have long thought that a platform for scientific publication where code and data are included would be valuable, and I'm glad to learn
by jcrites 10y ago
Thanks! That's awesome. I have long thought that a platform for scientific publication where code and data are included would be valuable, and I'm glad to learn it exists. Going further, imagine the code is like a container that anyone can run to reproduce the findings, including statistical analysis and summary on raw data sets -- the key findings should "build". Perhaps the platform also provides continuous build during "development" (research) so that researchers can work privately and then publish their "repo" publicly along with their paper. An easy way to clone and reproduce the build after publication: "fork my research"
As an extreme version of the idea, imagine if the actual paper itself (TeX) and all the data within it are also built as part of the repository; any graphs in the paper are rendered from data in the repo, any numbers are data accesses, etc. This probably wouldn't be helpful to researchers, but it would promote scientific reproducibility and aid everyone building on a researcher's work. Tremendous work goes into authoring the papers themselves, sometimes with methods or tricks that are private; laying it all out publicly would greatly help students of science.
Going even further: to avoid cherry picking of positive results, review boards expect experimental criteria to be published (at least privately to them) in advance, for research that involves capital-E experiments. Perhaps this includes analysis code at least in prototype form; like test driven development, the acceptance criteria are written first. When the paper is ready for review, the reviewers can compare the initial prototype analysis logic to the final form. Perhaps the board also expects all data and trials collected during experiments to be made available in the repository, whether positive or not. All collected data should be in the platform, in the most raw form it was originally recorded, as well as all steps of summary and analysis.
I wonder if a process and platform like this could contribute to the integrity and quality and reproducibility of scientific research. People funding research ought to ask for it, especially public funded research, and the whole repo is made open eventually if not initially.
Perhaps as part of the platform's value prop to researchers (on whom it is imposing probably more burdens than benefit, for sake of public benefit), the hosting is free and funded by a foundation, or steeply discounted. (OK, it won't pay for LHC scale data sets, but otherwise ...) So using it to host your data, code, and paper is free, at least up to a point. I would be interested to contribute time and resources toward building or supporting a platform like this.
- nitrogen 10y agoYour suggestion of dynamically built papers also reminds me of the work on interactive documents by Bret Victor. An example: http://worrydream.com/LadderOfAbstraction/ http://worrydream.com/LadderOfAbstraction/
- gcr 10y agoI don't think research should be as structured as software development. In CS, many of the most interesting papers come about when the authors discover something unexpected or non-intuitive and choose to explore down that thread. That's why it's research -- sometimes you can't know what you will find until you're there.
- tremon 10y agoI don't think research should be as structured as software development In many ways, it already is: good research requires meticulous log keeping in order to reproduce results, and equal effort must be spent on maintaining references to other literature, or you risk missing a citation in a published paper.
- lccanon 10y agoYou may be interested by looking into "An Effective Git And Org-Mode Based Workflow For Reproducible Research", which describes more or less your idea (https://hal.inria.fr/hal-01112795/document https://hal.inria.fr/hal-01112795/document).