4 ms·
I would like to eventually see a "science results build system", where experimenters can register (preferably ahead of time) the analytical methods they will us
by Pyxl101 11y ago
I would like to eventually see a "science results build system", where experimenters can register (preferably ahead of time) the analytical methods they will use to analyze their data, expressed as runnable code. Any data sets already available, on which they plan to rely, will be similarly registered and made available to the "build system".
When the study is complete, the researchers upload their data, in as raw a form as technology permits. All analysis performed on the data to achieve the results should be expressed through runnable code evaluated by the build system. Ideally 100% of this code would be registered in advance, though to be realistic, in practice we'd find some of it to be written along the way.
Whether the code is registered in advance or written and registered along the way, in both cases the build system will have the complete code and data set, and will be able to reproduce the results. Ideally the experimenters simply take the output of the build system as their actual results. The scientific community will have complete visibility into the analytical methods of the study, since all data and analysis will have been captured by the build system.
The build system would ideally be agnostic to tooling and data format. Perhaps it is something like a data store alongside a runtime platform, capable of taking a virtual machine image and a data set, and executing the VM with the data set as input.
Tagline: "GitHub for scientific data and runnable experimental analysis"
- pjschlic 11y agoThis reminds me of the recent Planet Money "The Experiment Experiment" and they pointed out some journals requesting pre-registered experiments (pre-commited experiment methods and analysis submitted to somewhere online before the experiment is run), it's at the 18-ish minute mark.
- kriro 11y agoThis sounds like a pretty great idea if I understand it correctly. My cynical view of the business of science says it'll never get traction but the base idea sounds like a great way to get rid of a lot of cheating methods. It would also result in research that I'd immediately trust. Alas it's probably possible to game this theoretical system because you can still upload faulty data that leads to the desired results since the registered code can probably be reversed somewhat easily/would be open. I think there's two issues (which can also be combined). 1) Fake/false data. 2) Fiddling with methods until you get the results you want given a fixed data set. Your solution would be excellent for getting rid of problem #2. I think #1 is quite a bit harder and I haven't come up with anything better than "as open as possible"