4 ms·
I'm not a Jupyter user, but I solve the reproducibility problem with Make. As a project moves from exploration toward production, the entire thing is wrapped i
by ISL 8y ago
I'm not a Jupyter user, but I solve the reproducibility problem with Make.
As a project moves from exploration toward production, the entire thing is wrapped into a Makefile that can flow from raw data to publication in a single call to make.
- ontouchstart 8y agoTo have reproducible prototypes, I use Make to wrap the whole workflow in docker. Then I push the code to gist and forget about it. Although GitHub gist doesn't allow binary file, images embedded in .ipynb (JSON), on the other hand works in gist. Here is an example. https://gist.github.com/ontouchstart/854a3c280b81f530d3ae9cb4c1bcbdc4 https://gist.github.com/ontouchstart/854a3c280b81f530d3ae9cb... The notebook generated by nbconvert (see the instruction in the Makefile) is too big to display in GitHub gist Web UI but works fine in nbviewer. https://nbviewer.jupyter.org/gist/ontouchstart/854a3c280b81f530d3ae9cb4c1bcbdc4/raf.nbconvert.ipynb https://nbviewer.jupyter.org/gist/ontouchstart/854a3c280b81f...
- kristjansson 8y agoThis has been my solution as well. There’s little that feels as good as running `make -B report` and watching the whole thing be rebuilt from scratch. How do you manage encapsulating each step, and passing data between them?
- canhascodez 8y agoWould that be easier with rake than with make?
- ISL 8y agoI don't do it well. See https://github.com/4kbt/ReplicableAnalysis https://github.com/4kbt/ReplicableAnalysis and https://github.com/4kbt/PlateWash https://github.com/4kbt/PlateWash as examples. The former is smaller/less complicated. The latter was my thesis work -- more complicated and (unfortunately) abuses recursive calls to Make.