3 ms·
> Yeah but jupyter notebooks suck at providing reproducible data science. That depends on how you use the notebooks. With just a tiny bit of discipline, you c
by enriquto 3y ago
> Yeah but jupyter notebooks suck at providing reproducible data science.
That depends on how you use the notebooks.
With just a tiny bit of discipline, you can integrate notebook users into your sane workflow. For example, encourage people to restart the kernel and run all cells a few times per day (and definitely, before sharing anything). Meaningful output artifacts can be saved into files, that are later read by the notebook and displayed.
Then, when users are satisfied with their notebook, they save it as a python file thanks to jupytext, and commit it to git.
This workflow integrates well with your makefile setup: to reproduce the notebook and obtain its results you simply run it as a script. If you want a pdf or a static html that shows the notebook as-is, you can nbconvert it from your makefile.
For example, if your makefile has lines like these:
%.ipynb : %.py ; jupytext $< --to notebook
%.html : %.ipynb ; jupyter nbconvert --execute --to html $<
Then you run "make foo.html" and it will convert "foo.py" to "foo.ipynb", run all the cells, and produce a static visualization "foo.html". Since the intermediary notebook is not marked as a precious file, it is deleted automatically by make.
Notice that you can simply run "python foo.py" as well, to produce the valuable output artifacts.
In the end, jupyter becomes just an editor of python files. A fancy editor, that allows interactive execution of pieces of code, which is great.