6 ms·

Git and Jupyter Notebooks Guide

176 points by sixhobbits 3y ago

enriquto 3y ago
Curious that they discuss several options, but ignore the totally obvious one: just use jupytext [0]. Jupytext is a (tiny) jupyter extension that reads/writes notebooks as python files, with text cells being represented as comments. With jupytext, you do away with the stupid .ipynb format. As long as you don't need to save the cell outputs, which is the case for version control, jupytext is the way to go. People: pip install jupytext. All your python files will become notebooks, and your notebooks will become python files. [0] https://jupytext.readthedocs.io/en/latest/ https://jupytext.readthedocs.io/en/latest/
- pletnes 3y ago
  What happens to the outputs in this case? I found the outputs to be both the most useful parts of notebooks, but also the most troublesome for diffing and versioning.
  - enriquto 3y ago
    Why would you commit the outputs into git? That would be like committing compiled binary objects or pdfs. Of course the outputs are useful, but you just want to commit the sources. The .ipynb stores inputs and outputs together in an unholy way. It is much cleaner to separate them. The inputs are python (or markdown) files that you can edit with a text editor and version control with git. The outputs are html, pdf, or whatever you want to nbconvert to and share. The .ipynb file would only be useful if you want to share a stateful notebook, whose state cannot be easily reproduced by the people who you share it with. But that would be really bizarre and definitely in bad taste. Sharing the .ipynb is akin to sharing your .pyc files. I love working with notebooks, but as a measure of hygiene I avoid .ipynb files altogether.
    - RobinL 3y ago
      Most of the time I agree, but if you want to e.g present a tutorial as a webpage, having an ipynb with both inputs and outputs becomes a feature. You can even bundle into docs e.g. https://moj-analytical-services.github.io/splink/demos/02_Exploratory_analysis.html https://moj-analytical-services.github.io/splink/demos/02_Ex...
    - apwheele 3y ago
      Some people use the outputs like documentation, since github renders the notebook contents nicely in the browser. I agree it is not the best practice in many situations, I like using it on occasion though. Another alternative if you want the outputs is to use nbconvert to convert the output to markdown, https://andrewpwheeler.com/2021/09/06/using-jupyter-notebooks-to-make-nice-readmes-for-github/ https://andrewpwheeler.com/2021/09/06/using-jupyter-notebook...
    - pletnes 3y ago
      Sometimes I work on software development, and this mindset («the only valuable asset is the code») makes total sense. But if I work on analytics / datascience projects, the analysis including outputs could be time consuming to run, validate, and visualize. In these cases, it might be required to version the outputs. I’ve never used jupyter for taking notes in a lab setting, but with more and more instruments being computer/network connected, I imagine this would make total sense - put your data and notes with your analytics work. Many jupyter users are not «software developers», they just use code to perform their work.
      - Zandikar 3y ago
        Precisely this. When your output is something like research data, or even just something that generally takes a long human time to complete (hours vs Microseconds) it makes a lot of sense to version and keep outputs, at least on major "versions".
        adament 3y ago
        But would you version it by storing it as output in an ipnyb file where it is overwritten if you rerun that cell? I would store the data in a versioned database or as separate data files in the repository (possibly stored in git-lfs). And I would store results of the analysis as data files / image files / whatever else, NOT as ephemereal outputs in an ipynb file. But I am pretty far down the “ipynb files are for local use only” path.
  - kzrdude 3y ago
    I use jupytext paired with ipynb files. Only store the .py files in git. The ipynb files act as a local cache of outputs. Outputs are loaded from the ipynb even if you open the .py notebook.
  - Helmut10001 3y ago
    I use Jupytext since years. It allows me to have three types of synced notebook versions: 1) .ipynb (for opening/running), 2) .md (formatted code+comments, without outputs) and 3) *.py (python formatted, code+comments). I commit the Markdown-version, but I also use the py-version of notebooks for chained notebook imports. Allows me to split larger notebooks into multiple smaller ones. Both of these options are a blessing and Jupytext works super-robust. Finally, when I want to archive (and share) notebooks _with_ outputs once in a while, I have a cell at the end to convert (nbconvert) to HTML, and I commit this html file. The Markdown-version remains as a clean basis for commit history. The HTML file is much better suited for sharing and archiving than the ipynb file.
    - Helmut10001 3y ago
      Here's another HN comment with links of an example repo [1]. [1]: https://news.ycombinator.com/item?id=36516836 https://news.ycombinator.com/item?id=36516836
  - bootsmann 3y ago
    We use jupytext with dvc. You can generate the notebook in dvc.yaml using the jupytext cli and then push this alongside the .py file.
- joouha 3y ago
  Euporie (my terminal Jupyter notebook editor) also supports Jupytext
- cycomanic 3y ago
  This was the first thing I wanted to post when reading the article. Jupytext is excellent, although i typically use MyST (an extended Markdown syntax).
milliams 3y ago
There is a draft JEP (Jupyter Enhancement Proposal) for Markdown-based notebooks (https://github.com/jupyter/enhancement-proposals/pull/103 https://github.com/jupyter/enhancement-proposals/pull/103) which will make it a little more RMarkdown-like.
wdroz 3y ago
If you don't need to "commit" the output, you can just use nbconvert[0]: jupyter nbconvert --clear-output --inplace my_notebook.ipynb So you can use git as usual, like for code. [0] -- https://nbconvert.readthedocs.io/en/latest/ https://nbconvert.readthedocs.io/en/latest/
- andrecosta 3y ago
  nbstripout[0] does that and installs a pre-commit hook [0] -- https://github.com/kynan/nbstripout https://github.com/kynan/nbstripout
sashk 3y ago
You don't need to commit output into the git. I used pre-commit filter in git, where it will strip all output from the notebook before it was committed into repository. This allowed us to review the code changes of notebooks.
joelschw 3y ago
The native GitHub feature in preview will make this a lot better for those able to use it https://github.blog/changelog/2023-03-01-feature-preview-rich-jupyter-notebook-diffs/ https://github.blog/changelog/2023-03-01-feature-preview-ric...
- SalsaCrotch 3y ago
  This feature has resolved the problem for our team.
sdfghswe 3y ago
I haven't read the link and I'm not going to. I realized that jupyter notebooks are a flawed idea when I've tried vs code. vs code uses jupyter-the-protocol (as opposed to jupyter-the-notebooks) in order to give you a notebook-like experience that doesn't involve the jupyter notebook file format. VS code's interactive files are valid python code. To me that killed jupyter notebooks. Why use something that is strictly worse in every respect?
- kortex 3y ago
  It sounds like you are using the tool wrong. Jupyter notebooks are strictly superior to anything else (namely: code only, spreadsheets, matlab/octave) at their primary use case, which is interactive data science (writing code to manipulate some data, while actively revising the code, or sharing the results of that code with others). Nothing even comes close. There's a reason it's dominant in the data science field. Your workflow works for you but the jupyter workflow works for millions of students, data scientists, and even developers. Heck I even know all the ways to avoid jupyter, and I still use it often, because it's so convenient.
  - kevinskii 3y ago
    I agree with the OP. VS Code using the Jupyter protocol is superior to notebooks in almost every respect in my experience. It gives you an excellent debugger, the ability to track changes in Git without any modification, and you can also run as a regular Python script.
  - sdfghswe 3y ago
    > It sounds like you are using the tool wrong. Jupyter notebooks are strictly superior to anything else (namely: code only, spreadsheets, matlab/octave) at their primary use case, which is interactive data science (writing code to manipulate some data, while actively revising the code, or sharing the results of that code with others). > Nothing even comes close. There's a reason it's dominant in the data science field. > Your workflow works for you but the jupyter workflow works for millions of students, data scientists, and even developers. Heck I even know all the ways to avoid jupyter, and I still use it often, because it's so convenient. Copy pasting your comment here so when you eventually delete it people can still see the ignorance. You have absolutely no clue what you're talking about. Worse, it seems like you didn't read what you're responding to.
kortex 3y ago
Wow, no mention of DVC (http://www.dvc.org http://www.dvc.org)? That has been invaluable for data scientist workflows. I definitely do like to strip notebooks and make them run-idempotent to the best of my ability, but sometimes you just need stateful notebooks. And since .ipynb are technically json but in reality act more like a binary file format (with respect to diffing), DVC is the ideal tool to store them. Don't get me started on git annex or LFS, both of those took years off my life due to stress of using them and them bugging out. Also I am hardly a fan of XML, but does anyone feel like notebook files would have been a near-ideal use-case of it? It's literally a collection of markup. The fact that json was chosen over xml I think is somewhat damning of xml as an application data storage format. I think xml is perfectly cromulent as a write-once-read-many presentation format or rendering target (html, svg, GeniCam api info), but it seems to flounder in virtually every other domain it's been shoehorned into, with the exception of office application formats. Actually, downthread there is a link to a jupyer enhancement proposal for a .nb.md markdown based format. I think this is great. One theme I keep coming across in my computer science journey is that formats which have mandatory closing endcaps are kind of a PITA. It seems the stream-of-containers (with state machines as needed) is all-around better. JSON-LD is better than JSON, streaming video formats are better than ones that stick metadata at the end, zip is... an eldritch horror, etc.
nvy 3y ago
Seems to me that this article does a great job explaining why jupyter notebooks are a poor collaboration tool. I wish that non-emacs implementations of org were more commonplace, as it's a pretty sane markup language and supports embedded code and graphics, diffs nicely, and doesn't introduce the insanity of JSON.
TeeWEE 3y ago
My quick solution is to not commit the result cells, only the commands. So its just code.
DryLabRebel 3y ago
You forgot another issue: - containing potentially sensitive data in your notebook

176 points by sixhobbits 3y ago

Curious that they discuss several options, but ignore the totally obvious one: just use jupytext [0]. Jupytext is a (tiny) jupyter extension that reads/writes notebooks as python files, with text cells being represented as comments. With jupytext, you do away with the stupid .ipynb format. As long as you don't need to save the cell outputs, which is the case for version control, jupytext is the way to go. People: pip install jupytext. All your python files will become notebooks, and your notebooks will become python files. [0] https://jupytext.readthedocs.io/en/latest/ https://jupytext.readthedocs.io/en/latest/

pletnes 3y ago
What happens to the outputs in this case? I found the outputs to be both the most useful parts of notebooks, but also the most troublesome for diffing and versioning.
- enriquto 3y ago
  Why would you commit the outputs into git? That would be like committing compiled binary objects or pdfs. Of course the outputs are useful, but you just want to commit the sources. The .ipynb stores inputs and outputs together in an unholy way. It is much cleaner to separate them. The inputs are python (or markdown) files that you can edit with a text editor and version control with git. The outputs are html, pdf, or whatever you want to nbconvert to and share. The .ipynb file would only be useful if you want to share a stateful notebook, whose state cannot be easily reproduced by the people who you share it with. But that would be really bizarre and definitely in bad taste. Sharing the .ipynb is akin to sharing your .pyc files. I love working with notebooks, but as a measure of hygiene I avoid .ipynb files altogether.
  - RobinL 3y ago
    Most of the time I agree, but if you want to e.g present a tutorial as a webpage, having an ipynb with both inputs and outputs becomes a feature. You can even bundle into docs e.g. https://moj-analytical-services.github.io/splink/demos/02_Exploratory_analysis.html https://moj-analytical-services.github.io/splink/demos/02_Ex...
  - apwheele 3y ago
    Some people use the outputs like documentation, since github renders the notebook contents nicely in the browser. I agree it is not the best practice in many situations, I like using it on occasion though. Another alternative if you want the outputs is to use nbconvert to convert the output to markdown, https://andrewpwheeler.com/2021/09/06/using-jupyter-notebooks-to-make-nice-readmes-for-github/ https://andrewpwheeler.com/2021/09/06/using-jupyter-notebook...
  - pletnes 3y ago
    Sometimes I work on software development, and this mindset («the only valuable asset is the code») makes total sense. But if I work on analytics / datascience projects, the analysis including outputs could be time consuming to run, validate, and visualize. In these cases, it might be required to version the outputs. I’ve never used jupyter for taking notes in a lab setting, but with more and more instruments being computer/network connected, I imagine this would make total sense - put your data and notes with your analytics work. Many jupyter users are not «software developers», they just use code to perform their work.
    - Zandikar 3y ago
      Precisely this. When your output is something like research data, or even just something that generally takes a long human time to complete (hours vs Microseconds) it makes a lot of sense to version and keep outputs, at least on major "versions".
      - adament 3y ago
        But would you version it by storing it as output in an ipnyb file where it is overwritten if you rerun that cell? I would store the data in a versioned database or as separate data files in the repository (possibly stored in git-lfs). And I would store results of the analysis as data files / image files / whatever else, NOT as ephemereal outputs in an ipynb file. But I am pretty far down the “ipynb files are for local use only” path.
- kzrdude 3y ago
  I use jupytext paired with ipynb files. Only store the .py files in git. The ipynb files act as a local cache of outputs. Outputs are loaded from the ipynb even if you open the .py notebook.
- Helmut10001 3y ago
  I use Jupytext since years. It allows me to have three types of synced notebook versions: 1) .ipynb (for opening/running), 2) .md (formatted code+comments, without outputs) and 3) *.py (python formatted, code+comments). I commit the Markdown-version, but I also use the py-version of notebooks for chained notebook imports. Allows me to split larger notebooks into multiple smaller ones. Both of these options are a blessing and Jupytext works super-robust. Finally, when I want to archive (and share) notebooks _with_ outputs once in a while, I have a cell at the end to convert (nbconvert) to HTML, and I commit this html file. The Markdown-version remains as a clean basis for commit history. The HTML file is much better suited for sharing and archiving than the ipynb file.
  - Helmut10001 3y ago
    Here's another HN comment with links of an example repo [1]. [1]: https://news.ycombinator.com/item?id=36516836 https://news.ycombinator.com/item?id=36516836
- bootsmann 3y ago
  We use jupytext with dvc. You can generate the notebook in dvc.yaml using the jupytext cli and then push this alongside the .py file.
joouha 3y ago
Euporie (my terminal Jupyter notebook editor) also supports Jupytext
cycomanic 3y ago
This was the first thing I wanted to post when reading the article. Jupytext is excellent, although i typically use MyST (an extended Markdown syntax).

milliams 3y ago

There is a draft JEP (Jupyter Enhancement Proposal) for Markdown-based notebooks (https://github.com/jupyter/enhancement-proposals/pull/103 https://github.com/jupyter/enhancement-proposals/pull/103) which will make it a little more RMarkdown-like.

wdroz 3y ago

If you don't need to "commit" the output, you can just use nbconvert[0]: jupyter nbconvert --clear-output --inplace my_notebook.ipynb So you can use git as usual, like for code. [0] -- https://nbconvert.readthedocs.io/en/latest/ https://nbconvert.readthedocs.io/en/latest/

andrecosta 3y ago
nbstripout[0] does that and installs a pre-commit hook [0] -- https://github.com/kynan/nbstripout https://github.com/kynan/nbstripout

sashk 3y ago

You don't need to commit output into the git. I used pre-commit filter in git, where it will strip all output from the notebook before it was committed into repository. This allowed us to review the code changes of notebooks.

joelschw 3y ago

The native GitHub feature in preview will make this a lot better for those able to use it https://github.blog/changelog/2023-03-01-feature-preview-rich-jupyter-notebook-diffs/ https://github.blog/changelog/2023-03-01-feature-preview-ric...

SalsaCrotch 3y ago
This feature has resolved the problem for our team.

sdfghswe 3y ago

I haven't read the link and I'm not going to. I realized that jupyter notebooks are a flawed idea when I've tried vs code. vs code uses jupyter-the-protocol (as opposed to jupyter-the-notebooks) in order to give you a notebook-like experience that doesn't involve the jupyter notebook file format. VS code's interactive files are valid python code. To me that killed jupyter notebooks. Why use something that is strictly worse in every respect?

kortex 3y ago
It sounds like you are using the tool wrong. Jupyter notebooks are strictly superior to anything else (namely: code only, spreadsheets, matlab/octave) at their primary use case, which is interactive data science (writing code to manipulate some data, while actively revising the code, or sharing the results of that code with others). Nothing even comes close. There's a reason it's dominant in the data science field. Your workflow works for you but the jupyter workflow works for millions of students, data scientists, and even developers. Heck I even know all the ways to avoid jupyter, and I still use it often, because it's so convenient.
- kevinskii 3y ago
  I agree with the OP. VS Code using the Jupyter protocol is superior to notebooks in almost every respect in my experience. It gives you an excellent debugger, the ability to track changes in Git without any modification, and you can also run as a regular Python script.
- sdfghswe 3y ago
  > It sounds like you are using the tool wrong. Jupyter notebooks are strictly superior to anything else (namely: code only, spreadsheets, matlab/octave) at their primary use case, which is interactive data science (writing code to manipulate some data, while actively revising the code, or sharing the results of that code with others). > Nothing even comes close. There's a reason it's dominant in the data science field. > Your workflow works for you but the jupyter workflow works for millions of students, data scientists, and even developers. Heck I even know all the ways to avoid jupyter, and I still use it often, because it's so convenient. Copy pasting your comment here so when you eventually delete it people can still see the ignorance. You have absolutely no clue what you're talking about. Worse, it seems like you didn't read what you're responding to.

kortex 3y ago

Wow, no mention of DVC (http://www.dvc.org http://www.dvc.org)? That has been invaluable for data scientist workflows. I definitely do like to strip notebooks and make them run-idempotent to the best of my ability, but sometimes you just need stateful notebooks. And since .ipynb are technically json but in reality act more like a binary file format (with respect to diffing), DVC is the ideal tool to store them. Don't get me started on git annex or LFS, both of those took years off my life due to stress of using them and them bugging out. Also I am hardly a fan of XML, but does anyone feel like notebook files would have been a near-ideal use-case of it? It's literally a collection of markup. The fact that json was chosen over xml I think is somewhat damning of xml as an application data storage format. I think xml is perfectly cromulent as a write-once-read-many presentation format or rendering target (html, svg, GeniCam api info), but it seems to flounder in virtually every other domain it's been shoehorned into, with the exception of office application formats. Actually, downthread there is a link to a jupyer enhancement proposal for a .nb.md markdown based format. I think this is great. One theme I keep coming across in my computer science journey is that formats which have mandatory closing endcaps are kind of a PITA. It seems the stream-of-containers (with state machines as needed) is all-around better. JSON-LD is better than JSON, streaming video formats are better than ones that stick metadata at the end, zip is... an eldritch horror, etc.

nvy 3y ago

Seems to me that this article does a great job explaining why jupyter notebooks are a poor collaboration tool. I wish that non-emacs implementations of org were more commonplace, as it's a pretty sane markup language and supports embedded code and graphics, diffs nicely, and doesn't introduce the insanity of JSON.

TeeWEE 3y ago

My quick solution is to not commit the result cells, only the commands. So its just code.

DryLabRebel 3y ago

You forgot another issue: - containing potentially sensitive data in your notebook