6 ms·

The Jupyter+Git problem is now solved (2022)

107 points by skadamat 3y ago

skadamat 3y ago
nbdev has been a god send and I'm really enjoying using it. Great to see fast.ai investing more into making notebooks even more usable
kingds 3y ago
seems very odd. the issue was that when Git marked lines in a file as being in conflict then the file was no longer a valid Jupyter file. and their solution was... to change the language so that Git's conflict syntax is valid?
- skadamat 3y ago
  yeah it's definitely some classic engineering around the problem, but hard to fix Github I guess. At a previous company, we switched to Gitlab for the specific reason that they handled notebook diffs slightly better. It's a struggle out here
  - Animux 3y ago
    Note that the git conflict markers have nothing to do with Github but are generated by git itself (the commmand line client running locally on your machine).
- CJefferson 3y ago
  Yes, I think you are right. Seems like quite a good plan to me. Nothing else obvious jumps to mind, as anyone using git is going to end up with git conflict markers in their files at some point.
- jph00 3y ago
  No the answer was simply that git doesn’t come with a merge driver for json, only one for line oriented text. So nbdev provides one for ipynb json documents.
nearting 3y ago
Cool to see that this is moving along - Jupyter merge conflicts have caused me a huge amount of headache over the years. My solution has been to switch over to Quarto notebooks (mentioned in the post with Jupytext), but I see the issue around saving cell outputs. I'm curious why one would specifically want to save cell outputs as is in the Jupyter notebook, rather than archiving that in some other format. Sure, that might require putting a lot of information in one page (e.g., if that output is dependent on many other code cells and their outputs), but that just moves the linkage problem around - you'd have to have some way of indicating that the specific cell output was generated by a specific version of cell code (and the order in which they were run, sometimes multiple times).
- skadamat 3y ago
  > I'm curious why one would specifically want to save cell outputs as is in the Jupyter notebook My blind guess is that it improves the readability of the notebook / promotes the literate programming mindset. But just a guess
- twelfthnight 3y ago
  Strongly agree with this. IIRC, in RMarkdown state is treated as a separate cache stored outside the notebook and loaded as needed. You could use something like dvc or gitlfs to manage those cache files, and since the Markdown file is plain text, use regular git to inspect changes to the notebook implementation. I feel like Jupyter notebooks are the PDFs of data science. They are super useful for displaying results, but bake that data in a super inconvenient way for doing anything but rendering the data to look nice.
- taeric 3y ago
  You want to share the outputs so that you can share the results while showing your work. That jupyter also inlines everything, such that even charts are stored in the document, makes that even more necessary. Though, I can see your point, I think. Why not include a build step that moves from your document to the generated output? My gut there is a large part of why the system got popular is that they worked hard on removing the friction that that would add. As a comparison and to your point, I've seen people try to build "literate test suites" that were in a notebook, very happy with how the output looked. Only to find later that if they had used some of the more common test frameworks, those already create very nice reports. And moving the report format/creation out of the specification allowed a ton of flexibility.
taeric 3y ago
I like json for data exchange, but I will never be able to endorse the notebook is a json document idea. It is the poster child for why you would want a document markup language, and they said "why not encode it in a textual data language instead?" Yes, even in a document model, merge conflicts can give you invalid documents. Programmers deal with this every day when they create invalid programs. Trying to hide the document into data complicates that in ways that are obvious in hindsight, and not that surprising with foresight.
- pphysch 3y ago
  I suspect that .ipynb will eventually be displaced by good old HTML documents loading PyScript (or more specifically, a notebook framework built on PyScript).
  - deleted 3y ago
    [deleted]
- mistertoolbox 3y ago
  Totally agree with this. I like the Markdown subset approach LiveBook (https://livebook.dev/ https://livebook.dev/) has taken to play nicely with version control in comparison.
softwaredoug 3y ago
Ok but I’m still not sold on the whole NBDev ecosystem. Is it worth building up lost of custom infra just so we can build in notebooks? Why should DS software dev be in a different special set of tools distinct from the other software it interacts with? And a host of other reasons https://gist.github.com/softwaredoug/d527a18643f29832b0f41af309f9ab98 https://gist.github.com/softwaredoug/d527a18643f29832b0f41af...
- tsss 3y ago
  Why shouldn't they use nbdev if it works for them? Plain Git obviously does not work and not only for data scientists. Git, with its inability to deal with non-textual artifacts, is completely unsuited to a modern software development workflow. Modern software development contains interactive graphs, audio and video assets, editable diagrams, vast amounts of binary data for reproduction of experiments, etc. etc. So much custom tooling has been built to somehow shoehorn this stuff into git and it never works well. I'm sick of diagramming software and IaC tools with horrible user experience just to keep a plaintext representation that no human dares to read anyway.
  - softwaredoug 3y ago
    I'm not against notebooks. I'm against the idea of doing all of your software development in notebooks. There's a sane way to use notebooks. For me, nbdev is a step too far, as it really pushes notebooks as the primary dev artifact. (All my opinions, people can do whatever they want)
    - taeric 3y ago
      I am assuming you've seen this? https://www.youtube.com/watch?v=7jiPeIFXb6U https://www.youtube.com/watch?v=7jiPeIFXb6U :D
      - softwaredoug 3y ago
        Yes I'm a big fan :) I also like the "Why I like notebooks" counterpoint talk. I'm just not personally convinced. While they make some great arguments about where notebooks can be really powerful for software development, I don't think the speaker makes entirely valid counterpoints to the original "I don't like notebooks" talk and most of the problems of notebooks still stand. See this gist I posted above https://gist.github.com/softwaredoug/d527a18643f29832b0f41af309f9ab98 https://gist.github.com/softwaredoug/d527a18643f29832b0f41af...
sdfghswe 3y ago
Jupyter+Git has no problem. The problem is Jupyter notebooks+Git. The solution is use Jupyter, but not Jupyter notebooks.
- SpriglyElixir12 3y ago
  What is the difference between the two? Googling it shows some comparisons between jupyter lab vs notebook but I can't find anything on plain jupyter.
  - sdfghswe 3y ago
    Jupyter is the protocol to communicate between a frontend and a backend. Jupyter notebooks is an interactive notebook that implements that protocol. Do you know what also implements the protocol? VS Code. And doesn't have any of the stupid problems that the Jupyter notebooks do.
    - Noumenon72 3y ago
      Are you saying that VS Code also lets you do all of the "Run just this cell" that notebooks do?
      - sdfghswe 3y ago
        Yes. In VS Code if you do as below: #%% print("my python print") #%% Then you can run it as a cell. Crucially however, note that those delimiters are ALSO valid python code. So you just version as with everything else.
- jph00 3y ago
  There isn’t a problem any more with notebooks and git because nbdev now provides a merge driver. So it’s fine to use notebooks with git now.
- esafak 3y ago
  Thanks, but this is a bit pedantic. For most people Jupyter is the notebook. You could just as well call the other one the "Jupyter protocol".
amirathi 3y ago
We all know .ipynb JSON format is not a great fit for Git. The Jupyter ecosystem has come a long way in the last few years. Solving this really comes down to a few tools - - JupyterLab Git Extension[1] for local diffs (pre-commit diffs) - nbdime[2] / nbdev[3] for resolving .ipynb git merge conflicts - GitHub PR code reviews with ReviewNB[4] - Alternatively, if you don't care about cell outputs then Jupytext[5] to sync .ipynb JSON to markdown Disclaimer: I built ReviewNB. It's a completely bootstrapped business, 5 years in the making and now used by leading DS teams at Meta, AWS, NASA JPL, AirBnB, Lyft, Affirm, AMD, Microsoft & more[6] for Jupyter Notebook code reviews on GitHub / Bitbucket. [1] https://github.com/jupyterlab/jupyterlab-git https://github.com/jupyterlab/jupyterlab-git [2] https://nbdime.readthedocs.io https://nbdime.readthedocs.io [3] https://nbdev.fast.ai https://nbdev.fast.ai [4] https://www.reviewnb.com https://www.reviewnb.com [5] https://github.com/mwouts/jupytext https://github.com/mwouts/jupytext [6] https://www.reviewnb.com/#customers https://www.reviewnb.com/#customers
- enriquto 3y ago
  > Alternatively, if you don't care about cell outputs then Jupytext[5] to sync .ipynb JSON to markdown Notice that using markdown is a possibility for jupytext, but not the only one. More interestingly, you can also store your notebooks as plain python files, whose comments are interpreted as the markdown cells of the notebook. This is very useful, and not only for version control: if your notebooks are python files they can be executed easily in CI or by third parties just by launching the interpreter. No need even of the jupyterlab dependency. With some care, you can craft a single python file "foo.py" that can be used at the same time as 1. an executable command-line program (that happens to be written in python) 2. an importable python module 3. a jupyter notebook (to open it you need the jupytext extension of jupyter) 4. the documentation with auto-generated figures, convertible to html or to pdf using "jupyter nbconvert --execute" 5. a regular .ipynb if for some reason you want to distribute the outputs in a re-executable format For small simple projects, to showcase, describe and illustrate an independent algorithm, we have found this structure invaluable.
  - kzrdude 3y ago
    And VS Code supports the py-percent format as a notebook too (that jupytext can use)
brasic 3y ago
Glad to see a custom merge driver being used here - they’re one of the most powerful of git’s obscure features. Large teams working on a monorepo inevitably start noticing that particular files are magnets for conflicts (or other times, in cases like this, some files are a huge pain to resolve whenever they conflict). This happens especially frequently if your team uses a lot of CIGARs (checked in generated artifacts) In most cases writing a simple driver to automatically handle the conflict resolution is pretty straightforward (especially if the resolution is usually just to regenerate a generated file) and well worth the up front effort to eliminate ongoing conflict headaches. https://git-scm.com/docs/gitattributes#_defining_a_custom_merge_driver https://git-scm.com/docs/gitattributes#_defining_a_custom_me... The main hassle is that for security reasons all developers need to opt in by registering the merge driver , which you can put in a project bootstrap script if you have one. Would be great if GitHub (disclaimer: where I used to work) would integrate custom merge drivers in their internal conflict resolution flow.
izzymiller 3y ago
just a data point: Hex we built a .yaml based import/export for our notebooks, more than partially to make it friendlier to work with git (also partially because as we added new features it became harder to express custom cell types in .ipynb). We still support ipynb import/export, but using yaml for our internal representation of notebooks has made it hugely easier to do human-readable diffs and makes git operations way easier. (https://hex.tech/blog/github-sync/ https://hex.tech/blog/github-sync/)
jmount 3y ago
I think a lot of us have our own solutions to Jupyter Workbooks in Git. My personal one is this: https://win-vector.com/2022/08/20/an-effective-personal-jupyter-data-science-workflow/ https://win-vector.com/2022/08/20/an-effective-personal-jupy... (which itself is sub-tooling for running lots of copies of a parameterized notebook, all easy as so much of nbconvert and Jupyter are exposed as APIs).
deleted 3y ago
[deleted]
aldanor 3y ago
Worth noting that GitHub has been rendering ipynb diffs for a while now: https://github.blog/changelog/2023-03-01-feature-preview-rich-jupyter-notebook-diffs/ https://github.blog/changelog/2023-03-01-feature-preview-ric...

107 points by skadamat 3y ago

nbdev has been a god send and I'm really enjoying using it. Great to see fast.ai investing more into making notebooks even more usable

kingds 3y ago

seems very odd. the issue was that when Git marked lines in a file as being in conflict then the file was no longer a valid Jupyter file. and their solution was... to change the language so that Git's conflict syntax is valid?

skadamat 3y ago
yeah it's definitely some classic engineering around the problem, but hard to fix Github I guess. At a previous company, we switched to Gitlab for the specific reason that they handled notebook diffs slightly better. It's a struggle out here
- Animux 3y ago
  Note that the git conflict markers have nothing to do with Github but are generated by git itself (the commmand line client running locally on your machine).
CJefferson 3y ago
Yes, I think you are right. Seems like quite a good plan to me. Nothing else obvious jumps to mind, as anyone using git is going to end up with git conflict markers in their files at some point.
jph00 3y ago
No the answer was simply that git doesn’t come with a merge driver for json, only one for line oriented text. So nbdev provides one for ipynb json documents.

nearting 3y ago

Cool to see that this is moving along - Jupyter merge conflicts have caused me a huge amount of headache over the years. My solution has been to switch over to Quarto notebooks (mentioned in the post with Jupytext), but I see the issue around saving cell outputs. I'm curious why one would specifically want to save cell outputs as is in the Jupyter notebook, rather than archiving that in some other format. Sure, that might require putting a lot of information in one page (e.g., if that output is dependent on many other code cells and their outputs), but that just moves the linkage problem around - you'd have to have some way of indicating that the specific cell output was generated by a specific version of cell code (and the order in which they were run, sometimes multiple times).

skadamat 3y ago
> I'm curious why one would specifically want to save cell outputs as is in the Jupyter notebook My blind guess is that it improves the readability of the notebook / promotes the literate programming mindset. But just a guess
twelfthnight 3y ago
Strongly agree with this. IIRC, in RMarkdown state is treated as a separate cache stored outside the notebook and loaded as needed. You could use something like dvc or gitlfs to manage those cache files, and since the Markdown file is plain text, use regular git to inspect changes to the notebook implementation. I feel like Jupyter notebooks are the PDFs of data science. They are super useful for displaying results, but bake that data in a super inconvenient way for doing anything but rendering the data to look nice.
taeric 3y ago
You want to share the outputs so that you can share the results while showing your work. That jupyter also inlines everything, such that even charts are stored in the document, makes that even more necessary. Though, I can see your point, I think. Why not include a build step that moves from your document to the generated output? My gut there is a large part of why the system got popular is that they worked hard on removing the friction that that would add. As a comparison and to your point, I've seen people try to build "literate test suites" that were in a notebook, very happy with how the output looked. Only to find later that if they had used some of the more common test frameworks, those already create very nice reports. And moving the report format/creation out of the specification allowed a ton of flexibility.

taeric 3y ago

I like json for data exchange, but I will never be able to endorse the notebook is a json document idea. It is the poster child for why you would want a document markup language, and they said "why not encode it in a textual data language instead?" Yes, even in a document model, merge conflicts can give you invalid documents. Programmers deal with this every day when they create invalid programs. Trying to hide the document into data complicates that in ways that are obvious in hindsight, and not that surprising with foresight.

pphysch 3y ago
I suspect that .ipynb will eventually be displaced by good old HTML documents loading PyScript (or more specifically, a notebook framework built on PyScript).
- deleted 3y ago
  [deleted]
mistertoolbox 3y ago
Totally agree with this. I like the Markdown subset approach LiveBook (https://livebook.dev/ https://livebook.dev/) has taken to play nicely with version control in comparison.

softwaredoug 3y ago

Ok but I’m still not sold on the whole NBDev ecosystem. Is it worth building up lost of custom infra just so we can build in notebooks? Why should DS software dev be in a different special set of tools distinct from the other software it interacts with? And a host of other reasons https://gist.github.com/softwaredoug/d527a18643f29832b0f41af309f9ab98 https://gist.github.com/softwaredoug/d527a18643f29832b0f41af...

tsss 3y ago
Why shouldn't they use nbdev if it works for them? Plain Git obviously does not work and not only for data scientists. Git, with its inability to deal with non-textual artifacts, is completely unsuited to a modern software development workflow. Modern software development contains interactive graphs, audio and video assets, editable diagrams, vast amounts of binary data for reproduction of experiments, etc. etc. So much custom tooling has been built to somehow shoehorn this stuff into git and it never works well. I'm sick of diagramming software and IaC tools with horrible user experience just to keep a plaintext representation that no human dares to read anyway.
- softwaredoug 3y ago
  I'm not against notebooks. I'm against the idea of doing all of your software development in notebooks. There's a sane way to use notebooks. For me, nbdev is a step too far, as it really pushes notebooks as the primary dev artifact. (All my opinions, people can do whatever they want)
  - taeric 3y ago
    I am assuming you've seen this? https://www.youtube.com/watch?v=7jiPeIFXb6U https://www.youtube.com/watch?v=7jiPeIFXb6U :D
    - softwaredoug 3y ago
      Yes I'm a big fan :) I also like the "Why I like notebooks" counterpoint talk. I'm just not personally convinced. While they make some great arguments about where notebooks can be really powerful for software development, I don't think the speaker makes entirely valid counterpoints to the original "I don't like notebooks" talk and most of the problems of notebooks still stand. See this gist I posted above https://gist.github.com/softwaredoug/d527a18643f29832b0f41af309f9ab98 https://gist.github.com/softwaredoug/d527a18643f29832b0f41af...

sdfghswe 3y ago

Jupyter+Git has no problem. The problem is Jupyter notebooks+Git. The solution is use Jupyter, but not Jupyter notebooks.

SpriglyElixir12 3y ago
What is the difference between the two? Googling it shows some comparisons between jupyter lab vs notebook but I can't find anything on plain jupyter.
- sdfghswe 3y ago
  Jupyter is the protocol to communicate between a frontend and a backend. Jupyter notebooks is an interactive notebook that implements that protocol. Do you know what also implements the protocol? VS Code. And doesn't have any of the stupid problems that the Jupyter notebooks do.
  - Noumenon72 3y ago
    Are you saying that VS Code also lets you do all of the "Run just this cell" that notebooks do?
    - sdfghswe 3y ago
      Yes. In VS Code if you do as below: #%% print("my python print") #%% Then you can run it as a cell. Crucially however, note that those delimiters are ALSO valid python code. So you just version as with everything else.
jph00 3y ago
There isn’t a problem any more with notebooks and git because nbdev now provides a merge driver. So it’s fine to use notebooks with git now.
esafak 3y ago
Thanks, but this is a bit pedantic. For most people Jupyter is the notebook. You could just as well call the other one the "Jupyter protocol".

amirathi 3y ago

We all know .ipynb JSON format is not a great fit for Git. The Jupyter ecosystem has come a long way in the last few years. Solving this really comes down to a few tools - - JupyterLab Git Extension[1] for local diffs (pre-commit diffs) - nbdime[2] / nbdev[3] for resolving .ipynb git merge conflicts - GitHub PR code reviews with ReviewNB[4] - Alternatively, if you don't care about cell outputs then Jupytext[5] to sync .ipynb JSON to markdown Disclaimer: I built ReviewNB. It's a completely bootstrapped business, 5 years in the making and now used by leading DS teams at Meta, AWS, NASA JPL, AirBnB, Lyft, Affirm, AMD, Microsoft & more[6] for Jupyter Notebook code reviews on GitHub / Bitbucket. [1] https://github.com/jupyterlab/jupyterlab-git https://github.com/jupyterlab/jupyterlab-git [2] https://nbdime.readthedocs.io https://nbdime.readthedocs.io [3] https://nbdev.fast.ai https://nbdev.fast.ai [4] https://www.reviewnb.com https://www.reviewnb.com [5] https://github.com/mwouts/jupytext https://github.com/mwouts/jupytext [6] https://www.reviewnb.com/#customers https://www.reviewnb.com/#customers

enriquto 3y ago
> Alternatively, if you don't care about cell outputs then Jupytext[5] to sync .ipynb JSON to markdown Notice that using markdown is a possibility for jupytext, but not the only one. More interestingly, you can also store your notebooks as plain python files, whose comments are interpreted as the markdown cells of the notebook. This is very useful, and not only for version control: if your notebooks are python files they can be executed easily in CI or by third parties just by launching the interpreter. No need even of the jupyterlab dependency. With some care, you can craft a single python file "foo.py" that can be used at the same time as 1. an executable command-line program (that happens to be written in python) 2. an importable python module 3. a jupyter notebook (to open it you need the jupytext extension of jupyter) 4. the documentation with auto-generated figures, convertible to html or to pdf using "jupyter nbconvert --execute" 5. a regular .ipynb if for some reason you want to distribute the outputs in a re-executable format For small simple projects, to showcase, describe and illustrate an independent algorithm, we have found this structure invaluable.
- kzrdude 3y ago
  And VS Code supports the py-percent format as a notebook too (that jupytext can use)

brasic 3y ago

Glad to see a custom merge driver being used here - they’re one of the most powerful of git’s obscure features. Large teams working on a monorepo inevitably start noticing that particular files are magnets for conflicts (or other times, in cases like this, some files are a huge pain to resolve whenever they conflict). This happens especially frequently if your team uses a lot of CIGARs (checked in generated artifacts) In most cases writing a simple driver to automatically handle the conflict resolution is pretty straightforward (especially if the resolution is usually just to regenerate a generated file) and well worth the up front effort to eliminate ongoing conflict headaches. https://git-scm.com/docs/gitattributes#_defining_a_custom_merge_driver https://git-scm.com/docs/gitattributes#_defining_a_custom_me... The main hassle is that for security reasons all developers need to opt in by registering the merge driver , which you can put in a project bootstrap script if you have one. Would be great if GitHub (disclaimer: where I used to work) would integrate custom merge drivers in their internal conflict resolution flow.

izzymiller 3y ago

just a data point: Hex we built a .yaml based import/export for our notebooks, more than partially to make it friendlier to work with git (also partially because as we added new features it became harder to express custom cell types in .ipynb). We still support ipynb import/export, but using yaml for our internal representation of notebooks has made it hugely easier to do human-readable diffs and makes git operations way easier. (https://hex.tech/blog/github-sync/ https://hex.tech/blog/github-sync/)

jmount 3y ago

I think a lot of us have our own solutions to Jupyter Workbooks in Git. My personal one is this: https://win-vector.com/2022/08/20/an-effective-personal-jupyter-data-science-workflow/ https://win-vector.com/2022/08/20/an-effective-personal-jupy... (which itself is sub-tooling for running lots of copies of a parameterized notebook, all easy as so much of nbconvert and Jupyter are exposed as APIs).

deleted 3y ago

[deleted]

aldanor 3y ago

Worth noting that GitHub has been rendering ipynb diffs for a while now: https://github.blog/changelog/2023-03-01-feature-preview-rich-jupyter-notebook-diffs/ https://github.blog/changelog/2023-03-01-feature-preview-ric...