14 ms·

The Jupyter+Git problem is now solved

301 points by jph00 4y ago

wasimlorgat 4y ago
Hi, I'm the author of the git merge driver and Jupyter save hook in nbdev2 :) I'd be happy to answer any questions you have about how we're handling using notebooks with git
- howon92 4y ago
  I enjoyed reading the writeup and think the solution is clean! Thanks for sharing
- jks 4y ago
  Can this do three-way merge? If I have to resolve two conflicting code blocks, it is often useful to know how each of them change the code from the shared parent.
  - wasimlorgat 4y ago
    It does an ordinary three-way git merge (treating notebooks as plaintext) then a two-way merge on conflicted bits. We opted for that approach because its incredibly simple and has worked perfectly for us (I think since we tend to work with small code cells). I think nbdime has a full-on three-way notebook merge if that's what you need, which can be used together with nbdev's Jupyter save hook to clan up unneeded metadata.
scombridae 4y ago
Subversion used to say, 'CVS done right.' With that slogan there is nowhere you can go. There is no way to do CVS right. -- Linus T. Jupyter's ipynb format is only slightly more amenable to git than say an MSWord doc. Nbdime and friends will never get you to a point where git+jupyter will be worth the ugly.
- jph00 4y ago
  What are the outstanding problems you feel are there even with the new nbdev2 functionality? Since I've been using it (the prerelease version) over the last few months I haven't come across a single problem, personally, despite doing a very large amount of collaborative notebook work.
- fragmede 4y ago
  slightly. Since, like, Office 2007, MSWord docs are zip files with xml inside.
whacked_new 4y ago
I used to use Jupytext a lot for this problem, and think it does a decent job. The main problem with Jupytext is its reliance on BYOD (D for discipline), which is a poor (but possibly best available) solution for human systems. IMHO the Jupyter+Git problem stems from the ipynb format. Jupytext does it "right" in the sense that you can work in .ipynb, and diff in .md. But as long as the base format is diff-unfriendly, all tools are methods of indirection of format complexity to tool complexity. That's not to take away from the tool -- it looks great. It also takes the D out of BYOD, which is a win. But I think "solving" it means that anybody who receives an ipynb is able to just look at it out of the box, like plain text, so we're still a ways off.
- wasimlorgat 4y ago
  I agree re format vs tool complexity. I don't think Jupyter is a particularly difficult format though, its mostly light JSON -- all human-readable. We realised after working with Jupyter+Git for a while that the pain-points were actually with Jupyter editors (and/or their conventions) rather than the format, because they do things like store user-metadata in the file which pollutes diffs and leads to merge conflicts. In fact, if Jupyter editors could handle merge conflicted files, we wouldn't need a custom merge driver either.
- wasimlorgat 4y ago
  Oh, on the topic of file formats: Quarto also lets you do plaintext notebooks in quite an interesting way, definitely worth checking out: https://quarto.org/docs/computations/python.html https://quarto.org/docs/computations/python.html
  - euler_angles 4y ago
    The latest release of nbdev has fully embraced Quarto! It's very awesome, check it out.
- cycomanic 4y ago
  I have been using jupytext as well and it really makes handling notebooks much easier. I think the decision for Jupyter to save to json was not a good one and they should instead have looked at systems like org mode for inspiration. I also don't understand what you mean by discipline. Yes you need to make sure that everyone has the jupytext extension installed, but that just becomes part of the needed dev environment. After that the whole experience becomes completely seemless.
Helmut10001 4y ago
Not criticizing the authors approach, but the Jupyter+Git Problem was solved for a long time with Jupytext [1]. Jupytext will convert Notebooks (.ipynb) files to Markdown (md) and Python (py) 'on the fly' (while working in Notebooks). - Markdown files can be added to git - Python and .ipynb files are added to .gitignore - Python files allow 'chained' import of notebooks (*.py verions), which allows to split larger notebooks into multiple smaller ones This is my folder structure: . ├── notebooks | ├── notebook1.ipynb # automatically generated from md | └── notebook2.ipynb # automatically generated from md ├── md | ├── notebook1.md # versioned in git | └── notebook2.md # versioned in git ├── py | ├──modules | | ├──__init__.py # empty | | └──tools.py # use for cross-project base tools | ├──__init__.py # empty | ├── notebook1.py # automatically generated from md | └── notebook2.py # automatically generated from md ├──jupytext.toml ├──.git └── README.md See an example here [2] Jupytext is mentioned as a 'potential' alternative. Re the "save" cell output: I usually produce html-files at the end of my notebooks (see the example), and add those either to git or auto-upload to an external webserver. The html is standalone and includes outputs, table of contents, and images (example [3]). I would advice against versioning all outputs (images) in git. Very happy with this approach for a long time now. Jupytext increased my productivity by a hundred percent. [1]: https://github.com/mwouts/jupytext https://github.com/mwouts/jupytext [2]: https://gitlab.vgiscience.de/ad/yfcc_gridagg https://gitlab.vgiscience.de/ad/yfcc_gridagg [3]: https://ad.vgiscience.org/tagmaps-mapnik-jupyter/01_mapnik-tagmaps.html https://ad.vgiscience.org/tagmaps-mapnik-jupyter/01_mapnik-t...
- jph00 4y ago
  The pros and cons of Jupytext are discussed in the linked post. It's a great approach, but wasn't sufficient for our needs -- so for us, at least, it didn't fully solve the Jupyter+git problem. Specifically, it doesn't handle the situation where you need cell outputs in version control -- since in that case, you still need the notebook, which results in all the usual problems occuring. With nbdev2, you don't need to think about anything or do anything special, and stuff like GitHub notebook rendering, nbviewer, ReviewNB, etc all just work. You just run a single command (`nb_install_hooks`) and that's it. Also, no-one has to install anything extra to view your notebooks, since they're stored in the regular notebook format.
jayd16 4y ago
You know, is there anything like the Language Server Protocol for diff/merge resolution? Seems like there's an opportunity to build a system for semantic aware merge that's language/format aware and tool agnostic (and auto configurable to boot). Suddenly binary formats could become mergeable.
- zie 4y ago
  See [Pijul](https://pijul.org/manual/theory.html https://pijul.org/manual/theory.html). They did that hard work.
  - samatman 4y ago
    Hmm I'm a huge fan of pijul, looks like the future of change management from where I'm sitting, but no: they have not. Semantic diffing needs something like pijul, but a system taking advantage of this doesn't yet exist. Pijul avoids some merge conflicts by design, won't do the wrong thing, and handles conflicts correctly: we still need tools with a fuller awareness of what strings mean to have rich semantic diffs.
    - zie 4y ago
      True, Pijul only offers the safe diff/patch part.
xcambar 4y ago
Stating that git breaks Jupyter notebooks is quite a flex. It stains the article from the very first paragraph.
- deleted 4y ago
  [deleted]
- wasimlorgat 4y ago
  Have you worked with Jupyter notebooks and git? It's a literally true statement :D and quite a struggle for many of us
  - cycomanic 4y ago
    It's the wrong way around though, Jupyter notebooks break a git work flow. I think the fault here is completely with the design of the Jupyter notebook file format (and the way editors save to it). I think it's quite unfortunate that they did not consider that the format would integrate well with version control systems when first designing ipython notebooks.
    - fumeux_fume 4y ago
      Nah man, you got it backwards. Git still works just fine while my notebooks are definitely broken. Not here to play the blame game, just trying to relate the practical results.
  - xcambar 4y ago
    If you leave git diffs in your files, whether Jupyter notebooks or otherwise, and run/compile them... They will break. If you give me a counter example, good for you, but my statement holds true 99%.
    - jsweojtj 4y ago
      You state in the top level comment that this claim stains the article: "Stating that git breaks Jupyter notebooks is quite a flex." But you are saying here: "If you leave git diffs in your files, whether Jupyter notebooks or otherwise, and run/compile them... They will break." Have you changed your mind in this thread? Or what's your objection?
p1necone 4y ago
I haven't used Jupyter but from what I can gather from this article they've built a simultaneous editing system on top of automatically committing to git in the background as multiple people edit things, and using that to share the changes between users. Do I have that right? Because that sounds /insane/.
- jph00 4y ago
  No it just uses normal git in the normal way. The simple trick is to use a jupyter-native git merge driver, so that merges are done at a cell level instead of a line level. Also, unneeded metadata is removed from the notebook when saving, so there's less changes to merge. Both these two things are done using standard hooks built into each of git and Jupyter. That is: git is written in such a way that it can fully support non line-oriented formats. We just took advantage of that capability.
  - p1necone 4y ago
    Ahh right, so you still make manual git commits normally, it's just that the Jupyter UI used to fall over when it encountered merge conflict markers in source files. And now it doesn't fall over any more and can nicely represent them because the conflict markers are no longer done for individual lines of text?
    - jph00 4y ago
      Yes exactly :D
- anigbrowl 4y ago
  If this doesn't work for you JetBrains' DataSpell might, it's oriented towards notebooks for teams. It has hiccups, things like ipywidgets don't always work as expected so I sometimes find myself falling back to Jupyterlab. But overall it's a very comfy chair.
cigrainger 4y ago
This is one reason I feel lucky to be working with Elixir. Livebook's livemd is basically just markdown. https://livebook.dev https://livebook.dev
- wodenokoto 4y ago
  So is jupytext, rmd and qmd. But what do you do about the output? The nice thing about markdown-like notebooks is that they play well with git. The nice thing about jupyter style notebooks is that they contain all the content needed to actually _read_ the notebook.
HuwFulcher 4y ago
Whenever I can I strongly recommend not using Jupyter for anything more than the most transient tasks. I don’t know whether it’s the Data Science culture or Jupyter but there is a big lack of discipline in writing maintainable code in DS and non-existent git support is part of that. I always strongly discouraged developing models using notebooks, instead advocating for using .py files and then using notebooks for sanity checking data. I don’t have any clever ideas for how we can move past Jupyter but the sooner we do the better.
- maegul 4y ago
  Yes, the Data Science culture around maintainable code does seem to be reaching a critical level of toxicity (in some environments at least). In line with a nephew comment of mine, I feel that bringing the immediate interactivity or iteration cycle of notebooks to the development experience would help a lot, and not be too bad a thing for common development either. I've heard of the related nbdev project, which seems like an interesting and compelling idea. But it'd be nice to see the reverse: something that makes ordinary python development more immediate than using a debugger/vanilla REPL.
  - Leo_Germond 4y ago
    I think that improving the shell experience and allowing e.g. multimedia content to be displayed and manipulated directly into the shell, would help a lot with interactivity. Maybe some specific terminal emulator (like kitty) with ipython would constitute a good starting point...
- tetris11 4y ago
  It depends how you use it. When you're still new to the data, using a freeflow text-and-codeblock workflow like jupyter or org-mode really speeds up the exploration phase. Once you have a consistent set of questions, and methods to answer them, then yes, copy off the relevant chunks into their own scripts and source these when using similar data to bring you up to speed, and modify them to your tastes. The issue with starting off with an external script initially is the distracting temptation to refine your code so it can be better used with future data, despite not yet having seen or not knowing what that future data is like. The initial "play and explore" phase of an analysis is very important imo, and notebooks really facilitate that.
medo-bear 4y ago
or ... why not just use org mode?
- faustlast 4y ago
  I wish org-mode was standard and more appreciated. It is so good, but I feel that I'm the only one using it and it is hard to sell emacs to others.
bsdz 4y ago
I use this plugin for my jupyter notebook git integration. It has a git diff option that's useful but gets very slow for complex documents. Perhaps under the hood it's using one of the other tools mentioned in the postscript. https://github.com/jupyterlab/jupyterlab-git https://github.com/jupyterlab/jupyterlab-git Edit: Looking at the source, it does appear to use nbdime under the hood.
flutetornado 4y ago
JupyterLab JupyterHub Jupytext - converts ipynb to py Nbstripout - strips all output from ipynb Nbmerge - resolves merge conflicts Vim-jupytext - vim plugin to auto convert ipynb to py Papermill - parameterize notebooks Git Pandas, Altair - data analysis / Visualization Phabricator - code reviews of notebooks Vimdiff + vim-jupytext - diffs in terminal This solved all my jupyter problems.
hendry 4y ago
If there a good place to see Jupyter note books solving a real problem? Idk, like importing some data and doing some analysis / forecasting? Most notebooks appear really bad quality. Worse internally. Better off looking at some excel https://github.com/martinshkreli/models https://github.com/martinshkreli/models
throwaway72937 4y ago
Does this work for editing notebooks in VS code? (Unclear to me where the saving hooks reside, and whether you have to edit them through Jupyter labs/notebook) Any issue if the notebooks reside on a remote server?
planede 4y ago
Is it possible to get a "diff3"-like conflict style? That is showing <<< side1 ||| ancestor === side2 >>>
wanderingmind 4y ago
I thought jupytext solved it long ago with percent formatted python file. Since its a python text file you can run automated formating, linting, static type checking and git version diff. What's new is being solved here?
liquids 4y ago
I’ve used this library for a number of projects and it’s a joy to use. I don’t think it’s an understatement to say it’s paradigm shifting – to the extent that once you have your environment set up, you are free to code, think, iterate, deploy and document your projects all at 99% of the speed of thought. There seems to be a lot of discussion in here around the pitfalls of jupyter, and notebooks, and the poor coding practices of data scientists. If you haven’t read the article or used the software I’d like to highlight that all of these (legitimate) complaints are exactly what nbdev2 was created to address, and in my opinion very successfully solves. The way it works is that everything runs off a master notebook, and then with one command: libraries are built, git diffs are magically fixed, tests are run, documentation is automatically created. It doesn’t fundamentally change your workflow in any way, it just abstracts and automates away all of these pain points. There’s a reason that everyone uses jupyter notebooks. They are fun to use, they are great for exploring and developing ideas. And (minus the aforementioned git collaboration issues) they are great for sharing with others, which is a huge part of the wider data science ecosystem. We don’t need to recommend avoiding notebooks, and allege they are just for beginners. We need to use tooling which addresses some of these final issues with writing mature software. And I'd like to thank the authors of nbdev for this. The people who look down their noses at notebooks can continue to do so – but what they will find is that nbdev quite effortlessly leap-frogs over these sneered complaints, and allows you to write better software more productively.
- no_identd 4y ago
  Okay, now gimme a PowerShell version of it.
kmod 4y ago
Streamlit has completely replaced my usage of Jupyter -- I find it to have the quick iteration speed and visual output of notebooks, but it's just normal python so all the normal tooling works (there is no "git problem") and you don't have the weird state problems of notebooks. Definitely recommend checking it out if you haven't already!
da39a3ee 4y ago
> Here at fast.ai we use Jupyter for everything. All our tests, documentation, and module source code for all of our many libraries is entirely developed in notebooks That sounds like a nightmare. Why would you want to develop a library in a jupyter notebook? > The solution presented here is the result of years of work by many people. It's a bit depressing that it came to this. It's hard not to think that it was a mistake from the beginning and that the format should have been based on using special comment markers in valid code, together with an accompanying JSON metadata file. Or something like that. One way or another, we have a very strong tradition of storing code in plain text files, not embedded in strings in JSON or otherwise embedded in any opaque format. Maybe there'll come a day when it's appropriate to abandon that to get some advantages, but I don't think that day was the original creation of Jupyter. I know it was created by thoughtful and expert software engineers, but I feel that it was a mistake and it's actually made a lot of data science / academia-oriented people less qualified to participate in industry software engineering, because of the poor practices forced upon them by the inability to use git with Jupyter, and notions like developing library code in notebook cells.
ticklemyelmo 4y ago
Why is there a "Jupyter+Git" problem specifically? Why aren't we worrying about the "C+Git" problem and the "XML+Git" problem and the "Python+Git" problem? Because merge markers break, well, every file format. Is it because Jupyter users in particular don't typically understand that there is a formatted text file behind the notebook, or how merge conflicts work?
- thefrozenone 4y ago
  This is a good question and made me think. I have come up with: "Jupyter notebooks can be thought of as Code and an IDE layout"
- Bjartr 4y ago
  It's because the primary editor of the notebook files barfs when presented with the file if it includes merge markers since it's no longer valid json. Imagine if one of your normal code-friendly text editors, or your ide, refused to open a .c or .py file and you had to open it in notepad to fix it. That's what it feels like to be forced to drop into a normal text editor rather than using the normal notebook ui to fix the conflicts.
cs702 4y ago
Wow, thank you to the authors! It looks like this tool, nbdev2, solves a real-world problem for Jupyter users, including me, with zero effort required to use it every day. It relies on clever hooks to get git to treat cells as first-class citizens (as opposed to lines of text, the default). Nice! Based on that alone, I would expect nbdev2 to be widely adopted over time. In fact, if it works as well as advertised, it should be incorporated into Jupyter. I, for one, will be giving it a try! If you use Jupyter to solve problems in your domain of expertise, feel free to ignore all the smart-sounding software engineers who will poo-pooh this tool only because they don't like notebooks and don't want anyone to use them. No matter what you do, there will always be people who look down on easy-to-use tools that enable scientists and practitioners from other disciplines to write, run, and explore ad-hoc code on-the-fly. EDIT: nbdev2's authors are on this page, answering questions. Thank you again!
ahurmazda 4y ago
Thanks but a hard pass from me. The original sin was using goofy JSON as the file format (and no! I dont care for your pretty 5MB pngs polluting my git tree). This is the nth attempt at applying lipstick on the pig (n-1 being jupytext)
boringg 4y ago
This actually works? Awesome - never really thought about how dysfunctional git is with jupyter - I always assumed that it just didn't work. Nice to have someone fix the problem that I just lived with :)
persedes 4y ago
No mention of https://github.com/srstevenson/nb-clean https://github.com/srstevenson/nb-clean ? Has been my go to for this. It seems like nbdev2 is fastais own cooked solution with a bunch of other tools.
killjoywashere 4y ago
Why is this not on brew yet! (╯°□°)╯︵ ┻━┻
danbmil99 4y ago
My dumb as a brick solution to this is just to clear all outputs every time I commit. Having a smart diff\merge sounds like it would make things a lot easier.

301 points by jph00 4y ago

Hi, I'm the author of the git merge driver and Jupyter save hook in nbdev2 :) I'd be happy to answer any questions you have about how we're handling using notebooks with git

howon92 4y ago
I enjoyed reading the writeup and think the solution is clean! Thanks for sharing
jks 4y ago
Can this do three-way merge? If I have to resolve two conflicting code blocks, it is often useful to know how each of them change the code from the shared parent.
- wasimlorgat 4y ago
  It does an ordinary three-way git merge (treating notebooks as plaintext) then a two-way merge on conflicted bits. We opted for that approach because its incredibly simple and has worked perfectly for us (I think since we tend to work with small code cells). I think nbdime has a full-on three-way notebook merge if that's what you need, which can be used together with nbdev's Jupyter save hook to clan up unneeded metadata.

scombridae 4y ago

Subversion used to say, 'CVS done right.' With that slogan there is nowhere you can go. There is no way to do CVS right. -- Linus T. Jupyter's ipynb format is only slightly more amenable to git than say an MSWord doc. Nbdime and friends will never get you to a point where git+jupyter will be worth the ugly.

jph00 4y ago
What are the outstanding problems you feel are there even with the new nbdev2 functionality? Since I've been using it (the prerelease version) over the last few months I haven't come across a single problem, personally, despite doing a very large amount of collaborative notebook work.
fragmede 4y ago
slightly. Since, like, Office 2007, MSWord docs are zip files with xml inside.

whacked_new 4y ago

I used to use Jupytext a lot for this problem, and think it does a decent job. The main problem with Jupytext is its reliance on BYOD (D for discipline), which is a poor (but possibly best available) solution for human systems. IMHO the Jupyter+Git problem stems from the ipynb format. Jupytext does it "right" in the sense that you can work in .ipynb, and diff in .md. But as long as the base format is diff-unfriendly, all tools are methods of indirection of format complexity to tool complexity. That's not to take away from the tool -- it looks great. It also takes the D out of BYOD, which is a win. But I think "solving" it means that anybody who receives an ipynb is able to just look at it out of the box, like plain text, so we're still a ways off.

wasimlorgat 4y ago
I agree re format vs tool complexity. I don't think Jupyter is a particularly difficult format though, its mostly light JSON -- all human-readable. We realised after working with Jupyter+Git for a while that the pain-points were actually with Jupyter editors (and/or their conventions) rather than the format, because they do things like store user-metadata in the file which pollutes diffs and leads to merge conflicts. In fact, if Jupyter editors could handle merge conflicted files, we wouldn't need a custom merge driver either.
wasimlorgat 4y ago
Oh, on the topic of file formats: Quarto also lets you do plaintext notebooks in quite an interesting way, definitely worth checking out: https://quarto.org/docs/computations/python.html https://quarto.org/docs/computations/python.html
- euler_angles 4y ago
  The latest release of nbdev has fully embraced Quarto! It's very awesome, check it out.
cycomanic 4y ago
I have been using jupytext as well and it really makes handling notebooks much easier. I think the decision for Jupyter to save to json was not a good one and they should instead have looked at systems like org mode for inspiration. I also don't understand what you mean by discipline. Yes you need to make sure that everyone has the jupytext extension installed, but that just becomes part of the needed dev environment. After that the whole experience becomes completely seemless.

Helmut10001 4y ago

Not criticizing the authors approach, but the Jupyter+Git Problem was solved for a long time with Jupytext [1]. Jupytext will convert Notebooks (.ipynb) files to Markdown (md) and Python (py) 'on the fly' (while working in Notebooks). - Markdown files can be added to git - Python and .ipynb files are added to .gitignore - Python files allow 'chained' import of notebooks (*.py verions), which allows to split larger notebooks into multiple smaller ones This is my folder structure: . ├── notebooks | ├── notebook1.ipynb # automatically generated from md | └── notebook2.ipynb # automatically generated from md ├── md | ├── notebook1.md # versioned in git | └── notebook2.md # versioned in git ├── py | ├──modules | | ├──__init__.py # empty | | └──tools.py # use for cross-project base tools | ├──__init__.py # empty | ├── notebook1.py # automatically generated from md | └── notebook2.py # automatically generated from md ├──jupytext.toml ├──.git └── README.md See an example here [2] Jupytext is mentioned as a 'potential' alternative. Re the "save" cell output: I usually produce html-files at the end of my notebooks (see the example), and add those either to git or auto-upload to an external webserver. The html is standalone and includes outputs, table of contents, and images (example [3]). I would advice against versioning all outputs (images) in git. Very happy with this approach for a long time now. Jupytext increased my productivity by a hundred percent. [1]: https://github.com/mwouts/jupytext https://github.com/mwouts/jupytext [2]: https://gitlab.vgiscience.de/ad/yfcc_gridagg https://gitlab.vgiscience.de/ad/yfcc_gridagg [3]: https://ad.vgiscience.org/tagmaps-mapnik-jupyter/01_mapnik-tagmaps.html https://ad.vgiscience.org/tagmaps-mapnik-jupyter/01_mapnik-t...

jph00 4y ago
The pros and cons of Jupytext are discussed in the linked post. It's a great approach, but wasn't sufficient for our needs -- so for us, at least, it didn't fully solve the Jupyter+git problem. Specifically, it doesn't handle the situation where you need cell outputs in version control -- since in that case, you still need the notebook, which results in all the usual problems occuring. With nbdev2, you don't need to think about anything or do anything special, and stuff like GitHub notebook rendering, nbviewer, ReviewNB, etc all just work. You just run a single command (`nb_install_hooks`) and that's it. Also, no-one has to install anything extra to view your notebooks, since they're stored in the regular notebook format.

jayd16 4y ago

You know, is there anything like the Language Server Protocol for diff/merge resolution? Seems like there's an opportunity to build a system for semantic aware merge that's language/format aware and tool agnostic (and auto configurable to boot). Suddenly binary formats could become mergeable.

zie 4y ago
See [Pijul](https://pijul.org/manual/theory.html https://pijul.org/manual/theory.html). They did that hard work.
- samatman 4y ago
  Hmm I'm a huge fan of pijul, looks like the future of change management from where I'm sitting, but no: they have not. Semantic diffing needs something like pijul, but a system taking advantage of this doesn't yet exist. Pijul avoids some merge conflicts by design, won't do the wrong thing, and handles conflicts correctly: we still need tools with a fuller awareness of what strings mean to have rich semantic diffs.
  - zie 4y ago
    True, Pijul only offers the safe diff/patch part.

xcambar 4y ago

Stating that git breaks Jupyter notebooks is quite a flex. It stains the article from the very first paragraph.

deleted 4y ago
[deleted]
wasimlorgat 4y ago
Have you worked with Jupyter notebooks and git? It's a literally true statement :D and quite a struggle for many of us
- cycomanic 4y ago
  It's the wrong way around though, Jupyter notebooks break a git work flow. I think the fault here is completely with the design of the Jupyter notebook file format (and the way editors save to it). I think it's quite unfortunate that they did not consider that the format would integrate well with version control systems when first designing ipython notebooks.
  - fumeux_fume 4y ago
    Nah man, you got it backwards. Git still works just fine while my notebooks are definitely broken. Not here to play the blame game, just trying to relate the practical results.
- xcambar 4y ago
  If you leave git diffs in your files, whether Jupyter notebooks or otherwise, and run/compile them... They will break. If you give me a counter example, good for you, but my statement holds true 99%.
  - jsweojtj 4y ago
    You state in the top level comment that this claim stains the article: "Stating that git breaks Jupyter notebooks is quite a flex." But you are saying here: "If you leave git diffs in your files, whether Jupyter notebooks or otherwise, and run/compile them... They will break." Have you changed your mind in this thread? Or what's your objection?

p1necone 4y ago

I haven't used Jupyter but from what I can gather from this article they've built a simultaneous editing system on top of automatically committing to git in the background as multiple people edit things, and using that to share the changes between users. Do I have that right? Because that sounds /insane/.

jph00 4y ago
No it just uses normal git in the normal way. The simple trick is to use a jupyter-native git merge driver, so that merges are done at a cell level instead of a line level. Also, unneeded metadata is removed from the notebook when saving, so there's less changes to merge. Both these two things are done using standard hooks built into each of git and Jupyter. That is: git is written in such a way that it can fully support non line-oriented formats. We just took advantage of that capability.
- p1necone 4y ago
  Ahh right, so you still make manual git commits normally, it's just that the Jupyter UI used to fall over when it encountered merge conflict markers in source files. And now it doesn't fall over any more and can nicely represent them because the conflict markers are no longer done for individual lines of text?
  - jph00 4y ago
    Yes exactly :D
anigbrowl 4y ago
If this doesn't work for you JetBrains' DataSpell might, it's oriented towards notebooks for teams. It has hiccups, things like ipywidgets don't always work as expected so I sometimes find myself falling back to Jupyterlab. But overall it's a very comfy chair.

cigrainger 4y ago

This is one reason I feel lucky to be working with Elixir. Livebook's livemd is basically just markdown. https://livebook.dev https://livebook.dev

wodenokoto 4y ago
So is jupytext, rmd and qmd. But what do you do about the output? The nice thing about markdown-like notebooks is that they play well with git. The nice thing about jupyter style notebooks is that they contain all the content needed to actually _read_ the notebook.

HuwFulcher 4y ago

Whenever I can I strongly recommend not using Jupyter for anything more than the most transient tasks. I don’t know whether it’s the Data Science culture or Jupyter but there is a big lack of discipline in writing maintainable code in DS and non-existent git support is part of that. I always strongly discouraged developing models using notebooks, instead advocating for using .py files and then using notebooks for sanity checking data. I don’t have any clever ideas for how we can move past Jupyter but the sooner we do the better.

maegul 4y ago
Yes, the Data Science culture around maintainable code does seem to be reaching a critical level of toxicity (in some environments at least). In line with a nephew comment of mine, I feel that bringing the immediate interactivity or iteration cycle of notebooks to the development experience would help a lot, and not be too bad a thing for common development either. I've heard of the related nbdev project, which seems like an interesting and compelling idea. But it'd be nice to see the reverse: something that makes ordinary python development more immediate than using a debugger/vanilla REPL.
- Leo_Germond 4y ago
  I think that improving the shell experience and allowing e.g. multimedia content to be displayed and manipulated directly into the shell, would help a lot with interactivity. Maybe some specific terminal emulator (like kitty) with ipython would constitute a good starting point...
tetris11 4y ago
It depends how you use it. When you're still new to the data, using a freeflow text-and-codeblock workflow like jupyter or org-mode really speeds up the exploration phase. Once you have a consistent set of questions, and methods to answer them, then yes, copy off the relevant chunks into their own scripts and source these when using similar data to bring you up to speed, and modify them to your tastes. The issue with starting off with an external script initially is the distracting temptation to refine your code so it can be better used with future data, despite not yet having seen or not knowing what that future data is like. The initial "play and explore" phase of an analysis is very important imo, and notebooks really facilitate that.

medo-bear 4y ago

or ... why not just use org mode?

faustlast 4y ago
I wish org-mode was standard and more appreciated. It is so good, but I feel that I'm the only one using it and it is hard to sell emacs to others.

bsdz 4y ago

I use this plugin for my jupyter notebook git integration. It has a git diff option that's useful but gets very slow for complex documents. Perhaps under the hood it's using one of the other tools mentioned in the postscript. https://github.com/jupyterlab/jupyterlab-git https://github.com/jupyterlab/jupyterlab-git Edit: Looking at the source, it does appear to use nbdime under the hood.

flutetornado 4y ago

JupyterLab JupyterHub Jupytext - converts ipynb to py Nbstripout - strips all output from ipynb Nbmerge - resolves merge conflicts Vim-jupytext - vim plugin to auto convert ipynb to py Papermill - parameterize notebooks Git Pandas, Altair - data analysis / Visualization Phabricator - code reviews of notebooks Vimdiff + vim-jupytext - diffs in terminal This solved all my jupyter problems.

hendry 4y ago

If there a good place to see Jupyter note books solving a real problem? Idk, like importing some data and doing some analysis / forecasting? Most notebooks appear really bad quality. Worse internally. Better off looking at some excel https://github.com/martinshkreli/models https://github.com/martinshkreli/models

throwaway72937 4y ago

Does this work for editing notebooks in VS code? (Unclear to me where the saving hooks reside, and whether you have to edit them through Jupyter labs/notebook) Any issue if the notebooks reside on a remote server?

planede 4y ago

Is it possible to get a "diff3"-like conflict style? That is showing <<< side1 ||| ancestor === side2 >>>

wanderingmind 4y ago

I thought jupytext solved it long ago with percent formatted python file. Since its a python text file you can run automated formating, linting, static type checking and git version diff. What's new is being solved here?

liquids 4y ago

I’ve used this library for a number of projects and it’s a joy to use. I don’t think it’s an understatement to say it’s paradigm shifting – to the extent that once you have your environment set up, you are free to code, think, iterate, deploy and document your projects all at 99% of the speed of thought. There seems to be a lot of discussion in here around the pitfalls of jupyter, and notebooks, and the poor coding practices of data scientists. If you haven’t read the article or used the software I’d like to highlight that all of these (legitimate) complaints are exactly what nbdev2 was created to address, and in my opinion very successfully solves. The way it works is that everything runs off a master notebook, and then with one command: libraries are built, git diffs are magically fixed, tests are run, documentation is automatically created. It doesn’t fundamentally change your workflow in any way, it just abstracts and automates away all of these pain points. There’s a reason that everyone uses jupyter notebooks. They are fun to use, they are great for exploring and developing ideas. And (minus the aforementioned git collaboration issues) they are great for sharing with others, which is a huge part of the wider data science ecosystem. We don’t need to recommend avoiding notebooks, and allege they are just for beginners. We need to use tooling which addresses some of these final issues with writing mature software. And I'd like to thank the authors of nbdev for this. The people who look down their noses at notebooks can continue to do so – but what they will find is that nbdev quite effortlessly leap-frogs over these sneered complaints, and allows you to write better software more productively.

no_identd 4y ago
Okay, now gimme a PowerShell version of it.

kmod 4y ago

Streamlit has completely replaced my usage of Jupyter -- I find it to have the quick iteration speed and visual output of notebooks, but it's just normal python so all the normal tooling works (there is no "git problem") and you don't have the weird state problems of notebooks. Definitely recommend checking it out if you haven't already!

da39a3ee 4y ago

> Here at fast.ai we use Jupyter for everything. All our tests, documentation, and module source code for all of our many libraries is entirely developed in notebooks That sounds like a nightmare. Why would you want to develop a library in a jupyter notebook? > The solution presented here is the result of years of work by many people. It's a bit depressing that it came to this. It's hard not to think that it was a mistake from the beginning and that the format should have been based on using special comment markers in valid code, together with an accompanying JSON metadata file. Or something like that. One way or another, we have a very strong tradition of storing code in plain text files, not embedded in strings in JSON or otherwise embedded in any opaque format. Maybe there'll come a day when it's appropriate to abandon that to get some advantages, but I don't think that day was the original creation of Jupyter. I know it was created by thoughtful and expert software engineers, but I feel that it was a mistake and it's actually made a lot of data science / academia-oriented people less qualified to participate in industry software engineering, because of the poor practices forced upon them by the inability to use git with Jupyter, and notions like developing library code in notebook cells.

ticklemyelmo 4y ago

Why is there a "Jupyter+Git" problem specifically? Why aren't we worrying about the "C+Git" problem and the "XML+Git" problem and the "Python+Git" problem? Because merge markers break, well, every file format. Is it because Jupyter users in particular don't typically understand that there is a formatted text file behind the notebook, or how merge conflicts work?

thefrozenone 4y ago
This is a good question and made me think. I have come up with: "Jupyter notebooks can be thought of as Code and an IDE layout"
Bjartr 4y ago
It's because the primary editor of the notebook files barfs when presented with the file if it includes merge markers since it's no longer valid json. Imagine if one of your normal code-friendly text editors, or your ide, refused to open a .c or .py file and you had to open it in notepad to fix it. That's what it feels like to be forced to drop into a normal text editor rather than using the normal notebook ui to fix the conflicts.

cs702 4y ago

Wow, thank you to the authors! It looks like this tool, nbdev2, solves a real-world problem for Jupyter users, including me, with zero effort required to use it every day. It relies on clever hooks to get git to treat cells as first-class citizens (as opposed to lines of text, the default). Nice! Based on that alone, I would expect nbdev2 to be widely adopted over time. In fact, if it works as well as advertised, it should be incorporated into Jupyter. I, for one, will be giving it a try! If you use Jupyter to solve problems in your domain of expertise, feel free to ignore all the smart-sounding software engineers who will poo-pooh this tool only because they don't like notebooks and don't want anyone to use them. No matter what you do, there will always be people who look down on easy-to-use tools that enable scientists and practitioners from other disciplines to write, run, and explore ad-hoc code on-the-fly. EDIT: nbdev2's authors are on this page, answering questions. Thank you again!

ahurmazda 4y ago

Thanks but a hard pass from me. The original sin was using goofy JSON as the file format (and no! I dont care for your pretty 5MB pngs polluting my git tree). This is the nth attempt at applying lipstick on the pig (n-1 being jupytext)

boringg 4y ago

This actually works? Awesome - never really thought about how dysfunctional git is with jupyter - I always assumed that it just didn't work. Nice to have someone fix the problem that I just lived with :)

persedes 4y ago

No mention of https://github.com/srstevenson/nb-clean https://github.com/srstevenson/nb-clean ? Has been my go to for this. It seems like nbdev2 is fastais own cooked solution with a bunch of other tools.

killjoywashere 4y ago

Why is this not on brew yet! (╯°□°)╯︵ ┻━┻

danbmil99 4y ago

My dumb as a brick solution to this is just to clear all outputs every time I commit. Having a smart diff\merge sounds like it would make things a lot easier.