3 ms·

Not criticizing the authors approach, but the Jupyter+Git Problem was solved for a long time with Jupytext [1]. Jupytext will convert Notebooks (.ipynb) files

Not criticizing the authors approach, but the Jupyter+Git Problem was solved for a long time with Jupytext [1]. Jupytext will convert Notebooks (.ipynb) files to Markdown (md) and Python (py) 'on the fly' (while working in Notebooks). - Markdown files can be added to git - Python and .ipynb files are added to .gitignore - Python files allow 'chained' import of notebooks (*.py verions), which allows to split larger notebooks into multiple smaller ones This is my folder structure: . ├── notebooks | ├── notebook1.ipynb # automatically generated from md | └── notebook2.ipynb # automatically generated from md ├── md | ├── notebook1.md # versioned in git | └── notebook2.md # versioned in git ├── py | ├──modules | | ├──__init__.py # empty | | └──tools.py # use for cross-project base tools | ├──__init__.py # empty | ├── notebook1.py # automatically generated from md | └── notebook2.py # automatically generated from md ├──jupytext.toml ├──.git └── README.md See an example here [2] Jupytext is mentioned as a 'potential' alternative. Re the "save" cell output: I usually produce html-files at the end of my notebooks (see the example), and add those either to git or auto-upload to an external webserver. The html is standalone and includes outputs, table of contents, and images (example [3]). I would advice against versioning all outputs (images) in git. Very happy with this approach for a long time now. Jupytext increased my productivity by a hundred percent. [1]: https://github.com/mwouts/jupytext https://github.com/mwouts/jupytext [2]: https://gitlab.vgiscience.de/ad/yfcc_gridagg https://gitlab.vgiscience.de/ad/yfcc_gridagg [3]: https://ad.vgiscience.org/tagmaps-mapnik-jupyter/01_mapnik-tagmaps.html https://ad.vgiscience.org/tagmaps-mapnik-jupyter/01_mapnik-t...

jph00 4y ago
The pros and cons of Jupytext are discussed in the linked post. It's a great approach, but wasn't sufficient for our needs -- so for us, at least, it didn't fully solve the Jupyter+git problem. Specifically, it doesn't handle the situation where you need cell outputs in version control -- since in that case, you still need the notebook, which results in all the usual problems occuring. With nbdev2, you don't need to think about anything or do anything special, and stuff like GitHub notebook rendering, nbviewer, ReviewNB, etc all just work. You just run a single command (`nb_install_hooks`) and that's it. Also, no-one has to install anything extra to view your notebooks, since they're stored in the regular notebook format.
- cycomanic 4y ago
  I'm not sure what your cell outputs are, but if you are doing plots or images inside your notebook, than I agree with the OP that it is generally not a good idea. You now store binary data inside your git repository (which sometimes just carries its own problems), but worse that binary data is mixed into your text diff. If you do a diff between two revisions where some figure changed you essentially will be swamped by the diff in the figure making it difficult to find what actually changed. Now tools like nbreview get around that, but now you're forcing everyone to use the same dev tools, and can't look at diffs any other way really.
  - glenngillen 4y ago
    It’s been a while, but last time I needed to GitHub at least had really great tooling for diffs between versions of image files. > but now you’re fixing everyone to use the same dev tools No they’re not. You can continue using whatever approach you’re using. Attempting to shut down alternatives like this though could be seen as forcing everyone to accept whatever that status quo and lowest common denominator solution, even if their dev tools could support something better.
wodenokoto 4y ago
This is mentioned and critiqued on both pro and cons in the article.
wasimlorgat 4y ago
Jupytext does a lot more than just fix Jupyter/git integration, which is great if you want to adopt its approach, but a bit too heavy IMO if you don't. The approach mentioned here is extremely lightweight and doesn't use too much more than built-in Jupyter/git functionality (and it all happens automatically behind the scenes)
spiim 4y ago
I find this conversion a little bit clunky. The approach that seems to work for me is to use quarto with it's .qmd format. https://quarto.org/ https://quarto.org/
- g8oz 4y ago
  Quarto looks amazing!
  - ellisv 4y ago
    It's pretty nice. At first I thought it was just a rebranding of R Markdown but it's been decently modernized/improved to the point where it makes sense that it is its own, separate thing.

jph00 4y ago

The pros and cons of Jupytext are discussed in the linked post. It's a great approach, but wasn't sufficient for our needs -- so for us, at least, it didn't fully solve the Jupyter+git problem. Specifically, it doesn't handle the situation where you need cell outputs in version control -- since in that case, you still need the notebook, which results in all the usual problems occuring. With nbdev2, you don't need to think about anything or do anything special, and stuff like GitHub notebook rendering, nbviewer, ReviewNB, etc all just work. You just run a single command (`nb_install_hooks`) and that's it. Also, no-one has to install anything extra to view your notebooks, since they're stored in the regular notebook format.

cycomanic 4y ago
I'm not sure what your cell outputs are, but if you are doing plots or images inside your notebook, than I agree with the OP that it is generally not a good idea. You now store binary data inside your git repository (which sometimes just carries its own problems), but worse that binary data is mixed into your text diff. If you do a diff between two revisions where some figure changed you essentially will be swamped by the diff in the figure making it difficult to find what actually changed. Now tools like nbreview get around that, but now you're forcing everyone to use the same dev tools, and can't look at diffs any other way really.
- glenngillen 4y ago
  It’s been a while, but last time I needed to GitHub at least had really great tooling for diffs between versions of image files. > but now you’re fixing everyone to use the same dev tools No they’re not. You can continue using whatever approach you’re using. Attempting to shut down alternatives like this though could be seen as forcing everyone to accept whatever that status quo and lowest common denominator solution, even if their dev tools could support something better.

wodenokoto 4y ago

This is mentioned and critiqued on both pro and cons in the article.

wasimlorgat 4y ago

Jupytext does a lot more than just fix Jupyter/git integration, which is great if you want to adopt its approach, but a bit too heavy IMO if you don't. The approach mentioned here is extremely lightweight and doesn't use too much more than built-in Jupyter/git functionality (and it all happens automatically behind the scenes)

spiim 4y ago

I find this conversion a little bit clunky. The approach that seems to work for me is to use quarto with it's .qmd format. https://quarto.org/ https://quarto.org/

g8oz 4y ago
Quarto looks amazing!
- ellisv 4y ago
  It's pretty nice. At first I thought it was just a rebranding of R Markdown but it's been decently modernized/improved to the point where it makes sense that it is its own, separate thing.