4 ms·

I used to use Jupytext a lot for this problem, and think it does a decent job. The main problem with Jupytext is its reliance on BYOD (D for discipline), which

I used to use Jupytext a lot for this problem, and think it does a decent job. The main problem with Jupytext is its reliance on BYOD (D for discipline), which is a poor (but possibly best available) solution for human systems. IMHO the Jupyter+Git problem stems from the ipynb format. Jupytext does it "right" in the sense that you can work in .ipynb, and diff in .md. But as long as the base format is diff-unfriendly, all tools are methods of indirection of format complexity to tool complexity. That's not to take away from the tool -- it looks great. It also takes the D out of BYOD, which is a win. But I think "solving" it means that anybody who receives an ipynb is able to just look at it out of the box, like plain text, so we're still a ways off.

wasimlorgat 4y ago
I agree re format vs tool complexity. I don't think Jupyter is a particularly difficult format though, its mostly light JSON -- all human-readable. We realised after working with Jupyter+Git for a while that the pain-points were actually with Jupyter editors (and/or their conventions) rather than the format, because they do things like store user-metadata in the file which pollutes diffs and leads to merge conflicts. In fact, if Jupyter editors could handle merge conflicted files, we wouldn't need a custom merge driver either.
wasimlorgat 4y ago
Oh, on the topic of file formats: Quarto also lets you do plaintext notebooks in quite an interesting way, definitely worth checking out: https://quarto.org/docs/computations/python.html https://quarto.org/docs/computations/python.html
- euler_angles 4y ago
  The latest release of nbdev has fully embraced Quarto! It's very awesome, check it out.
cycomanic 4y ago
I have been using jupytext as well and it really makes handling notebooks much easier. I think the decision for Jupyter to save to json was not a good one and they should instead have looked at systems like org mode for inspiration. I also don't understand what you mean by discipline. Yes you need to make sure that everyone has the jupytext extension installed, but that just becomes part of the needed dev environment. After that the whole experience becomes completely seemless.
maegul 4y ago
I feel like the BYODiscipline problem with jupytext could be solved with relatively rudimentary text-editor plugins. I've started rolling my own little plugin utilities and, so far, I have a (very) rudimentary notebook-like interface in plain text. Combine a proper attempt at such a thing with a good interface to a background ipython kernel system (for which ipython could do with some minor enhancements AFAICT), and you'd basically have the best of all worlds (all plain text editor features including version control and personalisation and customisation, and the iterative advantages of notebook code-cell-based runtimes). Hopefully, with such a combination functioning well, there'd be an emergent feature that allows one to more easily get interactive with a code-base for the purposes of understanding, debugging or developing it. Personally, my biggest gripe with Jupyter at the moment is that a few years ago they decided to try to create a quasi-IDE (where they'll probably be beaten by VSCode) rather than improve the general utility of the kernel (or kernel protocol/interface?) and/or the essential notebook UI. It's a personal gripe, and there's clearly value in the web-first interface they've made with JupyterLab (despite the not insubstantial growing pains that project has faced), but, watching ObservableHQ and Pluto (for Julia) focus just on the core notebook interface, while VSCode have focused on the IDE side and easily incorporated or recreated the now rather old/simple Jupyter Notebook interface, both with success, seem like some vindication on my gripe.
- zelphirkalt 4y ago
  By "essential notebook UI", do you mean the old notebook or the lab interface? The old notebook was painful to extend in JS compared to writing lab extensions in TS. In Jupyter Lab 3 they have taken questionable steps, but so far I have been able to work around issues.
  - maegul 4y ago
    I was referring to the notebook-like "part" of the interface, where JupyterLab is a notebook interface with IDE-like components wrapped around it (file explorer, terminal etc). The ObservableHQ interface, for instance, I'd classify as just a notebook interface. IE, individually manipulatable code-cells with a shared runtime. And yea, JupyterLab is better than the classic IMO. But, until recently I'd say, the notebook part of the interface hasn't gotten much love at all, while there've been steps, due to popular demand it seems, to provide alternative UIs that strip away much of what they've added on top of the notebook (ie, simple mode, and now Jupyter Lite). I haven't really got experience writing extensions in the old Jupyter notebook, and hardly any with JupyterLab, but my experience with JupyterLab was frustrating because it felt like they really killed the ability to implement small and hacky plugins like you could with with the old. This always struck me as a shame. A necessary one perhaps given what I presume is the increased power of their new framework. But it always felt like there was a mismatch between the complexity of the plugin framework (which is a full web-dev experience) and the base features of the "product", where customising my test-editor is now much easier AFACT. What issues and questionable steps were you thinking of?

wasimlorgat 4y ago

I agree re format vs tool complexity. I don't think Jupyter is a particularly difficult format though, its mostly light JSON -- all human-readable. We realised after working with Jupyter+Git for a while that the pain-points were actually with Jupyter editors (and/or their conventions) rather than the format, because they do things like store user-metadata in the file which pollutes diffs and leads to merge conflicts. In fact, if Jupyter editors could handle merge conflicted files, we wouldn't need a custom merge driver either.

wasimlorgat 4y ago

Oh, on the topic of file formats: Quarto also lets you do plaintext notebooks in quite an interesting way, definitely worth checking out: https://quarto.org/docs/computations/python.html https://quarto.org/docs/computations/python.html

euler_angles 4y ago
The latest release of nbdev has fully embraced Quarto! It's very awesome, check it out.

cycomanic 4y ago

I have been using jupytext as well and it really makes handling notebooks much easier. I think the decision for Jupyter to save to json was not a good one and they should instead have looked at systems like org mode for inspiration. I also don't understand what you mean by discipline. Yes you need to make sure that everyone has the jupytext extension installed, but that just becomes part of the needed dev environment. After that the whole experience becomes completely seemless.

maegul 4y ago

I feel like the BYODiscipline problem with jupytext could be solved with relatively rudimentary text-editor plugins. I've started rolling my own little plugin utilities and, so far, I have a (very) rudimentary notebook-like interface in plain text. Combine a proper attempt at such a thing with a good interface to a background ipython kernel system (for which ipython could do with some minor enhancements AFAICT), and you'd basically have the best of all worlds (all plain text editor features including version control and personalisation and customisation, and the iterative advantages of notebook code-cell-based runtimes). Hopefully, with such a combination functioning well, there'd be an emergent feature that allows one to more easily get interactive with a code-base for the purposes of understanding, debugging or developing it. Personally, my biggest gripe with Jupyter at the moment is that a few years ago they decided to try to create a quasi-IDE (where they'll probably be beaten by VSCode) rather than improve the general utility of the kernel (or kernel protocol/interface?) and/or the essential notebook UI. It's a personal gripe, and there's clearly value in the web-first interface they've made with JupyterLab (despite the not insubstantial growing pains that project has faced), but, watching ObservableHQ and Pluto (for Julia) focus just on the core notebook interface, while VSCode have focused on the IDE side and easily incorporated or recreated the now rather old/simple Jupyter Notebook interface, both with success, seem like some vindication on my gripe.

zelphirkalt 4y ago
By "essential notebook UI", do you mean the old notebook or the lab interface? The old notebook was painful to extend in JS compared to writing lab extensions in TS. In Jupyter Lab 3 they have taken questionable steps, but so far I have been able to work around issues.
- maegul 4y ago
  I was referring to the notebook-like "part" of the interface, where JupyterLab is a notebook interface with IDE-like components wrapped around it (file explorer, terminal etc). The ObservableHQ interface, for instance, I'd classify as just a notebook interface. IE, individually manipulatable code-cells with a shared runtime. And yea, JupyterLab is better than the classic IMO. But, until recently I'd say, the notebook part of the interface hasn't gotten much love at all, while there've been steps, due to popular demand it seems, to provide alternative UIs that strip away much of what they've added on top of the notebook (ie, simple mode, and now Jupyter Lite). I haven't really got experience writing extensions in the old Jupyter notebook, and hardly any with JupyterLab, but my experience with JupyterLab was frustrating because it felt like they really killed the ability to implement small and hacky plugins like you could with with the old. This always struck me as a shame. A necessary one perhaps given what I presume is the increased power of their new framework. But it always felt like there was a mismatch between the complexity of the plugin framework (which is a full web-dev experience) and the base features of the "product", where customising my test-editor is now much easier AFACT. What issues and questionable steps were you thinking of?