4 ms·

For a while I viewed Jupyter as a toy that is neither here nor there (sitting between the chairs of development and explanation, briefing or visualization and n

by ptero 3y ago

For a while I viewed Jupyter as a toy that is neither here nor there (sitting between the chairs of development and explanation, briefing or visualization and not doing either job great). But about 2 years ago when I changed jobs into a "Jupyter heavy" environment I was forced to learn it and have grown to really like it. I primarily use Jupyter for prototyping: trying ideas, plotting results and sharing notebooks for others to improve on (or punch holes in). Once a piece clearly shows promise I move it into a module. While this usually means a significant rewrite I actually see it as a benefit: I can be messy in original prototyping plus a rewrite after experimentation often leads to a better code with a small time investment. Beyond the minimal self-discipline of actually moving code into modules I still have two three-character friction points: vim and git. Pointers on addressing those appreciated! I would love, love, love a tight vim and Jupyter integration to be able to switch, easily and frequently, between editing a set of cells in vim and in Jupyter with solid sync between them. I am perfectly OK with vim ignoring the output. And I would love to have a git mode that only checks in the changes that cause actual differences in the python code; not timestamps or the output.

leksak 3y ago
jupytext will solve the git problems for you. Pair with a percent script, ignore the ipynb files.
- kzrdude 3y ago
  jupytext is great. It even allows you to use only .py files directly as notebooks, but I recommend "pair with ipynb" and version controlling the .py file. The ipynb acts like a cache of the cell outputs between invocations of jupyterlab, which is handy too.
  - slt2021 3y ago
    I always wanted a Jupyter like environment, but the one that would natively support and output .py files jupyter notebook as IDE, but with .py files instead of .ipynb
    - enriquto 3y ago
      This is exactly what jupytext gives you. The "pairing" is optional (and sometimed confusing). You can just work with .py files as notebooks and never ever see a .ipynb on your disk again. The notebook .py files are just regular python files with comments that can be edited at hand wih any text editor. Thus you can easily collaborate with your local graybeards that will dislike editing text on their web browsers.
      - kzrdude 3y ago
        What I miss is a notebook launcher button that creates a .py notebook. Don't understand why we don't have that (i could not myself find a way to configure and add it.) I'm not sure why you think it's viable to collaborate with others who don't use jupyter? I mean on any software project, if we don't agree how to compile or run a project, then we usually can't collaborate IMO. Changes become nonsensical (breaking the one or other mode that is not tested by the author of the change.)
        slt2021 3y ago
        I think collaborating with .py files is easier over git. I could not find a way how to check-in .ipynb file and let reviewer review code only, and ignore the metadata part if jupytext will allow me to work in jupyter, check in file to git and create MR and let other people easily see, review and comment my code that would be great
    - kzrdude 3y ago
      Great, then I'd recommend using jupyterlab 3.6 with jupytext right now
geysersam 3y ago
This is a plugin that makes browser text area elements into nvim buffers. https://github.com/glacambre/firenvim https://github.com/glacambre/firenvim Haven't tested it in combination with Jupyter but I imagine it should work
qumpis 3y ago
Are there some usecases where conventional IDE isn't up to the job as the notebooks? I can imagine this to be so for data analysis where pre-loading heavy datasets saves time. Anything else?
- analog31 3y ago
  For me its the matter of actually liking The notebook environment rather than overcoming the shortconings of IDEs.
- Espressosaurus 3y ago
  The notebook environment is excellent for data analysis and exploration. It matches my workflow of manhandling data as I reach for understanding. Once I figure things out of course, either I'm done with the notebook and can copy a few plots out for inclusion in a powerpoint, or I'm done with the notebook and extract a pile of functions into a utility script or package. Either way, it has served its purpose for rapid prototyping and one-off analysis.
mike_ivanov 3y ago
You can comfortably edit the whole thing as Markdown in VSCode (in vim mode, if you'd prefer) - and still have executable cells and nice pdf/html rendering. https://quarto.org/docs/tools/vscode.html https://quarto.org/docs/tools/vscode.html EDIT: actually, you can do it directly in neovim - https://quarto.org/docs/tools/neovim.html https://quarto.org/docs/tools/neovim.html
- deleted 3y ago
  [deleted]
analog31 3y ago
This seems like a nice approach. Mine is similar. I refactor often, by moving stuff from a lower level of discipline to a higher one: Enclose scripts in functions within the notebook, which minimizes the clutter of hidden state. I also have a habit of not walking away from a notebook without doing a "restart kernel and run all cells" to make sure the notebook works. I'm not dealing with giant data sets, so this doesn't cost me much. Frequently used functions go into .py files, using auto-reload to keep things synchronized while I'm working on them. Mature .py files that I might want to re-use in different projects get turned into pip-installable packages. The notebooks become informal tests of the packages. I've never used venv, and never encountered dependency version problems. Some of the dependency horror stories may be obsolete due to the maturation of the big packages such as numpy and matplotlib.
seanhunter 3y ago
There are lots of different versions of this. One of these approaches may work for you https://alpha2phi.medium.com/jupyter-notebook-vim-neovim-c2d67d56d563 https://alpha2phi.medium.com/jupyter-notebook-vim-neovim-c2d...
japanuspus 3y ago
This is my preferred way to work as well. Some techniques to address what you are missing: 1. Only checking in semantic differences (not output, timestamps, etc.): Use the `jupytext`-extension [0] to seamlessly pair you notebook with an lightweight markup version of the input only (which can be used to generate full notebook). 2. Being able to switch between text editor and notebook interface As other have mentioned, there are integrations for multiple editors. Another approach is to move the central code out to a python module which you edit in a text editor, and then use the `%autoreload` magic [1] to reimport that module whenever you execute a cell in the notebook. [0]: https://jupytext.readthedocs.io/en/latest/install.html https://jupytext.readthedocs.io/en/latest/install.html [1]: https://ipython.readthedocs.io/en/stable/config/extensions/autoreload.html https://ipython.readthedocs.io/en/stable/config/extensions/a...

by ptero 3y ago

leksak 3y ago

jupytext will solve the git problems for you. Pair with a percent script, ignore the ipynb files.

kzrdude 3y ago
jupytext is great. It even allows you to use only .py files directly as notebooks, but I recommend "pair with ipynb" and version controlling the .py file. The ipynb acts like a cache of the cell outputs between invocations of jupyterlab, which is handy too.
- slt2021 3y ago
  I always wanted a Jupyter like environment, but the one that would natively support and output .py files jupyter notebook as IDE, but with .py files instead of .ipynb
  - enriquto 3y ago
    This is exactly what jupytext gives you. The "pairing" is optional (and sometimed confusing). You can just work with .py files as notebooks and never ever see a .ipynb on your disk again. The notebook .py files are just regular python files with comments that can be edited at hand wih any text editor. Thus you can easily collaborate with your local graybeards that will dislike editing text on their web browsers.
    - kzrdude 3y ago
      What I miss is a notebook launcher button that creates a .py notebook. Don't understand why we don't have that (i could not myself find a way to configure and add it.) I'm not sure why you think it's viable to collaborate with others who don't use jupyter? I mean on any software project, if we don't agree how to compile or run a project, then we usually can't collaborate IMO. Changes become nonsensical (breaking the one or other mode that is not tested by the author of the change.)
      - slt2021 3y ago
        I think collaborating with .py files is easier over git. I could not find a way how to check-in .ipynb file and let reviewer review code only, and ignore the metadata part if jupytext will allow me to work in jupyter, check in file to git and create MR and let other people easily see, review and comment my code that would be great
  - kzrdude 3y ago
    Great, then I'd recommend using jupyterlab 3.6 with jupytext right now

geysersam 3y ago

This is a plugin that makes browser text area elements into nvim buffers. https://github.com/glacambre/firenvim https://github.com/glacambre/firenvim Haven't tested it in combination with Jupyter but I imagine it should work

qumpis 3y ago

Are there some usecases where conventional IDE isn't up to the job as the notebooks? I can imagine this to be so for data analysis where pre-loading heavy datasets saves time. Anything else?

analog31 3y ago
For me its the matter of actually liking The notebook environment rather than overcoming the shortconings of IDEs.
Espressosaurus 3y ago
The notebook environment is excellent for data analysis and exploration. It matches my workflow of manhandling data as I reach for understanding. Once I figure things out of course, either I'm done with the notebook and can copy a few plots out for inclusion in a powerpoint, or I'm done with the notebook and extract a pile of functions into a utility script or package. Either way, it has served its purpose for rapid prototyping and one-off analysis.

mike_ivanov 3y ago

You can comfortably edit the whole thing as Markdown in VSCode (in vim mode, if you'd prefer) - and still have executable cells and nice pdf/html rendering. https://quarto.org/docs/tools/vscode.html https://quarto.org/docs/tools/vscode.html EDIT: actually, you can do it directly in neovim - https://quarto.org/docs/tools/neovim.html https://quarto.org/docs/tools/neovim.html

deleted 3y ago
[deleted]

analog31 3y ago

This seems like a nice approach. Mine is similar. I refactor often, by moving stuff from a lower level of discipline to a higher one: Enclose scripts in functions within the notebook, which minimizes the clutter of hidden state. I also have a habit of not walking away from a notebook without doing a "restart kernel and run all cells" to make sure the notebook works. I'm not dealing with giant data sets, so this doesn't cost me much. Frequently used functions go into .py files, using auto-reload to keep things synchronized while I'm working on them. Mature .py files that I might want to re-use in different projects get turned into pip-installable packages. The notebooks become informal tests of the packages. I've never used venv, and never encountered dependency version problems. Some of the dependency horror stories may be obsolete due to the maturation of the big packages such as numpy and matplotlib.

seanhunter 3y ago

There are lots of different versions of this. One of these approaches may work for you https://alpha2phi.medium.com/jupyter-notebook-vim-neovim-c2d67d56d563 https://alpha2phi.medium.com/jupyter-notebook-vim-neovim-c2d...

japanuspus 3y ago

This is my preferred way to work as well. Some techniques to address what you are missing: 1. Only checking in semantic differences (not output, timestamps, etc.): Use the `jupytext`-extension [0] to seamlessly pair you notebook with an lightweight markup version of the input only (which can be used to generate full notebook). 2. Being able to switch between text editor and notebook interface As other have mentioned, there are integrations for multiple editors. Another approach is to move the central code out to a python module which you edit in a text editor, and then use the `%autoreload` magic [1] to reimport that module whenever you execute a cell in the notebook. [0]: https://jupytext.readthedocs.io/en/latest/install.html https://jupytext.readthedocs.io/en/latest/install.html [1]: https://ipython.readthedocs.io/en/stable/config/extensions/autoreload.html https://ipython.readthedocs.io/en/stable/config/extensions/a...