7 ms·

Jupytext: Jupyter notebooks as Python scripts

151 points by JA7Cal 8y ago

thomasfedb 8y ago
This is really cool stuff. Editing notebooks in the browser is pretty good, but it can't (yet) match the experience of a well configured (and extended) text editor.
- subw00f 8y ago
  I've been using vscode with the jupyter extension (https://github.com/DonJayamanne/vscodeJupyter https://github.com/DonJayamanne/vscodeJupyter) and it's exactly what you describe, experience of a good text editor and the visual notebook workflow.
  - nojvek 8y ago
    While working on udacity classes, I found vscode python jupyter plugin very helpful. Having a nice editor makes you so much more productive. I wrote a little python pip module to help me convert between .ipynb and .py. https://github.com/nojvek/vscode-ipynb-py-converter https://github.com/nojvek/vscode-ipynb-py-converter Made a world of difference.
  - thomasfedb 8y ago
    Neat, I'll give that a shot. Does it handle all kernels?
    - kylebarron 8y ago
      It doesn't appear to. Helpfully, Hydrogen [0] for Atom was designed to be entirely language agnostic. [0]: https://github.com/nteract/hydrogen https://github.com/nteract/hydrogen
kankroc 8y ago
Can someone provide me a usecase for this when compared to executing jupyter straight into VSCode/Atom with vscodeJupyter/Hydrogen?
- enriquto 8y ago
  The idea is that you can edit your notebooks with your favorite text editor, use git for version tracking them, and no need to mess with a web browser at all.
- kylebarron 8y ago
  In the medium term, it's likely that Hydrogen will be able to read and write .ipynb files. It can currently export .ipynb files but has some bugs [0]. [0]: https://github.com/nteract/hydrogen/issues/1296 https://github.com/nteract/hydrogen/issues/1296
hcrisp 8y ago
Could this be the answer to "I don't like notebooks" [0]? Or does calling scripts from within a notebook still do horrible things to the managed state? [0] https://news.ycombinator.com/item?id=17856700 https://news.ycombinator.com/item?id=17856700
- zimablue 8y ago
  I've never thought this criticism was that relevant. Notebooks have been incredibly successful, and if it's possible to call your code in the wrong order and create a mess, it's probably a problem with your code not your coding environment. The more functional your code, the less of a problem jupyter notebooks are. They encourage better code not worse imo.
  - mikepurvis 8y ago
    The "managed state" is important, though. For example I deal with cases where I'm doing some big expensive operations upfront (eg, processing a bunch of data coming in over a network share— ROS bag files, which are also bzipped, adding more upfront processing cost), and I'm interesting in tweaking the plots and analysis coming out the other end without having to re-run the processing each time. Even as a relatively experienced developer, I've found it hard to reason about what exactly it is that Jupyter is doing under the hood; basically things will break in weird ways, and rerunning everything will mysteriously fix it, or I have to use hacks to force it to re-import certain modules in order to get things the way I expect. Basically, it ends up being easier to just manage this myself, like doing the processing and pickling the result, then using other, standalone notebooks to load and render the pickle. But this shouldn't be necessary.
    - mynewtb 8y ago
      That sounds really weird. The only time I noticed such weirdness was when a colleague and me were simultaneously editing a notebook on Google Colab. The code was shared but not the state, super weird...
- misnome 8y ago
enriquto 8y ago
This is fantastic! Now, they just need to adapt jupyter itself so that it reads the python files directly, dispensing with the need of the silly ipynb!
- dfee 8y ago
  I think that’s called IPython. But I agree with your sentiment.
  - enriquto 8y ago
    I was not being ironic. Notebooks are lovely as an output format for showing your research and allowing others to fiddle with your code. However, for people who do not want to edit inside a web browser window, jupytext is a godsend!
    - dec0dedab0de 8y ago
      At that point you should put most of your work in a separate file, and just import the stuff you need for your demo.
      - PeterisP 8y ago
        In such a use case, the whole point is showing your work in a 'literate programming' manner i.e. having the appropriate code that's being run be right there between the descriptive text, instead of just running do_something() that you imported from somewhere.
      - enriquto 8y ago
        I do not want to hide anything, my only reason to use notebooks is to put all the computations inside them.
Grumbledour 8y ago
This seems neat, but also a bit strange. Are Jupyter notebooks so prevalent that it makes sense to ghave an exporter that makes them into markdown and back again? I mean, I always thought this workflow was neat, but isn't one of the selling points of jupyter the WYSIWYG Interface? And if you like working in code, would it not be easier to create a markdown extension for embedded code (Or does something like this already exist?) and create editor plugins or command line tools to render said document or provide a live preview? Personally, I totally want that functionality but don't want to mess with running jupyter kernels or using my browser to work in my notebook. So where is my markdown with embedded code WYSIWYG editor that saves into plain text files? ;-)
- mic47 8y ago
  1 thing. Code review. Even thou notebooks are not meant for production code, it's good to check them into repository to document what you did (let's ignore out of order computation and other horrors), and get feedback on your code / experiment. Right now, ipython notebook is stored as json, where code is list of json strings (so that you have it line by line). Additionally, the "WYSIWYG Interface" is horrible in comparison to vim, emacs, idea, pycharm or whatever you are using. What I like about notebooks is being able to present things as experiment: this is the code, and there are the outputs. "WYSIWYG Interface" is good if you want to tweak few things, or if you are starting out.
  - Grumbledour 8y ago
    I think you misunderstood me. My question was not why you would use an external editor (and get all the benefits you provided), but rather, why work back from jupyter to achieve this rather than working forward from simple markdown and allow code blocks to be embedded and run from the editor. But maybe I am missing some other features than the interface that the ipython kernel provides? I haven't used it much myself, since I found the setup and usage rather complicated, though to be fair, I don't really use python and its ecosystem much.
    - mic47 8y ago
micheda 8y ago
Nice! similar to https://github.com/minodes/pynb https://github.com/minodes/pynb , that converts Jupyter Notebooks to Python scripts you can run and back, preserving Markdown cells.
drej 8y ago
If you're worried about diffs, check out https://github.com/jupyter/nbdime https://github.com/jupyter/nbdime
- julienchastang 8y ago
  That's true but ideally you would like the native format to be easily diffable (e.g., plain text) and not require special tools.
dec0dedab0de 8y ago
The Python script can be opened with PyCharm This is going to be a bit pedantic, but for anyone that doesn't realize. Pycharm can already open ipynb files, and it will connect to a running notebook. I don't do it very often, usually just to quickly find my way around some library code. This project looks great though. I will likely use it, as I tend to use Jupyter to work out code before copying it over to my main project.
- Seanny123 8y ago
  In my personal experience, PyCharm really mangles Jupyter Notebooks. Has it gotten to better recently?
  - sgillen 8y ago
    I have not experienced much mangeling, but the newest CE edition still doesn't do notebooks very well IMO. to the point where I use jupyter-lab for my notebooks/prototypes and pycahrm for packages.
xaedes 8y ago
I really like the markdown conversion! This should make it easy to import ipynb notebooks into markdown based documentations.
turingcompeteme 8y ago
This is neat. If you are looking for something a bit simpler: put your code in a python package, install it, and then use %autoreload You can now develop in your favorite editor, and changes are automatically reloaded. More detail here: https://blog.godatadriven.com/write-less-terrible-notebook-code https://blog.godatadriven.com/write-less-terrible-notebook-c...
- xaedes 8y ago
  Well, thank you! No more annoying kernel restarts, that really bugged me a long a time.
azag0 8y ago
I have written https://github.com/azag0/knitj https://github.com/azag0/knitj to solve a similar use case, but with an alternative interactive browser interface

151 points by JA7Cal 8y ago

This is really cool stuff. Editing notebooks in the browser is pretty good, but it can't (yet) match the experience of a well configured (and extended) text editor.

subw00f 8y ago
I've been using vscode with the jupyter extension (https://github.com/DonJayamanne/vscodeJupyter https://github.com/DonJayamanne/vscodeJupyter) and it's exactly what you describe, experience of a good text editor and the visual notebook workflow.
- nojvek 8y ago
  While working on udacity classes, I found vscode python jupyter plugin very helpful. Having a nice editor makes you so much more productive. I wrote a little python pip module to help me convert between .ipynb and .py. https://github.com/nojvek/vscode-ipynb-py-converter https://github.com/nojvek/vscode-ipynb-py-converter Made a world of difference.
- thomasfedb 8y ago
  Neat, I'll give that a shot. Does it handle all kernels?
  - kylebarron 8y ago
    It doesn't appear to. Helpfully, Hydrogen [0] for Atom was designed to be entirely language agnostic. [0]: https://github.com/nteract/hydrogen https://github.com/nteract/hydrogen

kankroc 8y ago

Can someone provide me a usecase for this when compared to executing jupyter straight into VSCode/Atom with vscodeJupyter/Hydrogen?

enriquto 8y ago
The idea is that you can edit your notebooks with your favorite text editor, use git for version tracking them, and no need to mess with a web browser at all.
kylebarron 8y ago
In the medium term, it's likely that Hydrogen will be able to read and write .ipynb files. It can currently export .ipynb files but has some bugs [0]. [0]: https://github.com/nteract/hydrogen/issues/1296 https://github.com/nteract/hydrogen/issues/1296

hcrisp 8y ago

Could this be the answer to "I don't like notebooks" [0]? Or does calling scripts from within a notebook still do horrible things to the managed state? [0] https://news.ycombinator.com/item?id=17856700 https://news.ycombinator.com/item?id=17856700

zimablue 8y ago
I've never thought this criticism was that relevant. Notebooks have been incredibly successful, and if it's possible to call your code in the wrong order and create a mess, it's probably a problem with your code not your coding environment. The more functional your code, the less of a problem jupyter notebooks are. They encourage better code not worse imo.
- mikepurvis 8y ago
  The "managed state" is important, though. For example I deal with cases where I'm doing some big expensive operations upfront (eg, processing a bunch of data coming in over a network share— ROS bag files, which are also bzipped, adding more upfront processing cost), and I'm interesting in tweaking the plots and analysis coming out the other end without having to re-run the processing each time. Even as a relatively experienced developer, I've found it hard to reason about what exactly it is that Jupyter is doing under the hood; basically things will break in weird ways, and rerunning everything will mysteriously fix it, or I have to use hacks to force it to re-import certain modules in order to get things the way I expect. Basically, it ends up being easier to just manage this myself, like doing the processing and pickling the result, then using other, standalone notebooks to load and render the pickle. But this shouldn't be necessary.
  - mynewtb 8y ago
    That sounds really weird. The only time I noticed such weirdness was when a colleague and me were simultaneously editing a notebook on Google Colab. The code was shared but not the state, super weird...
misnome 8y ago

enriquto 8y ago

This is fantastic! Now, they just need to adapt jupyter itself so that it reads the python files directly, dispensing with the need of the silly ipynb!

dfee 8y ago
I think that’s called IPython. But I agree with your sentiment.
- enriquto 8y ago
  I was not being ironic. Notebooks are lovely as an output format for showing your research and allowing others to fiddle with your code. However, for people who do not want to edit inside a web browser window, jupytext is a godsend!
  - dec0dedab0de 8y ago
    At that point you should put most of your work in a separate file, and just import the stuff you need for your demo.
    - PeterisP 8y ago
      In such a use case, the whole point is showing your work in a 'literate programming' manner i.e. having the appropriate code that's being run be right there between the descriptive text, instead of just running do_something() that you imported from somewhere.
    - enriquto 8y ago
      I do not want to hide anything, my only reason to use notebooks is to put all the computations inside them.

Grumbledour 8y ago

This seems neat, but also a bit strange. Are Jupyter notebooks so prevalent that it makes sense to ghave an exporter that makes them into markdown and back again? I mean, I always thought this workflow was neat, but isn't one of the selling points of jupyter the WYSIWYG Interface? And if you like working in code, would it not be easier to create a markdown extension for embedded code (Or does something like this already exist?) and create editor plugins or command line tools to render said document or provide a live preview? Personally, I totally want that functionality but don't want to mess with running jupyter kernels or using my browser to work in my notebook. So where is my markdown with embedded code WYSIWYG editor that saves into plain text files? ;-)

mic47 8y ago
1 thing. Code review. Even thou notebooks are not meant for production code, it's good to check them into repository to document what you did (let's ignore out of order computation and other horrors), and get feedback on your code / experiment. Right now, ipython notebook is stored as json, where code is list of json strings (so that you have it line by line). Additionally, the "WYSIWYG Interface" is horrible in comparison to vim, emacs, idea, pycharm or whatever you are using. What I like about notebooks is being able to present things as experiment: this is the code, and there are the outputs. "WYSIWYG Interface" is good if you want to tweak few things, or if you are starting out.
- Grumbledour 8y ago
  I think you misunderstood me. My question was not why you would use an external editor (and get all the benefits you provided), but rather, why work back from jupyter to achieve this rather than working forward from simple markdown and allow code blocks to be embedded and run from the editor. But maybe I am missing some other features than the interface that the ipython kernel provides? I haven't used it much myself, since I found the setup and usage rather complicated, though to be fair, I don't really use python and its ecosystem much.
  - mic47 8y ago

micheda 8y ago

Nice! similar to https://github.com/minodes/pynb https://github.com/minodes/pynb , that converts Jupyter Notebooks to Python scripts you can run and back, preserving Markdown cells.

drej 8y ago

If you're worried about diffs, check out https://github.com/jupyter/nbdime https://github.com/jupyter/nbdime

julienchastang 8y ago
That's true but ideally you would like the native format to be easily diffable (e.g., plain text) and not require special tools.

dec0dedab0de 8y ago

The Python script can be opened with PyCharm This is going to be a bit pedantic, but for anyone that doesn't realize. Pycharm can already open ipynb files, and it will connect to a running notebook. I don't do it very often, usually just to quickly find my way around some library code. This project looks great though. I will likely use it, as I tend to use Jupyter to work out code before copying it over to my main project.

Seanny123 8y ago
In my personal experience, PyCharm really mangles Jupyter Notebooks. Has it gotten to better recently?
- sgillen 8y ago
  I have not experienced much mangeling, but the newest CE edition still doesn't do notebooks very well IMO. to the point where I use jupyter-lab for my notebooks/prototypes and pycahrm for packages.

xaedes 8y ago

I really like the markdown conversion! This should make it easy to import ipynb notebooks into markdown based documentations.

turingcompeteme 8y ago

This is neat. If you are looking for something a bit simpler: put your code in a python package, install it, and then use %autoreload You can now develop in your favorite editor, and changes are automatically reloaded. More detail here: https://blog.godatadriven.com/write-less-terrible-notebook-code https://blog.godatadriven.com/write-less-terrible-notebook-c...

xaedes 8y ago
Well, thank you! No more annoying kernel restarts, that really bugged me a long a time.

azag0 8y ago

I have written https://github.com/azag0/knitj https://github.com/azag0/knitj to solve a similar use case, but with an alternative interactive browser interface