4 ms·

I don't understand why Jupyter notebooks are still in use as a writable format when there are editors like VSCode that can treat ordinary python files as notebo

I don't understand why Jupyter notebooks are still in use as a writable format when there are editors like VSCode that can treat ordinary python files as notebooks. The Python extension can submit code blocks to the kernel and import and export Jupyter notebooks, but VSCode is also a real editor on top of that (debugging, git, vim mode, hover info, etc.). For distribution, as a read-only format (like PDF), notebooks are great. But why do people continue to do their work in one?

kaba0 6y ago
Maybe I misunderstand something, but I use jupyter for sage for example and the point of it is that graphical results can be shown there immediately. Some plots can be even made interactive.
- bobbylarrybobby 6y ago
  When running a Python kernel, VSCode splits the editor into two panes, the input .py file pane and the output ipython pane. The output pane supports interactive graphs just like Jupyter does.
  - abdullahkhalids 6y ago
    1. That's still less intuitive for someone who wants to understand the relationship between the code and output. 2. Following from 1, notebooks are meant to be shared with output in place, so others can understand without executing code.
    - Jugurtha 6y ago
      Exactly. When you want to show results to a client, for example. >2. Following from 1, notebooks are meant to be shared with output in place, so others can understand without executing code. We wanted to let others execute the code without being overwhelmed by the code, but we also wanted our colleagues not to write an application or parametrize the notebook by adding metadata or tagging cells, so we added a feature named AppBooks[0]. We automatically detect parameters you want to expose, build a form on top of your notebook, and then serve a clean page. The user gets a page with a form with fields corresponding to your parameters, they just change the values and hit Run. The notebook runs with the new parameters and the client/stakeholder gets the result. This solves the problem of: "I have a notebook, how do I not only show my work, but allow domain experts to tinker and run this with specific values". Especially useful when the parameters are pretty domain specific (like instrumentation for nuclear power plants where the client wants to try the model on edge values). These runs are tracked in case you want to share a training AppBook where it produces a better model, for example. - https://iko.ai/docs/appbook https://iko.ai/docs/appbook
alex-treebeard 6y ago
> why do people continue to do their work in one? Most nb users don't use vscode. I'd say main reason is you can't open a notebook without starting a kernel which takes >1 second. It also is harder to setup. In terms of a flat file formats, there is actually a mark down extension which I would say is better than vscode's format because it is more standardised: https://myst-nb.readthedocs.io/en/latest/ https://myst-nb.readthedocs.io/en/latest/
- cat199 6y ago
  just discovered 'jupytext', which is also pretty good for this: https://jupytext.readthedocs.io/en/latest/ https://jupytext.readthedocs.io/en/latest/
ajford 6y ago
I don't think anyone who writes production code is using Jupyter as a development env. It's more like a self-documenting, persistent repl, or perhaps an interactive doc. I've used it as a way to document data exploration for later use and as an interactive teaching tool. For that, being able to plot inline and embed interactive widgets is great. Also, mixing in rendered markdown content. If you're using Jupyter to write python software and the software is the end result and not the data analysis/process, I think you might be doing it wrong. Not that I'm discounting data scientists (or the like) who use it as their primary environment. Their end result is not a packaged/deployed software and the code itself is just ancillary. For those users, the code and environment is a tool, and the extra weight of what a full IDE (like VSCode or PyCharm) and the features they bring to the table are unnecessary and likely counter-productive.
- teruakohatu 6y ago
  > Not that I'm discounting data scientists (or the like) who use it as their primary environment. Unfortunately it often ends up needing to be executed as a batch job or integrated into a pipeline, which then starts the fun job of extracting the code from the notebook piece by piece. Jupyter having a plain text file format instead of json would make life so much easier, and vastly improve version control.
  - boublepop 6y ago
    For batch jobs you have papermill, for plaintext format you have the jupytext extension that pairs the Jain notebook with a script with notebook markup in comments. I honestly wish that they had used that format as just being the notebook format, but it’s a nice enough and easy fix.
- nojito 6y ago
  Netflix does and it’s absolutely horrifying. https://netflixtechblog.com/notebook-innovation-591ee3221233 https://netflixtechblog.com/notebook-innovation-591ee3221233 https://netflixtechblog.com/scheduling-notebooks-348e6c14cfd6 https://netflixtechblog.com/scheduling-notebooks-348e6c14cfd...
klelatti 6y ago
I think you're suggesting that if you want to retain both code and results you need to manage two different files - raw python and exported notebook. I think many notebook users would see that as inconvenient (and many are not familiar with VSCode).
- jVinc 6y ago
  Jupyterlab has an extention that does this automatically on save, jupytext, also keeps both files in sync over time.
cat199 6y ago
> when there are editors like VSCode that can treat ordinary python files as notebooks was doing this in emacs before there was a jupyter notebook.. as the name implies, notebooks aren't for development, they are for interactive, shareable presentation, and presentation that also works over HTTP without needing local tooling
SatvikBeri 6y ago
I use PyCharm + Jupyter Notebooks with autoreload. I'll build up functions in the notebook by running small bits of code, then move them to a Python file. It's like a supercharged version of a REPL. VSCode's version is nice for sure, but not nearly as nice as being able to go back and read/modify old cells.

kaba0 6y ago

Maybe I misunderstand something, but I use jupyter for sage for example and the point of it is that graphical results can be shown there immediately. Some plots can be even made interactive.

bobbylarrybobby 6y ago
When running a Python kernel, VSCode splits the editor into two panes, the input .py file pane and the output ipython pane. The output pane supports interactive graphs just like Jupyter does.
- abdullahkhalids 6y ago
  1. That's still less intuitive for someone who wants to understand the relationship between the code and output. 2. Following from 1, notebooks are meant to be shared with output in place, so others can understand without executing code.
  - Jugurtha 6y ago
    Exactly. When you want to show results to a client, for example. >2. Following from 1, notebooks are meant to be shared with output in place, so others can understand without executing code. We wanted to let others execute the code without being overwhelmed by the code, but we also wanted our colleagues not to write an application or parametrize the notebook by adding metadata or tagging cells, so we added a feature named AppBooks[0]. We automatically detect parameters you want to expose, build a form on top of your notebook, and then serve a clean page. The user gets a page with a form with fields corresponding to your parameters, they just change the values and hit Run. The notebook runs with the new parameters and the client/stakeholder gets the result. This solves the problem of: "I have a notebook, how do I not only show my work, but allow domain experts to tinker and run this with specific values". Especially useful when the parameters are pretty domain specific (like instrumentation for nuclear power plants where the client wants to try the model on edge values). These runs are tracked in case you want to share a training AppBook where it produces a better model, for example. - https://iko.ai/docs/appbook https://iko.ai/docs/appbook

alex-treebeard 6y ago

> why do people continue to do their work in one? Most nb users don't use vscode. I'd say main reason is you can't open a notebook without starting a kernel which takes >1 second. It also is harder to setup. In terms of a flat file formats, there is actually a mark down extension which I would say is better than vscode's format because it is more standardised: https://myst-nb.readthedocs.io/en/latest/ https://myst-nb.readthedocs.io/en/latest/

cat199 6y ago
just discovered 'jupytext', which is also pretty good for this: https://jupytext.readthedocs.io/en/latest/ https://jupytext.readthedocs.io/en/latest/

ajford 6y ago

I don't think anyone who writes production code is using Jupyter as a development env. It's more like a self-documenting, persistent repl, or perhaps an interactive doc. I've used it as a way to document data exploration for later use and as an interactive teaching tool. For that, being able to plot inline and embed interactive widgets is great. Also, mixing in rendered markdown content. If you're using Jupyter to write python software and the software is the end result and not the data analysis/process, I think you might be doing it wrong. Not that I'm discounting data scientists (or the like) who use it as their primary environment. Their end result is not a packaged/deployed software and the code itself is just ancillary. For those users, the code and environment is a tool, and the extra weight of what a full IDE (like VSCode or PyCharm) and the features they bring to the table are unnecessary and likely counter-productive.

teruakohatu 6y ago
> Not that I'm discounting data scientists (or the like) who use it as their primary environment. Unfortunately it often ends up needing to be executed as a batch job or integrated into a pipeline, which then starts the fun job of extracting the code from the notebook piece by piece. Jupyter having a plain text file format instead of json would make life so much easier, and vastly improve version control.
- boublepop 6y ago
  For batch jobs you have papermill, for plaintext format you have the jupytext extension that pairs the Jain notebook with a script with notebook markup in comments. I honestly wish that they had used that format as just being the notebook format, but it’s a nice enough and easy fix.
nojito 6y ago
Netflix does and it’s absolutely horrifying. https://netflixtechblog.com/notebook-innovation-591ee3221233 https://netflixtechblog.com/notebook-innovation-591ee3221233 https://netflixtechblog.com/scheduling-notebooks-348e6c14cfd6 https://netflixtechblog.com/scheduling-notebooks-348e6c14cfd...

klelatti 6y ago

I think you're suggesting that if you want to retain both code and results you need to manage two different files - raw python and exported notebook. I think many notebook users would see that as inconvenient (and many are not familiar with VSCode).

jVinc 6y ago
Jupyterlab has an extention that does this automatically on save, jupytext, also keeps both files in sync over time.

cat199 6y ago

> when there are editors like VSCode that can treat ordinary python files as notebooks was doing this in emacs before there was a jupyter notebook.. as the name implies, notebooks aren't for development, they are for interactive, shareable presentation, and presentation that also works over HTTP without needing local tooling

SatvikBeri 6y ago

I use PyCharm + Jupyter Notebooks with autoreload. I'll build up functions in the notebook by running small bits of code, then move them to a Python file. It's like a supercharged version of a REPL. VSCode's version is nice for sure, but not nearly as nice as being able to go back and read/modify old cells.