4 ms·

I haven't looked at the storage format here, but for anyone looking to make a Jupyter-allike, please make the storage format friendly for code unmodified code r

by KMag 6y ago

I haven't looked at the storage format here, but for anyone looking to make a Jupyter-allike, please make the storage format friendly for code unmodified code review systems. Jupyter stores python inside JSON, and you can read the diffs, but it's a bit painful. Jupyter has an API for writing your own storage back-end, and I whipped up a prototype to store as regular python files, except adding a space after the first character in any line matching /#[ ]*#/ and putting all of the Jupyter metadata on lines beginning ##. Stripping out the metadata and getting back the original formatting means just removing all lines beginning ##, and removing the second character in every line matching /#[ ]+#/. Unfortunately, under my setup, I had lots of trouble with Jupyter timing out its calls to my storage back-end, likely due to running on a heavily loaded virtualized host.

mathnmusic 6y ago
I think every new notebook project should take ideas from Julia's Pluto: https://www.youtube.com/watch?v=IAF8DjrQSSk https://www.youtube.com/watch?v=IAF8DjrQSSk Working directly with the source files is just one of the niceties it has.
- KMag 6y ago
  Very interesting. Thanks for the pointer. LWN has a nice textual introduction: https://lwn.net/Articles/835930/ https://lwn.net/Articles/835930/
- KMag 6y ago
  Thanks again for the pointer. Pluto really seems to nail the major pain points from notebooks: for a quick summary of Julia's Pluto from https://lwn.net/Articles/835930/ https://lwn.net/Articles/835930/ : 1. Pluto's on-disk format is genuine executable Julia, with metadata in comments. Code review happens on the actually executed code, so highly regulated/otherwise conservative environments don't have to worry about the single source of truth deviating from what code reviewers see. 2. Pluto disallows multiple declarations of global variables. This prevents problems with getting different results, depending in which snippets within the notebook get executed in which order. 3. Pluto makes sure the notebook's displayed output is always in-sync with its source code; it eliminates the problem of someone getting some result, changing some code, and sharing a notebook with results that don't match the code. It does this by performing dependency analysis across the code fragments and topologically sorting them. It then re-evaluates all dependent code snippets when a given snippet is modified. Wow. This is the way.
  - dnautics 6y ago
    My only complaint about the pluto system is it's use of strange extended ascii characters.
    - st1x7 6y ago
      That's a Julia thing, not a Pluto thing. Not a fan personally but I think that's a minority opinion in the Julia community.
      - dnautics 6y ago
        I'm fine with Julia's use of unicode for math symbols in general. The strange ascii characters I'm speaking of here are used to annotate block boundaries in pluto's generated code, they are literally in # comments so they could have used anything. I think they are the old school ascii box drawing characters. They might live in a separate unicode era these days.
contravariant 6y ago
Jupytext makes this all a lot less painful by keeping a Jupyter notebook up to date with a markdown counterpart (without output). It works a lot better to only keep the markdown in source control. The downside is that anyone who starts from a clean repository scratch will need instructions on how to generate the .ipynb files.
ims 6y ago
As far as storing a human readable version for code reviews and diffs, we use nbautoexport for this - https://nbautoexport.drivendata.org/ https://nbautoexport.drivendata.org/ - which automatically exports plain text source files. (Disclaimer: this tool was written and maintained by colleagues of mine.)
- KMag 6y ago
  This tool looks cool. Unfortunately, it looks like this tool in export-only. Ideally, one wants a single source of truth, and if your reviewed artifacts can't be imported for execution, then your code review can't be reviewing your source of truth. Don't get me wrong: it's much better than reviewing JSON diffs, but in highly regulated/conservative industries, we'd prefer to be reviewing the actual code that gets compiled/interpreted.
freedomben 6y ago
This does the same thing. An example is on the main page of the repo: https://github.com/jonklein/niex#notebook-format https://github.com/jonklein/niex#notebook-format
nyanpasu64 6y ago
How do KDE Cantor's project files compare to Jupyter? I want to try out Cantor next time I want to work in a notebook.
unishark 6y ago
Can you customize the diff command used by your code review software? There is a diff tool for Jupyter called nbdime you can install with pip. Of course it doesn't know that you only care about the code, but the diffs are a lot cleaner to read. There's also a version of the diff that will basically create a nicely formatted notbook for viewing differnces in a browser.
andylynch 6y ago
R notebooks get this right. They are written in Markdown. And they support Python and friends too.

by KMag 6y ago

mathnmusic 6y ago

I think every new notebook project should take ideas from Julia's Pluto: https://www.youtube.com/watch?v=IAF8DjrQSSk https://www.youtube.com/watch?v=IAF8DjrQSSk Working directly with the source files is just one of the niceties it has.

KMag 6y ago
Very interesting. Thanks for the pointer. LWN has a nice textual introduction: https://lwn.net/Articles/835930/ https://lwn.net/Articles/835930/
KMag 6y ago
Thanks again for the pointer. Pluto really seems to nail the major pain points from notebooks: for a quick summary of Julia's Pluto from https://lwn.net/Articles/835930/ https://lwn.net/Articles/835930/ : 1. Pluto's on-disk format is genuine executable Julia, with metadata in comments. Code review happens on the actually executed code, so highly regulated/otherwise conservative environments don't have to worry about the single source of truth deviating from what code reviewers see. 2. Pluto disallows multiple declarations of global variables. This prevents problems with getting different results, depending in which snippets within the notebook get executed in which order. 3. Pluto makes sure the notebook's displayed output is always in-sync with its source code; it eliminates the problem of someone getting some result, changing some code, and sharing a notebook with results that don't match the code. It does this by performing dependency analysis across the code fragments and topologically sorting them. It then re-evaluates all dependent code snippets when a given snippet is modified. Wow. This is the way.
- dnautics 6y ago
  My only complaint about the pluto system is it's use of strange extended ascii characters.
  - st1x7 6y ago
    That's a Julia thing, not a Pluto thing. Not a fan personally but I think that's a minority opinion in the Julia community.
    - dnautics 6y ago
      I'm fine with Julia's use of unicode for math symbols in general. The strange ascii characters I'm speaking of here are used to annotate block boundaries in pluto's generated code, they are literally in # comments so they could have used anything. I think they are the old school ascii box drawing characters. They might live in a separate unicode era these days.

contravariant 6y ago

Jupytext makes this all a lot less painful by keeping a Jupyter notebook up to date with a markdown counterpart (without output). It works a lot better to only keep the markdown in source control. The downside is that anyone who starts from a clean repository scratch will need instructions on how to generate the .ipynb files.

ims 6y ago

As far as storing a human readable version for code reviews and diffs, we use nbautoexport for this - https://nbautoexport.drivendata.org/ https://nbautoexport.drivendata.org/ - which automatically exports plain text source files. (Disclaimer: this tool was written and maintained by colleagues of mine.)

KMag 6y ago
This tool looks cool. Unfortunately, it looks like this tool in export-only. Ideally, one wants a single source of truth, and if your reviewed artifacts can't be imported for execution, then your code review can't be reviewing your source of truth. Don't get me wrong: it's much better than reviewing JSON diffs, but in highly regulated/conservative industries, we'd prefer to be reviewing the actual code that gets compiled/interpreted.

freedomben 6y ago

This does the same thing. An example is on the main page of the repo: https://github.com/jonklein/niex#notebook-format https://github.com/jonklein/niex#notebook-format

nyanpasu64 6y ago

How do KDE Cantor's project files compare to Jupyter? I want to try out Cantor next time I want to work in a notebook.

unishark 6y ago

Can you customize the diff command used by your code review software? There is a diff tool for Jupyter called nbdime you can install with pip. Of course it doesn't know that you only care about the code, but the diffs are a lot cleaner to read. There's also a version of the diff that will basically create a nicely formatted notbook for viewing differnces in a browser.

andylynch 6y ago

R notebooks get this right. They are written in Markdown. And they support Python and friends too.