6 ms·

I love the Jupyter ecosystem but it's frustratingly messy sometimes. There's already "nbconvert" built in to Jupyter (lab), built on Pandoc. It's extremely unde

I love the Jupyter ecosystem but it's frustratingly messy sometimes. There's already "nbconvert" built in to Jupyter (lab), built on Pandoc. It's extremely under-documented but it's possible to extend it with templates to generate arbitrary outputs. I've done this to directly generate branded PDF reports via latex. Jupyter Book has a lot of overlap with this feature, but rather than building on nbconvert/pandoc, it starts from scratch with Sphinx. Oh well! Keeps things interesting, I suppose!

chrisjsewell 6y ago
nbconvert/pandoc is very limited in how it can be extended because it only relies on jinja templating rather than a proper AST. You couldn't achieve a lot of the things that sphinx/jupyter book can now do with that. Trust me I know because, before starting work on https://github.com/executablebooks/MyST-Parser https://github.com/executablebooks/MyST-Parser I used the nbconvert/pandoc approach to create https://github.com/chrisjsewell/ipypublish https://github.com/chrisjsewell/ipypublish ;-)
- zdw 6y ago
  This looks great, as extended commonmark support knocks down the last major pain point to switching to Sphinx for docs in the projects I've worked on. The only remaining tooling gap is a markdown linter implemented in python.
  - dmix 6y ago
    Why a python linter? Text editors are usually language agnostic and separate from other tooling.
    - zdw 6y ago
      If the rest of the documentation workflow is in python, it would remove the need to install another entirely separate scripting language runtime to do those checks on every developer laptop, CI system, etc.
- heisenzombie 6y ago
  This is interesting, so do you think the JupyterBook solution could/should replace nbconvert? I was always a bit frustrated by the way nbconvert pulled in pandoc as a dependency, but then didn't make use of any of its power (pandoc surely has a powerful AST). Instead they re-implemented filters and templates themselves. It struck me that it would have been much cleaner to write a proper 'ipynb > AST' Pandoc reader, and then expose pandoc AST filters and templates. I was a bit suspicious of moving away from pandoc to sphinx, because sphinx seems markedly less powerful and general than pandoc. However, you seem to think it's the opposite! Does Sphinx have an AST? I can't see any reference?
  - chrisjsewell 6y ago
    pandocs AST is actually a little limited, in terms of extensability, because it has to resove to JSON. The scorreia.com/software/panflute/code.html is a good python API for it. By contrast sphinx uses docutils AST: https://github.com/live-clones/docutils/blob/master/docutils/docutils/nodes.py https://github.com/live-clones/docutils/blob/master/docutils..., the documentation is not great (working on that) but it is pretty powerful.
  - fiddlosopher 6y ago
    In fact, pandoc now does have an ipynb reader and writer. You can convert between ipynb and any of the formats pandoc supports, and you can manipulate the AST using filters.
    - chrisjsewell 6y ago
      Now use that to replicate the example pages here: https://myst-nb.readthedocs.io/en/latest/examples/basic.html#markdown https://myst-nb.readthedocs.io/en/latest/examples/basic.html... and here https://jupyterbook.org/interactive/launchbuttons.html https://jupyterbook.org/interactive/launchbuttons.html, and let me know how you get on ;-)
    - heisenzombie 6y ago
      Oh! Thanks! I'll definitely take a look at that!
abhishekjha 6y ago
>I've done this to directly generate branded PDF reports via latex. Can you elaborate on this a little? Do you mean via Jupyter notebooks?
- heisenzombie 6y ago
  Yes, via Jupyter notebooks -- using the same machinery that gives the 'File > Export Notebook As...' functionality. My method is roughly this: 1. Create a Python package that subclasses nbconvert.exporters.LatexExporter -- in the setup.py you register the class to 'nbconvert.exporters' in the 'entry_points' dict. 2. You can do general filtering/compiling by registering 'Exporter.filters'. The compile process can be modified by implementing a custom 'from_notebook_node()' 3. You can include a folder containing jinja template(s) in the package to implement whatever styling you want 4. I have some standard metadata that I pull from the notebook/cell metadata (using the standard Jupyter/lab frontend metadata sidebar) to define things like document titles, authors, etc. 5. Do the conversion like: jupyter nbconvert --to=myfancyexportformat notebook.ipynb
  - carreau 6y ago
    Wow someone use the features I wrote years ago ! <3 You might want to submit a blog post to Jupyter.org to show that this is possible and make it more known.
    - heisenzombie 6y ago
      That work you did those years ago made me a hero at work :) Thanks!
      - carreau 6y ago
        Never had the opportunity to really document it, so thinks for finding it; I would have hoped for a much wider variety of exporter to be published. I'm still happy about the "pip install package" --to=qualname. I thin you can hardly make it simpler.
- 6y ago
jjoonathan 6y ago
The JupyterLab extension system is the biggest offender imo. "Oh, I need to install the python package and the jupyter extension separately? And there are no breadcrumbs if I install one but not the other? And I have to figure out which versions are mutually compatible with each other and jupyterlab because 'everything latest' sure isn't? And I have to install extensions to get features that used to be built-in? And then I have to explain the install procedure to the person I'm trying to share with, twice, because he didn't believe me the first time? UGGGGGGGHHHHHHHH"
- ycan 6y ago
  Imagine you manage all this but they change something fundamental and you have to (get everyone else to) reinstall all of it.
  - dividedbyzero 6y ago
    We maintain a custom internal Docker image that everyone uses for their JupyterLab needs to make this manageable; that way, we only have to get everything to work once, and everyone else can just docker pull :latest and be good. It's become a rather large image, since it needs to have everything anyone uses, but for a small-ish team, this has worked well for about two years or so now. Another upside: There is a GCP gcloud one-liner to start a VM for a given Docker image, and the image is designed to work both locally and in a GCP VM, with notebooks in a git repository that gets checked out automatically on container creation, so switching from laptop to 16-core VM to churn through a large dataset from a bucket is pretty seamless.

chrisjsewell 6y ago

nbconvert/pandoc is very limited in how it can be extended because it only relies on jinja templating rather than a proper AST. You couldn't achieve a lot of the things that sphinx/jupyter book can now do with that. Trust me I know because, before starting work on https://github.com/executablebooks/MyST-Parser https://github.com/executablebooks/MyST-Parser I used the nbconvert/pandoc approach to create https://github.com/chrisjsewell/ipypublish https://github.com/chrisjsewell/ipypublish ;-)

zdw 6y ago
This looks great, as extended commonmark support knocks down the last major pain point to switching to Sphinx for docs in the projects I've worked on. The only remaining tooling gap is a markdown linter implemented in python.
- dmix 6y ago
  Why a python linter? Text editors are usually language agnostic and separate from other tooling.
  - zdw 6y ago
    If the rest of the documentation workflow is in python, it would remove the need to install another entirely separate scripting language runtime to do those checks on every developer laptop, CI system, etc.
heisenzombie 6y ago
This is interesting, so do you think the JupyterBook solution could/should replace nbconvert? I was always a bit frustrated by the way nbconvert pulled in pandoc as a dependency, but then didn't make use of any of its power (pandoc surely has a powerful AST). Instead they re-implemented filters and templates themselves. It struck me that it would have been much cleaner to write a proper 'ipynb > AST' Pandoc reader, and then expose pandoc AST filters and templates. I was a bit suspicious of moving away from pandoc to sphinx, because sphinx seems markedly less powerful and general than pandoc. However, you seem to think it's the opposite! Does Sphinx have an AST? I can't see any reference?
- chrisjsewell 6y ago
  pandocs AST is actually a little limited, in terms of extensability, because it has to resove to JSON. The scorreia.com/software/panflute/code.html is a good python API for it. By contrast sphinx uses docutils AST: https://github.com/live-clones/docutils/blob/master/docutils/docutils/nodes.py https://github.com/live-clones/docutils/blob/master/docutils..., the documentation is not great (working on that) but it is pretty powerful.
- fiddlosopher 6y ago
  In fact, pandoc now does have an ipynb reader and writer. You can convert between ipynb and any of the formats pandoc supports, and you can manipulate the AST using filters.
  - chrisjsewell 6y ago
    Now use that to replicate the example pages here: https://myst-nb.readthedocs.io/en/latest/examples/basic.html#markdown https://myst-nb.readthedocs.io/en/latest/examples/basic.html... and here https://jupyterbook.org/interactive/launchbuttons.html https://jupyterbook.org/interactive/launchbuttons.html, and let me know how you get on ;-)
  - heisenzombie 6y ago
    Oh! Thanks! I'll definitely take a look at that!

abhishekjha 6y ago

>I've done this to directly generate branded PDF reports via latex. Can you elaborate on this a little? Do you mean via Jupyter notebooks?

heisenzombie 6y ago
Yes, via Jupyter notebooks -- using the same machinery that gives the 'File > Export Notebook As...' functionality. My method is roughly this: 1. Create a Python package that subclasses nbconvert.exporters.LatexExporter -- in the setup.py you register the class to 'nbconvert.exporters' in the 'entry_points' dict. 2. You can do general filtering/compiling by registering 'Exporter.filters'. The compile process can be modified by implementing a custom 'from_notebook_node()' 3. You can include a folder containing jinja template(s) in the package to implement whatever styling you want 4. I have some standard metadata that I pull from the notebook/cell metadata (using the standard Jupyter/lab frontend metadata sidebar) to define things like document titles, authors, etc. 5. Do the conversion like: jupyter nbconvert --to=myfancyexportformat notebook.ipynb
- carreau 6y ago
  Wow someone use the features I wrote years ago ! <3 You might want to submit a blog post to Jupyter.org to show that this is possible and make it more known.
  - heisenzombie 6y ago
    That work you did those years ago made me a hero at work :) Thanks!
    - carreau 6y ago
      Never had the opportunity to really document it, so thinks for finding it; I would have hoped for a much wider variety of exporter to be published. I'm still happy about the "pip install package" --to=qualname. I thin you can hardly make it simpler.
6y ago

jjoonathan 6y ago

The JupyterLab extension system is the biggest offender imo. "Oh, I need to install the python package and the jupyter extension separately? And there are no breadcrumbs if I install one but not the other? And I have to figure out which versions are mutually compatible with each other and jupyterlab because 'everything latest' sure isn't? And I have to install extensions to get features that used to be built-in? And then I have to explain the install procedure to the person I'm trying to share with, twice, because he didn't believe me the first time? UGGGGGGGHHHHHHHH"

ycan 6y ago
Imagine you manage all this but they change something fundamental and you have to (get everyone else to) reinstall all of it.
- dividedbyzero 6y ago
  We maintain a custom internal Docker image that everyone uses for their JupyterLab needs to make this manageable; that way, we only have to get everything to work once, and everyone else can just docker pull :latest and be good. It's become a rather large image, since it needs to have everything anyone uses, but for a small-ish team, this has worked well for about two years or so now. Another upside: There is a GCP gcloud one-liner to start a VM for a given Docker image, and the image is designed to work both locally and in a GCP VM, with notebooks in a git repository that gets checked out automatically on container creation, so switching from laptop to 16-core VM to churn through a large dataset from a bucket is pretty seamless.