6 ms·

Jupyter Notebooks as E2E Tests

81 points by freetonik 2y ago

wodenokoto 2y ago
It looks like the final solution will have to run notebooks twice - once to check for errors and once more to render to documentation. The concept of running code examples inside documentation as a part of tests is well known, and extending it to end-to-end tests / user guides is a good idea. Next step might be to add hidden code cells with asserts, to check that the code not only runs, but creates the expected output.
- freetonik 2y ago
  The documentation is rendered without running the notebook in my case, because we store notebooks with all the outputs, so the doc builder (Sphinx) just converts them to HTML as is.
  - wodenokoto 2y ago
    To me, the reasonable path would be to have the CI built the notebook, check it for errors and then render it to the docs. As per the article, they have to run the notebook locally, commit all the outputs, then the CI checks if the notebook can run, and then renders the _committed_ output to the docs. This means that the verified code and output can be out of sync, partly defeating the purpose of the CI.
- foundart 2y ago
  Having the user guides be notebooks is a very interesting idea and the step from there to e2e test as the author describes seems like a very logical one.
  - foundart 2y ago
    The short article is definitely worth a read, as is the Netflix post it links to
freetonik 2y ago
I’m confused. I’ve submitted this link almost two days ago, but now it says “3 hours ago”, as if the timestamp was modified.
- abdullahkhalids 2y ago
  This is intended HN behavior. Someone else submitted it again today and due to the closeness of times (within a few days), a new item is not created but the old item is boosted.
  - freetonik 2y ago
    Ah, thanks for clarifying, did not know this. I thought when someone submits an existing link it either has no effect (and the submitter is redirected to the old post), or a new post is created if enough time had passed since the original submission.
    - bobnamob 2y ago
      There’s also a chance it was placed on the second chance queue and reposted “automatically”
abdullahkhalids 2y ago
You can just use nbclient/nbconvert as a library instead of on cmd. Execute the notebook, and the cell outputs are directly available as python objects [1]. Error handling is also inbuilt. This should make integration with pytest etc, much simpler. [1] https://nbconvert.readthedocs.io/en/latest/execute_api.html#executing-notebooks-using-the-python-api-interface https://nbconvert.readthedocs.io/en/latest/execute_api.html#...
- westurner 2y ago
  There are a number of ways to run tests within notebooks and to run notebooks as tests. ipytest: https://github.com/chmp/ipytest https://github.com/chmp/ipytest nbval: https://github.com/computationalmodelling/nbval https://github.com/computationalmodelling/nbval papermill: https://github.com/nteract/papermill https://github.com/nteract/papermill awesome-jupyter > Testing: https://github.com/markusschanta/awesome-jupyter https://github.com/markusschanta/awesome-jupyter "Is there a cell tag convention to skip execution?" https://discourse.jupyter.org/t/is-there-a-cell-tag-convention-to-skip-execution/5445/10 https://discourse.jupyter.org/t/is-there-a-cell-tag-conventi... Methods for importing notebooks like regular Python modules: ipynb, import_ipnb, nbimporter, nbdev nbimporter README on why not to import notebooks like modules: https://github.com/grst/nbimporter#update-2019-06-i-do-not-recommend-any-more-to-use-nbimporter https://github.com/grst/nbimporter#update-2019-06-i-do-not-r... nbdev parses jupyter notebook cell comments to determine whether to export a module to a .py file: https://nbdev1.fast.ai/export.html https://nbdev1.fast.ai/export.html There's a Jupyter kernel for running ansible playbooks as notebooks with output. dask-labextension does more of a retry-on-failure workflow
udioron 2y ago
Very cool! One can write a pytest plugin that executes notebooks from a folder with custom pytest fixtures support.
- japanuspus 2y ago
  You don't even need a plugin: I have been using a setup with parametrized tests and `runpy` for a while [0]. Works especially well if you use jupytext[1] for just storing the input part of the notebook as a python script -- which is good for version control sanity in any case. [0]: https://stackoverflow.com/a/56813896/212538 https://stackoverflow.com/a/56813896/212538 [1]: https://jupytext.readthedocs.io/en/latest/ https://jupytext.readthedocs.io/en/latest/
zurfer 2y ago
for anybody who wants to schedule or automatically run jupyter notebooks I recommend also looking into papermill: https://papermill.readthedocs.io/en/latest/ https://papermill.readthedocs.io/en/latest/
- rmholt 2y ago
  Agreed! I use it and it's a breath of fresh air, I have to use Jupyter because of colleagues but I really prefer python scripts, and this let's me kind of run a Jupyter notebook as if it were a script, even with cli flags
CJefferson 2y ago
I've used this in a few systems, using (in my case) nbconvert. As I write more code, I increasingly find the most important thing about tests early on is that they are easy to write and maintain. The help that, I find one of the best 'quick test suites' is "run program, save output, run 'git diff' to see if anything changes". This has several advantages. If you have lots of small programs it's trivial to parallelise. It's easy to see what outputs have changed. It's very easy to write weird one-off special tests that need to do something a bit unusual. Yes, eventually you will probably want some nicer test framework, but even then I often keep this framework around, as there will still often be a few tests that don't fit nicely in whatever fancy testing library I'm trying to use (for example, checking a program's front end produces correct error messages when given invalid input).
miohtama 2y ago
Notebooks are a wonderful tool, especially now when Visual Studio Code has superb editor for them. Before this, using web based Jupyter was a bit pain. I use PyCharm for .py files, which is still a bit broken for notebooks, but always go to VSC for notebooks. If someone needs here is an some sample code to run notebooks programically, and tune the output and formatting: https://github.com/tradingstrategy-ai/trade-executor/blob/master/tradeexecutor/backtest/tearsheet.py https://github.com/tradingstrategy-ai/trade-executor/blob/ma...
- qsort 2y ago
  What's broken about them in pycharm? I've been using pycharm as my primary IDE for a while now and I haven't ever experienced major problems, this includes working on projects where other people were using VSC.
  - dehrmann 2y ago
    Using notebooks while remoting (not just a remote Jupyter instance) is broken. It's not this particular bug, but it's similar: https://youtrack.jetbrains.com/issue/PY-71195/Remote-Development-Jupyter-Notebook-not-rendered https://youtrack.jetbrains.com/issue/PY-71195/Remote-Develop...
batmansmk 2y ago
Maintaining e2e tests is a pain. Maintaining a notebook is a pain. It seems it was a given somebody would make this match made in heaven!
- vvladymyrov 2y ago
  I can suggest improvement - combining long running e2e tests with notebooks is even better match
pplonski86 2y ago
Mixing code, markdown and execution results in one output file gives Jupyter notebooks superpower. You can really have anything in the ipynb file, stored in JSON. I wish there was two types of ipynb files, one for file with just code and markdown (for example ipynbc), and one for keeping code+markdown+results. BTW, sometime ago I wrote an article about surprising things that you can build with Jupyter notebook https://mljar.com/blog/how-to-use-jupyter-notebook/ https://mljar.com/blog/how-to-use-jupyter-notebook/ You will find in the list: blog, book, website, web app, dashboards, REST API, even full packages :)
- armanboyaci 2y ago
  > I wish there was two types of ipynb files, one for file with just code and markdown (for example ipynbc), and one for keeping code+markdown+results. I believe you can achieve that if you use jupytext library, right?
taeric 2y ago
I find the gigantic swing of a lot of software developers from the pure separation of data and styling into notebooks rather amusing. In particular, there is very little that a "notebook" style environment can get you that you couldn't have gotten as output from any previous testing regime. Styling test results used to be a thing people spent a fair amount of time on so that test reports were pretty and informative. Reports on aggregate results could show trends in how they have been executing in the past. Now, I grant that this article is subtly different. In particular, the notebooks are an artifact that they are testing anyway. So, having more reliance on that technology may itself be a good end goal. I still have a hard time shaking that notebooks are being treated as a hammer looking for nails by so many teams.
tpoacher 2y ago
I really don't get (jupyter) notebooks. It always feels like an over-engineered solution to a simple problem. The only advantage they have over 'proper' code generating a 'proper' report is being able to modify code directly on the report (i.e. as opposed to having to change the code in a separate file instead). Which sounds good and all, but this is rarely what I see notebooks actually used for. For the most part, people use them as an IDE, and a notebook is a terrible choice of IDE compared to even something as simple as editing in nano. And the disadvantages of using one generally completely outweigh the benefit of live-editing in the first place, and encourage all sorts of bad programming habits / hacky code structure. Whenever I've required a report which intermingles code, text, and outputs, a simple bash-script processing my (literate programming) codebase has always done a far more impressive job of generating a useful report. And, without having to sacrifice proper packaging / code-organisation semantics just for the sake of a report. I find it a big shame that the iodide / pyodide notebooks didn't take off in the same way, at least. Those seemed tightly integrated to html in a way that was very elegant. (they're not completely gone though, it was nice to see this example on the front page earlier: https://news.ycombinator.com/item?id=42425489 https://news.ycombinator.com/item?id=42425489 )
- deleted 2y ago
  [deleted]
- parpfish 2y ago
  i mostly agree, but one way that i've found to make notebooks good/usable is this: - move all of the code out of the notebook and into a nearby python module where it can be linted/mypy-d/version-controlled/code-reviewed easily - tell the notebook to import that module and make a small number of function calls to get whatever data you need and make plots. at this point the notebook is really nothing more than a REPL with inlined plots/graphics
  - youainti 2y ago
    This is what I've found works the best as well.
- dr_kiszonka 2y ago
arminiusreturns 2y ago
I prefer emacs org-mode babel.
remram 2y ago
Ever since Marimo launched, I haven't looked back at Jupyter: https://marimo.io/ https://marimo.io/ https://news.ycombinator.com/item?id=38971966 https://news.ycombinator.com/item?id=38971966 On top of their reactive goodness, Marimo notebooks are .py files, which makes it very suitable for this kind of (ab)use.
mscolnick 2y ago
this sort of hackery to work with jupyter notebooks is exactly why we are building marimo (https://github.com/marimo-team/marimo https://github.com/marimo-team/marimo). this should not be the standard

81 points by freetonik 2y ago

It looks like the final solution will have to run notebooks twice - once to check for errors and once more to render to documentation. The concept of running code examples inside documentation as a part of tests is well known, and extending it to end-to-end tests / user guides is a good idea. Next step might be to add hidden code cells with asserts, to check that the code not only runs, but creates the expected output.

freetonik 2y ago
The documentation is rendered without running the notebook in my case, because we store notebooks with all the outputs, so the doc builder (Sphinx) just converts them to HTML as is.
- wodenokoto 2y ago
  To me, the reasonable path would be to have the CI built the notebook, check it for errors and then render it to the docs. As per the article, they have to run the notebook locally, commit all the outputs, then the CI checks if the notebook can run, and then renders the _committed_ output to the docs. This means that the verified code and output can be out of sync, partly defeating the purpose of the CI.
foundart 2y ago
Having the user guides be notebooks is a very interesting idea and the step from there to e2e test as the author describes seems like a very logical one.
- foundart 2y ago
  The short article is definitely worth a read, as is the Netflix post it links to

freetonik 2y ago

I’m confused. I’ve submitted this link almost two days ago, but now it says “3 hours ago”, as if the timestamp was modified.

abdullahkhalids 2y ago
This is intended HN behavior. Someone else submitted it again today and due to the closeness of times (within a few days), a new item is not created but the old item is boosted.
- freetonik 2y ago
  Ah, thanks for clarifying, did not know this. I thought when someone submits an existing link it either has no effect (and the submitter is redirected to the old post), or a new post is created if enough time had passed since the original submission.
  - bobnamob 2y ago
    There’s also a chance it was placed on the second chance queue and reposted “automatically”

abdullahkhalids 2y ago

You can just use nbclient/nbconvert as a library instead of on cmd. Execute the notebook, and the cell outputs are directly available as python objects [1]. Error handling is also inbuilt. This should make integration with pytest etc, much simpler. [1] https://nbconvert.readthedocs.io/en/latest/execute_api.html#executing-notebooks-using-the-python-api-interface https://nbconvert.readthedocs.io/en/latest/execute_api.html#...

westurner 2y ago
There are a number of ways to run tests within notebooks and to run notebooks as tests. ipytest: https://github.com/chmp/ipytest https://github.com/chmp/ipytest nbval: https://github.com/computationalmodelling/nbval https://github.com/computationalmodelling/nbval papermill: https://github.com/nteract/papermill https://github.com/nteract/papermill awesome-jupyter > Testing: https://github.com/markusschanta/awesome-jupyter https://github.com/markusschanta/awesome-jupyter "Is there a cell tag convention to skip execution?" https://discourse.jupyter.org/t/is-there-a-cell-tag-convention-to-skip-execution/5445/10 https://discourse.jupyter.org/t/is-there-a-cell-tag-conventi... Methods for importing notebooks like regular Python modules: ipynb, import_ipnb, nbimporter, nbdev nbimporter README on why not to import notebooks like modules: https://github.com/grst/nbimporter#update-2019-06-i-do-not-recommend-any-more-to-use-nbimporter https://github.com/grst/nbimporter#update-2019-06-i-do-not-r... nbdev parses jupyter notebook cell comments to determine whether to export a module to a .py file: https://nbdev1.fast.ai/export.html https://nbdev1.fast.ai/export.html There's a Jupyter kernel for running ansible playbooks as notebooks with output. dask-labextension does more of a retry-on-failure workflow

udioron 2y ago

Very cool! One can write a pytest plugin that executes notebooks from a folder with custom pytest fixtures support.

japanuspus 2y ago
You don't even need a plugin: I have been using a setup with parametrized tests and `runpy` for a while [0]. Works especially well if you use jupytext[1] for just storing the input part of the notebook as a python script -- which is good for version control sanity in any case. [0]: https://stackoverflow.com/a/56813896/212538 https://stackoverflow.com/a/56813896/212538 [1]: https://jupytext.readthedocs.io/en/latest/ https://jupytext.readthedocs.io/en/latest/

zurfer 2y ago

for anybody who wants to schedule or automatically run jupyter notebooks I recommend also looking into papermill: https://papermill.readthedocs.io/en/latest/ https://papermill.readthedocs.io/en/latest/

rmholt 2y ago
Agreed! I use it and it's a breath of fresh air, I have to use Jupyter because of colleagues but I really prefer python scripts, and this let's me kind of run a Jupyter notebook as if it were a script, even with cli flags

CJefferson 2y ago

I've used this in a few systems, using (in my case) nbconvert. As I write more code, I increasingly find the most important thing about tests early on is that they are easy to write and maintain. The help that, I find one of the best 'quick test suites' is "run program, save output, run 'git diff' to see if anything changes". This has several advantages. If you have lots of small programs it's trivial to parallelise. It's easy to see what outputs have changed. It's very easy to write weird one-off special tests that need to do something a bit unusual. Yes, eventually you will probably want some nicer test framework, but even then I often keep this framework around, as there will still often be a few tests that don't fit nicely in whatever fancy testing library I'm trying to use (for example, checking a program's front end produces correct error messages when given invalid input).

miohtama 2y ago

Notebooks are a wonderful tool, especially now when Visual Studio Code has superb editor for them. Before this, using web based Jupyter was a bit pain. I use PyCharm for .py files, which is still a bit broken for notebooks, but always go to VSC for notebooks. If someone needs here is an some sample code to run notebooks programically, and tune the output and formatting: https://github.com/tradingstrategy-ai/trade-executor/blob/master/tradeexecutor/backtest/tearsheet.py https://github.com/tradingstrategy-ai/trade-executor/blob/ma...

qsort 2y ago
What's broken about them in pycharm? I've been using pycharm as my primary IDE for a while now and I haven't ever experienced major problems, this includes working on projects where other people were using VSC.
- dehrmann 2y ago
  Using notebooks while remoting (not just a remote Jupyter instance) is broken. It's not this particular bug, but it's similar: https://youtrack.jetbrains.com/issue/PY-71195/Remote-Development-Jupyter-Notebook-not-rendered https://youtrack.jetbrains.com/issue/PY-71195/Remote-Develop...

batmansmk 2y ago

Maintaining e2e tests is a pain. Maintaining a notebook is a pain. It seems it was a given somebody would make this match made in heaven!

vvladymyrov 2y ago
I can suggest improvement - combining long running e2e tests with notebooks is even better match

pplonski86 2y ago

Mixing code, markdown and execution results in one output file gives Jupyter notebooks superpower. You can really have anything in the ipynb file, stored in JSON. I wish there was two types of ipynb files, one for file with just code and markdown (for example ipynbc), and one for keeping code+markdown+results. BTW, sometime ago I wrote an article about surprising things that you can build with Jupyter notebook https://mljar.com/blog/how-to-use-jupyter-notebook/ https://mljar.com/blog/how-to-use-jupyter-notebook/ You will find in the list: blog, book, website, web app, dashboards, REST API, even full packages :)

armanboyaci 2y ago
> I wish there was two types of ipynb files, one for file with just code and markdown (for example ipynbc), and one for keeping code+markdown+results. I believe you can achieve that if you use jupytext library, right?

taeric 2y ago

I find the gigantic swing of a lot of software developers from the pure separation of data and styling into notebooks rather amusing. In particular, there is very little that a "notebook" style environment can get you that you couldn't have gotten as output from any previous testing regime. Styling test results used to be a thing people spent a fair amount of time on so that test reports were pretty and informative. Reports on aggregate results could show trends in how they have been executing in the past. Now, I grant that this article is subtly different. In particular, the notebooks are an artifact that they are testing anyway. So, having more reliance on that technology may itself be a good end goal. I still have a hard time shaking that notebooks are being treated as a hammer looking for nails by so many teams.

tpoacher 2y ago

I really don't get (jupyter) notebooks. It always feels like an over-engineered solution to a simple problem. The only advantage they have over 'proper' code generating a 'proper' report is being able to modify code directly on the report (i.e. as opposed to having to change the code in a separate file instead). Which sounds good and all, but this is rarely what I see notebooks actually used for. For the most part, people use them as an IDE, and a notebook is a terrible choice of IDE compared to even something as simple as editing in nano. And the disadvantages of using one generally completely outweigh the benefit of live-editing in the first place, and encourage all sorts of bad programming habits / hacky code structure. Whenever I've required a report which intermingles code, text, and outputs, a simple bash-script processing my (literate programming) codebase has always done a far more impressive job of generating a useful report. And, without having to sacrifice proper packaging / code-organisation semantics just for the sake of a report. I find it a big shame that the iodide / pyodide notebooks didn't take off in the same way, at least. Those seemed tightly integrated to html in a way that was very elegant. (they're not completely gone though, it was nice to see this example on the front page earlier: https://news.ycombinator.com/item?id=42425489 https://news.ycombinator.com/item?id=42425489 )

deleted 2y ago
[deleted]
parpfish 2y ago
i mostly agree, but one way that i've found to make notebooks good/usable is this: - move all of the code out of the notebook and into a nearby python module where it can be linted/mypy-d/version-controlled/code-reviewed easily - tell the notebook to import that module and make a small number of function calls to get whatever data you need and make plots. at this point the notebook is really nothing more than a REPL with inlined plots/graphics
- youainti 2y ago
  This is what I've found works the best as well.
dr_kiszonka 2y ago

arminiusreturns 2y ago

I prefer emacs org-mode babel.

remram 2y ago

Ever since Marimo launched, I haven't looked back at Jupyter: https://marimo.io/ https://marimo.io/ https://news.ycombinator.com/item?id=38971966 https://news.ycombinator.com/item?id=38971966 On top of their reactive goodness, Marimo notebooks are .py files, which makes it very suitable for this kind of (ab)use.

mscolnick 2y ago

this sort of hackery to work with jupyter notebooks is exactly why we are building marimo (https://github.com/marimo-team/marimo https://github.com/marimo-team/marimo). this should not be the standard