6 ms·
Papermill: Parameterizing, executing, and analyzing Jupyter Notebooks
- edublancas 2y agoPapermill is great but has quite some limitations because it spins up a new process to run the notebook: - You cannot extract live variables (needed for testing) - Cannot use pdb for debugging - Cannot profile memory usage You can do all of that with ploomber-engine (https://github.com/ploomber/ploomber-engine https://github.com/ploomber/ploomber-engine). Disclaimer: I'm the author of this package
- ziddoap 2y agoNot disclosed in this comment is that edublancas is >Ploomber (YC W22) co-founder.
- Kalanos 2y agowho is a great technologist with a lot of hands on experience. if it made sense to leverage papermill, he would have done so and focused on something else.
- ziddoap 2y agoWhat does any of this have to do with disclosure?
- Kalanos 2y agocalling attention to disclosure suggests bias. i'm obviously saying that i trust him not to be biased.
- fastasucan 2y agoNot obvious at all. And no. It doesn't have to suggest bias, just a lack of disclosure.
- throwpoaster 2y agoiirc, a few years back I was able to do all of these things with the Papermill IPython runtime. Papermill is great, but yes: lots of room to hack on it and make it better.
- edublancas 2y agohas papermill deprecated the ipython runtime? I used papermill extensively in the past and I never saw that in their docs.
- throwpoaster 2y agoIt’s been a while but you do it with a custom kernel and maybe some entry point tweaks. IIRC.
- mcpar-land 2y agoWhat is the benefit of parameterizing a jupyter notebook over just writing python that's not in a jupyter notebook? I like jupyter notebooks for rapid prototyping but once I want to dial some logic in, I switch to just writing a .py file.
- zhoujing204 2y agoIt might be a pretty useful tool for education. College courses related to Python and AI on Coursera have heavily used Jupyter Notebook for assignments and labs.
- jsemrau 2y agoI used papermill a while ago to automate a long-running python-based data aggregation task. Airflow would log in remotely to the server, kick-off papermill and track it's progress. Initially I wanted to use pure python, but the connection disconnected frequently disallowing me to track the progress, and also jupyter enabled quick debugging where something went wrong. Not one of my proudest moments, but it got the job done.
- __MatrixMan__ 2y agoI think there are places where the figure-it-out-in-a-notebook part is one person's job, and then including it in a pipeline is another person's job. If they can call the notebook like a function, the second person's job becomes much easier.
- crabbone 2y agoI've been that person, and no it doesn't. It makes my life suck, if I have to include a notebook instead of an actual program in a larger program. Notebooks don't compose well, they are too dependent on the specifics of the environment in which they were launched, they have excessive source code that's also machine-generated and is hard to work with for humans. As a stop-gap solution, for cases like a single presentation / proof-of-concept that doesn't need to live on and be reused -- it would work. Anything that doesn't match this description will accumulate technical debt very quickly.
- notpushkin 2y ago> Do you want to run a notebook and depending on its results, choose a particular notebook to run next? Hell no. I want to rewrite all that as a proper script or Python module.
- p4ul 2y agoIndeed! I feel like we as a community have taken a wrong turn with our use of notebooks. I think they have benefits in some specific use cases (e.g., teaching, demos, etc.), but otherwise, I think they mostly encourage bad practices for software development.
- fastasucan 2y agoLuckily each person can descide what is best for their use case and does not need to follow what the "community" does.
- morkalork 2y agoI once built an unholy combination of papermill and nbconvert to mass produce monthly reports using a "template" notebook. All the code was imported from a .py file so the template just took a client ID as input and called out to render_xyz(...) in each section. It was nice because it produced a bunch of self-contained static files and wrote them to a network drive. It was definitely a solution to the problem.
- pplonski86 2y agoI have few ML pipelines that are simply using nbconvert to execute notebooks. Regarding python script vs notebook debate I think it all depends on your use case. I like that I can display plots in notebooks without any additional work.
- v3ss0n 2y agoHow hard it is copy , paste and run the note book code within proper http server?
- sa-code 2y agoThere is some utility in seeing the code and the output right below it.
- miohtama 2y agoI looked Papermill back in a day, but found it easier call nbclient and nbconvert directly https://github.com/tradingstrategy-ai/trade-executor/blob/master/tradeexecutor/backtest/tearsheet.py https://github.com/tradingstrategy-ai/trade-executor/blob/ma...
- Kalanos 2y agoIs this still being developed? The last commit to the main library was 5 months ago and its tied to exceptions/tests.
- barrrrald 2y agoSeems like it's mostly died off, most people I know have moved to hosted solutions like Hex or Colab
- reeboo 2y agoIt's a thin wrapper around notebooks. Does it really need more features? Not saying that it couldn't, but it is feature complete for what its job is.
- Kalanos 2y agothings break due to shifting dependencies. also, if it isn't maintained by the company that made it, then it is a good sign that they are no longer using it. it suggests that there is a better solution elsewhere.
- fluidcruft 2y agoMatthew Seal who was a major driving force behind papermill went on to create notable.io which shut down a few months ago. https://news.ycombinator.com/item?id=38659315 https://news.ycombinator.com/item?id=38659315 A lot of people seemed to move to nbdev which tried to improve on many aspects of notebook dev culture, but nbdev's maintainers include GitHub employees who are can only comprehend doing anything via GitHub integrations and get huffy and how-dare-you-take-that-tone-with-us if you ever ask about how you would go about doing something without relying on GitHub. So personally I've just been focusing on nbconvert and rolling my own things.
- iamleppert 2y agoJupyter notebooks are missing strict types, a linter and unit tests. When can those features be added?
- big-chungus4 2y agovscode jupyter uses the same extensions as vscode, so you can get a linter and scrict type checking. Not sure about tests though
- ogrisel 2y agoWith papermill you can parametrize a notebook and run it on different inputs to check that it is not raising uncaught exceptions. This can be wrapped to be part of a pytest test suite, possibly via a some ad-hoc pytest fixture or plugin. If the notebooks themselves contain assertions to check that expectations on the outputs are met, then you have an automated way to check that the notebooks behave the way you want on some test inputs. For long notebooks, this is more like integration/functional tests rather than unit tests, but I think this is already an improvement over manually run notebooks. Note sure about strict types: you mean running mypy on a notebook? Maybe this can be helpful: - https://pypi.org/project/nb-mypy/ https://pypi.org/project/nb-mypy/ About linters, you can install `jupyterlab-lsp` and `python-lsp-ruff` together for instance.
- jamesblonde 2y agoFor MLOps platforms, Papermill is the one of the reasons why we don't include experiment tracking out of the box any longer in Hopsworks. You can easily see the results of training runs as notebooks - including loss curves, etc. Any models that completed get registered in the model registry along with plots, a model card, and model evaluation/validation metrics.
- __mharrison__ 2y agoI teach a lot using Jupyter. It is certainly possible to use SWE worst practices in Jupyter easily. I am often in front of folks who "aren't computer programmers" but need to use Python tools to be successful. One of my covert goals is to teach SWE best practices inside of notebooks. It requires a little more typing but eases the use of notebooks, refactoring, testing, moving to scripts, and using tooling like Papermill.
- akshayka 2y agoHave you considered using marimo notebooks? https://github.com/marimo-team/marimo https://github.com/marimo-team/marimo marimo notebooks are stored as pure Python (executable as scripts, versionable with git), and they largely eliminate the hidden state problem that affects Jupyter notebooks -- delete a variable and it's automatically removed from program memory, run a cell and all other cells that use its variables are marked as stale. marimo notebooks are also readily parametrized with CLI arguments, so you can do: python notebook.py -- -foo 1 -bar 2 ... Disclosure: I'm a marimo developer.
- ThouYS 2y agothanks mate, exactly what I've been looking for
- cycomanic 2y agoThere is also jupytext which converts Jupyter notebooks on the fly to a number of different formats (Markdown, python,...). It's at the core of the Jupyterbook project IIRC and IMO the best method to use Jupyter with git.
- __mharrison__ 2y agoI use Jupytext (and my own conversion utilities) all the time. I write my books inside of Jupyter these days.
- sa1 2y agoYou should make a new post about the new sandbox feature.
- cmcconomy 2y agoIn the past, I've used this to generate HTML outputs to reflect a series of calculations and visualizations so that we can share with clients.
- etbebl 2y agoAm I the only one who cringes at the name of this tool? The implication being that it's a way to churn out low-quality papers, probably with some p-hacking along the way.