4 ms·
Papermill is great but has quite some limitations because it spins up a new process to run the notebook: - You cannot extract live variables (needed for testin
by edublancas 2y ago
Papermill is great but has quite some limitations because it spins up a new process to run the notebook:
- You cannot extract live variables (needed for testing)
- Cannot use pdb for debugging
- Cannot profile memory usage
You can do all of that with ploomber-engine (https://github.com/ploomber/ploomber-engine https://github.com/ploomber/ploomber-engine).
Disclaimer: I'm the author of this package
- ziddoap 2y agoNot disclosed in this comment is that edublancas is >Ploomber (YC W22) co-founder.
- Kalanos 2y agowho is a great technologist with a lot of hands on experience. if it made sense to leverage papermill, he would have done so and focused on something else.
- ziddoap 2y agoWhat does any of this have to do with disclosure?
- Kalanos 2y agocalling attention to disclosure suggests bias. i'm obviously saying that i trust him not to be biased.
- fastasucan 2y agoNot obvious at all. And no. It doesn't have to suggest bias, just a lack of disclosure.
- throwpoaster 2y agoiirc, a few years back I was able to do all of these things with the Papermill IPython runtime. Papermill is great, but yes: lots of room to hack on it and make it better.
- edublancas 2y agohas papermill deprecated the ipython runtime? I used papermill extensively in the past and I never saw that in their docs.
- throwpoaster 2y agoIt’s been a while but you do it with a custom kernel and maybe some entry point tweaks. IIRC.