4 ms·

nbdev looks promising. I'm wondering if it solves what I see as the biggest pain of using notebooks when I'm doing data science work. When a notebook gets larg

nbdev looks promising. I'm wondering if it solves what I see as the biggest pain of using notebooks when I'm doing data science work. When a notebook gets large, it can be difficult to keep track of dependencies between cells. For workflows in which you have to run cells n_1, n_2, ..., n_k before running cell n. I try to organize my cells so that if I run them from first to last, all dependencies are covered (e.g. "Restart kernel and run all cells). Unfortunately, this doesn't help when I discover a bug in cell n_2 and don't want to run ALL cells n_2 + 1, ..., n-1, n because some of them carry out expensive operations. When working in my editor, the way I resolve this is to make a light CLI wrapper around my program (if __name__ == "__main__": import argparse; ...) and my CLI commands encode all this dependency information. Is it possible to get this kind of experience in a Jupyter notebook without building a custom plugin (I think a frontend plugin would suffice)?

Jugurtha 6y ago
>Unfortunately, this doesn't help when I discover a bug in cell n_2 and don't want to run ALL cells n_2 + 1, ..., n-1, n because some of them carry out expensive operations. Running all cells has its disadvantages, but what I found out was that often, there are bugs elsewhere than in cell n_2. I treat that as unit tests: run them all because even though I think only this is breaking, fixing it could have broken some other part. Many other "Jupyter killers", as sensationalist blog post titles call them, claim they have done away with Jupyter's dirty "hidden state", but reading further about how the "Jupyter killer" avoids re-doing heavy computation by "caching" compute-intensive results and you want to tell them "get your mind right, which is it?". The way we do it is schedule the notebook[0] to make sure everything works. - [0]: https://iko.ai/docs/notebook/#long-running-notebooks https://iko.ai/docs/notebook/#long-running-notebooks

Jugurtha 6y ago

>Unfortunately, this doesn't help when I discover a bug in cell n_2 and don't want to run ALL cells n_2 + 1, ..., n-1, n because some of them carry out expensive operations. Running all cells has its disadvantages, but what I found out was that often, there are bugs elsewhere than in cell n_2. I treat that as unit tests: run them all because even though I think only this is breaking, fixing it could have broken some other part. Many other "Jupyter killers", as sensationalist blog post titles call them, claim they have done away with Jupyter's dirty "hidden state", but reading further about how the "Jupyter killer" avoids re-doing heavy computation by "caching" compute-intensive results and you want to tell them "get your mind right, which is it?". The way we do it is schedule the notebook[0] to make sure everything works. - [0]: https://iko.ai/docs/notebook/#long-running-notebooks https://iko.ai/docs/notebook/#long-running-notebooks