3 ms·

Simply doing “restart + run all” ensures both readability and reproducibility

orwin 3y ago
If you have four of five sklearn tiles that take 20 minute to an hour to execute, you might not want to do that too often. I did not use jupiter notebook/jupyterLab much, but each time, it was in the context of datascience. The first was on an OCR during my internship, the second for data exploration (mix of quantitative/qualitative, but the project was scrapped after a week or two). In both case, having to re-run all each time the kernel was shut down was actually a pain point.
- singhrac 3y ago
  I encountered this a lot in my previous work and found workarounds that I write about here: https://rachitsingh.com/collaborating-jupyter/ https://rachitsingh.com/collaborating-jupyter/. At a high level I think being scared of rerunning your kernel is indicative of a code smell, and there's relatively easy ways to combat that.
  - smacke 3y ago
    Personally I don't like the "write to disk" approach; I think it kind of just punts the state problem somewhere else (i.e. from memory to disk). Writing to a database and adding versioning is better, but that's a lot of machinery to expect a notebook user to adopt (though maybe better tooling could help). Also a lot of Python objects are not out-of-the-box pickleable (e.g. generators). Also pickle is a mess.
    - singhrac 3y ago
      I definitely agree (and I think given what you work on you would be horrified by how I define cached functions that capture locals), but I think in practice getting to a state where you can restart your kernel often makes it easier to reason about state. But you’re definitely right, it would be better to reason correctly here. One thing I’ve toyed with is writing a Jupyter kernel extension that notes what new locals you’ve defined in a cell, figures out what locals are read, and creates a (cached) function from the cell. E.g. a cell that has `y = a @ x + b` becomes @cache_to_disk def compute_y(a, x, b): return a @ x + b y = compute_y(a, x, b) I don’t worry much about serialization - 90% of the time what I need to cache is dataframes (write to parquet), and the rest is trained models (custom serializer). People rarely need to cache generators, in my opinion.
  - orwin 3y ago
    Yeah, I was definitely a python newb, barely used generators or context managers. I ran them once every morning still.
- TheAlchemist 3y ago
  Yes indeed - not having to re-run expensive calculations is one of the major appeals of the notebooks. But still, I believe it should be done reasonably often - at least once, before committing for example (if you're using tools like jupytext to convert notebooks to .py)

If you have four of five sklearn tiles that take 20 minute to an hour to execute, you might not want to do that too often. I did not use jupiter notebook/jupyterLab much, but each time, it was in the context of datascience. The first was on an OCR during my internship, the second for data exploration (mix of quantitative/qualitative, but the project was scrapped after a week or two). In both case, having to re-run all each time the kernel was shut down was actually a pain point.

singhrac 3y ago
I encountered this a lot in my previous work and found workarounds that I write about here: https://rachitsingh.com/collaborating-jupyter/ https://rachitsingh.com/collaborating-jupyter/. At a high level I think being scared of rerunning your kernel is indicative of a code smell, and there's relatively easy ways to combat that.
- smacke 3y ago
  Personally I don't like the "write to disk" approach; I think it kind of just punts the state problem somewhere else (i.e. from memory to disk). Writing to a database and adding versioning is better, but that's a lot of machinery to expect a notebook user to adopt (though maybe better tooling could help). Also a lot of Python objects are not out-of-the-box pickleable (e.g. generators). Also pickle is a mess.
  - singhrac 3y ago
    I definitely agree (and I think given what you work on you would be horrified by how I define cached functions that capture locals), but I think in practice getting to a state where you can restart your kernel often makes it easier to reason about state. But you’re definitely right, it would be better to reason correctly here. One thing I’ve toyed with is writing a Jupyter kernel extension that notes what new locals you’ve defined in a cell, figures out what locals are read, and creates a (cached) function from the cell. E.g. a cell that has `y = a @ x + b` becomes @cache_to_disk def compute_y(a, x, b): return a @ x + b y = compute_y(a, x, b) I don’t worry much about serialization - 90% of the time what I need to cache is dataframes (write to parquet), and the rest is trained models (custom serializer). People rarely need to cache generators, in my opinion.
- orwin 3y ago
  Yeah, I was definitely a python newb, barely used generators or context managers. I ran them once every morning still.
TheAlchemist 3y ago
Yes indeed - not having to re-run expensive calculations is one of the major appeals of the notebooks. But still, I believe it should be done reasonably often - at least once, before committing for example (if you're using tools like jupytext to convert notebooks to .py)