3 ms·

Hi -- author here. I just presented this work at JupyterCon 2023 so I figured it was time to advertise more broadly. There are a few rough edges but my hope is

by smacke 3y ago

Hi -- author here. I just presented this work at JupyterCon 2023 so I figured it was time to advertise more broadly. There are a few rough edges but my hope is that, by making all the reactive behavior opt-in and only enabled for in-order execution (i.e., cells above the one I execute will never reactively execute by default), it can be predictable enough to be useful in practice. There's still a long way to go to get e.g. full dataflow understanding of all the common libraries, understanding file paths, autoreload integration, etc., but after nearly 3 years of on-and-off development I think it's finally useable-ish.

analog31 3y ago
I'll give this a try. Managing "hidden state" in notebooks is a known flaw of Jupyter. If nothing else, an indicator that says, "this code is dirty" would be useful. I have a long standing habit of doing "restart kernel and run all cells" before walking away from a session, to help avoid this. I'd rather see it break in front of me than have it break 6 months later or in someone else's use.
Micoloth 3y ago
Crazy seeing this here! I searched for this last week, as I'm playing with building the same thing but as a VSCode extension.. See here [1] I found another similar project on Github, but it was from many years ago. Yours did not turn up.. Very interested in finding out how you implemented it [1] https://github.com/micoloth/vscode-reactive-jupyter#readme https://github.com/micoloth/vscode-reactive-jupyter#readme
- smacke 3y ago
  We have a couple of papers that go into some of the details. https://smacke.net/papers/nbsafety.pdf https://smacke.net/papers/nbsafety.pdf https://smacke.net/papers/nbslicer.pdf https://smacke.net/papers/nbslicer.pdf It looks like you are using a static approach for dependency inference. There are a lot of benefits to static approaches, but they can only get you so far. My JupyterCon presentation includes a bunch of examples where dynamic approaches are a must: https://t.ly/78rS https://t.ly/78rS Besides that, there are a bunch of interesting design decisions about when to add edges between cells, when to break them, what metadata to annotate edges with, etc. I'm hoping to abstract a way a bunch of the complexity by developing something like a runtime version of a language server protocol (working name "language kernel protocol") so that any editor that implements the protocol would get reactivity for free when running a kernel that likewise implements the protocol. I have an early version of this which is how IPyflow works for both Jupyter and JupyterLab; VSCode would be a great editor to add support for next.
eigenspace 3y ago
Will this approach ever be usable with other Jupyter langauges? Like, do you have an API for another language to tell you what the code dependency graph is? Or is Python a fundamental assumption here?
- smacke 3y ago
  For this particular project, Python is a requirement. For the general approach, the answer is more complicated. It depends on what hooks the language implementation exposes -- and even if it exposes enough to make this work in theory, tracking dataflow at the same level of accuracy and granularity as IPyflow does may not be possible without taking an unacceptable performance hit, or without sacrificing portability across language versions. My hope is that the approach can scale to languages like Julia or R, but I'm not as familiar with those languages as I am with Python, and I kind of suspect each language may require its own bespoke tricks. Regardless, for Python it was a journey roughly 3 years in the making (and still ongoing) -- other languages would be easier now that I've learned a fair amount, but the work to add this kind of support is by far the most complex I've ever done.
spenrose 3y ago
Kudos! When at Mozilla c.2016, I tried to work with the core Jupyter team on solving the stale-cell problem. I couldn't find a path forward that they would consider. Glad to see someone making progress.
- smacke 3y ago
  I would be very surprised if something like this gets support in core Jupyter -- there's a lot of added complexity. Fortunately it is doable as extensions for Jupyter / JupyterLab.

by smacke 3y ago

analog31 3y ago

I'll give this a try. Managing "hidden state" in notebooks is a known flaw of Jupyter. If nothing else, an indicator that says, "this code is dirty" would be useful. I have a long standing habit of doing "restart kernel and run all cells" before walking away from a session, to help avoid this. I'd rather see it break in front of me than have it break 6 months later or in someone else's use.

Micoloth 3y ago

Crazy seeing this here! I searched for this last week, as I'm playing with building the same thing but as a VSCode extension.. See here [1] I found another similar project on Github, but it was from many years ago. Yours did not turn up.. Very interested in finding out how you implemented it [1] https://github.com/micoloth/vscode-reactive-jupyter#readme https://github.com/micoloth/vscode-reactive-jupyter#readme

smacke 3y ago
We have a couple of papers that go into some of the details. https://smacke.net/papers/nbsafety.pdf https://smacke.net/papers/nbsafety.pdf https://smacke.net/papers/nbslicer.pdf https://smacke.net/papers/nbslicer.pdf It looks like you are using a static approach for dependency inference. There are a lot of benefits to static approaches, but they can only get you so far. My JupyterCon presentation includes a bunch of examples where dynamic approaches are a must: https://t.ly/78rS https://t.ly/78rS Besides that, there are a bunch of interesting design decisions about when to add edges between cells, when to break them, what metadata to annotate edges with, etc. I'm hoping to abstract a way a bunch of the complexity by developing something like a runtime version of a language server protocol (working name "language kernel protocol") so that any editor that implements the protocol would get reactivity for free when running a kernel that likewise implements the protocol. I have an early version of this which is how IPyflow works for both Jupyter and JupyterLab; VSCode would be a great editor to add support for next.

eigenspace 3y ago

Will this approach ever be usable with other Jupyter langauges? Like, do you have an API for another language to tell you what the code dependency graph is? Or is Python a fundamental assumption here?

smacke 3y ago
For this particular project, Python is a requirement. For the general approach, the answer is more complicated. It depends on what hooks the language implementation exposes -- and even if it exposes enough to make this work in theory, tracking dataflow at the same level of accuracy and granularity as IPyflow does may not be possible without taking an unacceptable performance hit, or without sacrificing portability across language versions. My hope is that the approach can scale to languages like Julia or R, but I'm not as familiar with those languages as I am with Python, and I kind of suspect each language may require its own bespoke tricks. Regardless, for Python it was a journey roughly 3 years in the making (and still ongoing) -- other languages would be easier now that I've learned a fair amount, but the work to add this kind of support is by far the most complex I've ever done.

spenrose 3y ago

Kudos! When at Mozilla c.2016, I tried to work with the core Jupyter team on solving the stale-cell problem. I couldn't find a path forward that they would consider. Glad to see someone making progress.

smacke 3y ago
I would be very surprised if something like this gets support in core Jupyter -- there's a lot of added complexity. Fortunately it is doable as extensions for Jupyter / JupyterLab.