4 ms·

I love how notebooks allow the mixing of code and output and support incremental development by letting you choose which cells to execute. But I find the semant

I love how notebooks allow the mixing of code and output and support incremental development by letting you choose which cells to execute. But I find the semantics horrible. Each time you execute a cell you do so in an environment that depends on your entire history and cannot be figured out by simply reading the notebook. I wish for a environment which would have the same semantics as a script but which would snapshot the environment at the entry to each cell so that when a cell is modified execution does not have to resume at the beginning. Even better if downstream data dependencies are tracked so that after modifying and reexecuting a cell we know which downstream results have become stale. Does such an environment exist?

98Windows 10y ago
That's why I've just resorted to using 'Restart & Run All' as my default way to run a notebook I've finished coding up.
mjlm 10y ago
I think this persistent state is one of the main advantages of the notebook environment, or the Matlab workspace, which I guess it was inspired by. It allows you to quickly try alternative values for certain variables without having to re-calculate everything. Saving snapshots would not be feasible if the project contains large amounts of data. If you want to reset everything, just "run all" from the beginning, or use a conventional IDE with a debugger.
- yomritoyj 10y ago
  But you cannot recalculate if your calculation has trashed your inputs.And if it hasn't then the snapshot does not impose a cost. If you are willing to forego the opportunity to replay to save memory, just put the producer and consumer in the same cell.
  - lqdc13 10y ago
    It is essentially already that if you are careful with the variables. Currently, you can easily control which things are saved and which aren't. So it is the best of both worlds.
- rhodin 10y ago
  No, not Matlab but Mathematica: "We were inspired originally by the excellent implementation in Mathematica" [1]. [1] http://ipython.org/ipython-doc/dev/whatsnew/version0.12.html#an-interactive-browser-based-notebook-with-rich-media-support http://ipython.org/ipython-doc/dev/whatsnew/version0.12.html...
  - brians 10y ago
    And that came from Emacs and old Lisp environments---and perhaps something yet earlier? As late as 2000, this was the single biggest advantage and single biggest impediment to new programmers in MIT's 6.001 lab: a bunch of nonvisible state, mutated by every C-x C-e. The student has tweaked two dozen points trying to fix a small program, and re-evaluated definitions after many of them, but maybe not all. The most straightforward help from a teacher is to get the buffer into a form such that M-x eval-region paves over all that, sets a known environment of top level definitions, and---more than half the time---the student's code now works. I have similar concerns about much of Victor's work, for the same reason. Managing a mental model of complex state is a n important skill for programming, but it's best learned incrementally over long experience with more complex programs. These very interactive environments front load the need for that skill without giving any obvious structure for helping the student learn. Contrast Excel and HyperCard, which have no invisible state: you can click and see everything.
cdavid 10y ago
tonicdev seems to do it, though I have never used it: http://blog.tonicdev.com/2015/09/10/time-traveling-in-node.js-notebooks.html http://blog.tonicdev.com/2015/09/10/time-traveling-in-node.j... note: it does undo, but not recalculation on demand ala excel.
dalailambda 10y ago
I imagine with a little introspection, a notebook can figure out dependencies between cells and cascade as needed.
- yomritoyj 10y ago
  I guess the challenge is to do it efficiently in the presence of pointers, implicit or explicit.
  - brians 10y ago
    There's great work on this called differential dataflow. You have to trace at a finer grain than cells, but can then avoid lots of recalculation.
denfromufa 10y ago
What you describe looks like reactive programming and lazy evaluation and Mathematica has support for this using the notion of Dynamic. This is how actually Manipulate is implemented.
thanatropism 10y ago
The real simple thing would be to paint a border or just a sideline in cells that have already been executed. Just that would make everything much, much better.
- jasongrout 10y ago
  Thanks, that makes a lot of sense. I've opened an issue: https://github.com/jupyter/jupyterlab/issues/405 https://github.com/jupyter/jupyterlab/issues/405.

98Windows 10y ago

That's why I've just resorted to using 'Restart & Run All' as my default way to run a notebook I've finished coding up.

mjlm 10y ago

I think this persistent state is one of the main advantages of the notebook environment, or the Matlab workspace, which I guess it was inspired by. It allows you to quickly try alternative values for certain variables without having to re-calculate everything. Saving snapshots would not be feasible if the project contains large amounts of data. If you want to reset everything, just "run all" from the beginning, or use a conventional IDE with a debugger.

yomritoyj 10y ago
But you cannot recalculate if your calculation has trashed your inputs.And if it hasn't then the snapshot does not impose a cost. If you are willing to forego the opportunity to replay to save memory, just put the producer and consumer in the same cell.
- lqdc13 10y ago
  It is essentially already that if you are careful with the variables. Currently, you can easily control which things are saved and which aren't. So it is the best of both worlds.
rhodin 10y ago
No, not Matlab but Mathematica: "We were inspired originally by the excellent implementation in Mathematica" [1]. [1] http://ipython.org/ipython-doc/dev/whatsnew/version0.12.html#an-interactive-browser-based-notebook-with-rich-media-support http://ipython.org/ipython-doc/dev/whatsnew/version0.12.html...
- brians 10y ago
  And that came from Emacs and old Lisp environments---and perhaps something yet earlier? As late as 2000, this was the single biggest advantage and single biggest impediment to new programmers in MIT's 6.001 lab: a bunch of nonvisible state, mutated by every C-x C-e. The student has tweaked two dozen points trying to fix a small program, and re-evaluated definitions after many of them, but maybe not all. The most straightforward help from a teacher is to get the buffer into a form such that M-x eval-region paves over all that, sets a known environment of top level definitions, and---more than half the time---the student's code now works. I have similar concerns about much of Victor's work, for the same reason. Managing a mental model of complex state is a n important skill for programming, but it's best learned incrementally over long experience with more complex programs. These very interactive environments front load the need for that skill without giving any obvious structure for helping the student learn. Contrast Excel and HyperCard, which have no invisible state: you can click and see everything.

cdavid 10y ago

tonicdev seems to do it, though I have never used it: http://blog.tonicdev.com/2015/09/10/time-traveling-in-node.js-notebooks.html http://blog.tonicdev.com/2015/09/10/time-traveling-in-node.j... note: it does undo, but not recalculation on demand ala excel.

dalailambda 10y ago

I imagine with a little introspection, a notebook can figure out dependencies between cells and cascade as needed.

yomritoyj 10y ago
I guess the challenge is to do it efficiently in the presence of pointers, implicit or explicit.
- brians 10y ago
  There's great work on this called differential dataflow. You have to trace at a finer grain than cells, but can then avoid lots of recalculation.

denfromufa 10y ago

What you describe looks like reactive programming and lazy evaluation and Mathematica has support for this using the notion of Dynamic. This is how actually Manipulate is implemented.

thanatropism 10y ago

The real simple thing would be to paint a border or just a sideline in cells that have already been executed. Just that would make everything much, much better.

jasongrout 10y ago
Thanks, that makes a lot of sense. I've opened an issue: https://github.com/jupyter/jupyterlab/issues/405 https://github.com/jupyter/jupyterlab/issues/405.