3 ms·

Tools like Jupyter (and Mathematica) don't really match my mental model. I'm fine using them as a REPL (in which case they're like a bloated ipython or ROOT pro

by cozzyd 9y ago

Tools like Jupyter (and Mathematica) don't really match my mental model. I'm fine using them as a REPL (in which case they're like a bloated ipython or ROOT prompt), but as soon as I go back and change something, I get confused about the internal state.

analog31 9y ago
I don't know if it's exactly the same as what you're getting at, but I'm a heavy Jupyter user, and find that as notebooks get more elaborate, I have to work with great care to avoid problems with everything having global scope. So I have to refactor my code once in a while, get things into subroutines, etc. All of my good and bad coding habits are laid bare in Jupyter. Also, this is not for the faint of heart, but before closing a session, I do a "restart kernel and run all." This ensures that my notebook is running the way I'd think if I open it up later and try to re-run it.
- dswalter 9y ago
  It's one of those difficult things. You have to restart and run all to make sure your code is working and your state hasn't gone fudgy, but you also don't want to re-run that cell that took 3 hours to run unless you have to. My personal best practice is to 'restart and run all' to run the entire thing before I commit it to github. Jupyter git integration is a whole separate bag of worms, as the combination of presentation/code violates some core git assumptions.
  - goerz 9y ago
    Any 3-hour calculation should be cached to disk, so that when you re-rerun the notebook you get the result instantaneously
carreau 9y ago
Internal state is often a problem, look at project like stichfix nodebook, and dataflow kernels, they make things a bit easier by re-executing cells. On of the issue is you always have internal state as soon as you interact with a data source or sink. If you read/write from a API, then rest is stateful. Your file system is stateful... etc. It's an interesting but hard problem, we'll be happy to have more help with. https://github.com/dataflownb/dfkernel https://github.com/dataflownb/dfkernel https://multithreaded.stitchfix.com/blog/2017/07/26/nodebook/ https://multithreaded.stitchfix.com/blog/2017/07/26/nodebook...
- gnulinux 9y ago
  Problem with that is, sometimes some cell in the middle is computationally intensive and I don't wanna run it again. Just going back and changing one function shouldn't run the whole computation.
- cozzyd 9y ago
  Thanks for the links, those are interesting solutions.

by cozzyd 9y ago

analog31 9y ago

I don't know if it's exactly the same as what you're getting at, but I'm a heavy Jupyter user, and find that as notebooks get more elaborate, I have to work with great care to avoid problems with everything having global scope. So I have to refactor my code once in a while, get things into subroutines, etc. All of my good and bad coding habits are laid bare in Jupyter. Also, this is not for the faint of heart, but before closing a session, I do a "restart kernel and run all." This ensures that my notebook is running the way I'd think if I open it up later and try to re-run it.

dswalter 9y ago
It's one of those difficult things. You have to restart and run all to make sure your code is working and your state hasn't gone fudgy, but you also don't want to re-run that cell that took 3 hours to run unless you have to. My personal best practice is to 'restart and run all' to run the entire thing before I commit it to github. Jupyter git integration is a whole separate bag of worms, as the combination of presentation/code violates some core git assumptions.
- goerz 9y ago
  Any 3-hour calculation should be cached to disk, so that when you re-rerun the notebook you get the result instantaneously

carreau 9y ago

Internal state is often a problem, look at project like stichfix nodebook, and dataflow kernels, they make things a bit easier by re-executing cells. On of the issue is you always have internal state as soon as you interact with a data source or sink. If you read/write from a API, then rest is stateful. Your file system is stateful... etc. It's an interesting but hard problem, we'll be happy to have more help with. https://github.com/dataflownb/dfkernel https://github.com/dataflownb/dfkernel https://multithreaded.stitchfix.com/blog/2017/07/26/nodebook/ https://multithreaded.stitchfix.com/blog/2017/07/26/nodebook...

gnulinux 9y ago
Problem with that is, sometimes some cell in the middle is computationally intensive and I don't wanna run it again. Just going back and changing one function shouldn't run the whole computation.
cozzyd 9y ago
Thanks for the links, those are interesting solutions.