4 ms·

Maybe I can provide a bit of perspective on this, as I have lots of conflicting feelings about Jupyter. When I'm doing some bit of data wrangling or just explo

Maybe I can provide a bit of perspective on this, as I have lots of conflicting feelings about Jupyter. When I'm doing some bit of data wrangling or just exploratory work with data I quite like it - at least at first. As you pointed out, it's really easy to extremely quickly iterate on things and start to get an idea of what's in the data, what techniques work, and which don't. It's great. Until it isn't. I'm probably "using it wrong" but all the suggestions I've seen for doing it "right" start cutting into the iteration time. If I'm in Jupyter it's because I want to flail around real fast and see what happens. And that causes all sorts of problems. Which cells do I need to rerun together? Which cells should I not run again. What the heck is actually in all these variables right now anyway, because I don't remember which order I've run (and rerun) all the cells in. And what was that one approach or parameter that really worked well that one time? I don't remember. These sorts of things aren't an issue with a non-Jupyter approach. And because I'm taking my time anyway I'm probably being diligent with source control, and all that. Even pulling code out of a notebook into a more stable workflow is a pain, because you have no assurances you can just copy/paste a cell and have it work. In my experience, it never will. It's one of those things that has a pretty heavy costs and benefits. And unfortunately they aren't really possible to separate. Not coincidentally, I feel the same about dynamic languages and even less usefully typed static languages. Jupyter takes an already fairly extreme position on the stable(safe)/easy continuum and really ramps it up to a point where it's hard to wrangle time and work done easy into some sort of stability. It's great, and it's awful.

williamstein 8y ago
> And what was that one approach or parameter that really worked well that one time? I gave a talk at JupyterCon this week about Cocalc which partly solves this one problem by providing a Time travel slider with complete history.
- aportnoy 8y ago
  I took abstract algebra with Prof. Kedlaya at UCSD who likes to use Cocalc for his other courses, haven't used it though.
aportnoy 8y ago
> Which cells do I need to rerun together? Which cells should I not run again. What the heck is actually in all these variables right now anyway, because I don't remember which order I've run (and rerun) all the cells in. And what was that one approach or parameter that really worked well that one time? I don't remember. Right. Get that all the time. My solution is to try to condense the useful code built up across different cells into a reusable block, most often a function. Do that while the 'state' is still fresh in your mind and you remember the order of execution. It also helps to break up a long notebook into sections using headings and markdown cells with comments. I often start by writing down a question/problem I am trying to solve in prose, that makes it easier to recover the context of the code cells that follow.
- dnbgfher 8y ago
  > My solution is to try to condense the useful code built up across different cells into a reusable block, most often a function. Do that while the 'state' is still fresh in your mind and you remember the order of execution. Right, that's the sort of thing I was talking about in terms of "using it right." It makes sense and is really a good idea, it just starts eating into the advantages you get from using Jupyter in the first place. I'm still trying to figure out how best to integrate it in my workflow. So far I've mostly settled on using it to get some rough ideas about the data and libraries I'm using, and then just reworking from scratch in a more stable setting. As needed I'll go back and try things out there, possibly in a new notebook entirely and just do my best to copy/paste the setup over. In getting it working the setup stuff at least gets consolidated. It's... well awful but eh, I haven't really found a better way.
moultano 8y ago
I think notebooks could be hugely improved by saving a snapshot of state after each cell, and having the linear order of cells only move forward in time. Accidental variable reuse (especially from a cell that is in the "future" or maybe no longer exists) is a huge source of bugs.
- dnbgfher 8y ago
  From other comments, it sounds like something called Cocalc might be of interest to you. I haven't even looked at it yet but it sounds like it does at least some of those things.

williamstein 8y ago

> And what was that one approach or parameter that really worked well that one time? I gave a talk at JupyterCon this week about Cocalc which partly solves this one problem by providing a Time travel slider with complete history.

aportnoy 8y ago
I took abstract algebra with Prof. Kedlaya at UCSD who likes to use Cocalc for his other courses, haven't used it though.

aportnoy 8y ago

> Which cells do I need to rerun together? Which cells should I not run again. What the heck is actually in all these variables right now anyway, because I don't remember which order I've run (and rerun) all the cells in. And what was that one approach or parameter that really worked well that one time? I don't remember. Right. Get that all the time. My solution is to try to condense the useful code built up across different cells into a reusable block, most often a function. Do that while the 'state' is still fresh in your mind and you remember the order of execution. It also helps to break up a long notebook into sections using headings and markdown cells with comments. I often start by writing down a question/problem I am trying to solve in prose, that makes it easier to recover the context of the code cells that follow.

dnbgfher 8y ago
> My solution is to try to condense the useful code built up across different cells into a reusable block, most often a function. Do that while the 'state' is still fresh in your mind and you remember the order of execution. Right, that's the sort of thing I was talking about in terms of "using it right." It makes sense and is really a good idea, it just starts eating into the advantages you get from using Jupyter in the first place. I'm still trying to figure out how best to integrate it in my workflow. So far I've mostly settled on using it to get some rough ideas about the data and libraries I'm using, and then just reworking from scratch in a more stable setting. As needed I'll go back and try things out there, possibly in a new notebook entirely and just do my best to copy/paste the setup over. In getting it working the setup stuff at least gets consolidated. It's... well awful but eh, I haven't really found a better way.

moultano 8y ago

I think notebooks could be hugely improved by saving a snapshot of state after each cell, and having the linear order of cells only move forward in time. Accidental variable reuse (especially from a cell that is in the "future" or maybe no longer exists) is a huge source of bugs.

dnbgfher 8y ago
From other comments, it sounds like something called Cocalc might be of interest to you. I haven't even looked at it yet but it sounds like it does at least some of those things.