3 ms·
I also work in scientific computing, and I love Jupyter. I think out-of-order or unknown-order execution is the biggest source of my own mistakes / bugs. If th
by analog31 8y ago
I also work in scientific computing, and I love Jupyter.
I think out-of-order or unknown-order execution is the biggest source of my own mistakes / bugs. If the order in which your cells are executed affects your result, you've got a bug waiting to bite you or someone else. I've been there. And we tend to run cells more than once, while refining our work.
This potentially affects reproducibility, which is a tenet of scientific computing. So it's important in our world.
I personally think there might be a couple of ways to deal with this issue. One is to put handcuffs on Jupyter and inadvertently make it more difficult or cumbersome to use. But this might be valuable if the size of your project and importance of your result warrants it.
Another way is for us to recognize that at some point we have to grow up and learn some techniques from the programmers. In my revisionist history, software engineering began when programs got too big to see the whole thing on one screen and intuitively understand.
One of my little habits is that I do a "restart kernel and run all cells" when I'm ready to take a break or close out at the end of the day. If if fails, the causes of the problem tend to be superficial, and nothing to lose sleep over.
Another habit is to take cells that work and are nice, and enclose their innards in a function so that their impact on the global variable space is minimized. This can be done during a sleepy time in the afternoon when you're not really in flow anyway.
- yodon 8y ago> In my revisionist history, software engineering began when programs got too big to see the whole thing on one screen and intuitively understand. Laughing, politely, at that summary of "history." Software engineering started when there were no screens. Code was written on typewriters (called "terminals") and then printed to cards or long rolls of paper (and occasionally using switches to indicate the 1's and 0's you wanted set as bits in your code). Before the Mac a "screen" of code was pretty universally 24 lines of green or white text on a black background, 80 monospace characters per line. A few specialized terminals had 25x80 so they could show a status line below the 24x80 standard area, but they were rare beasts. Yes, there were a few systems like Plato that predated the Mac but if you need a tl;dr it's the observation that, like most revisionist histories, this one is unfortunately total nonsense when looked at in the context of actual history (even if lots of people who didn't experience the actual history believe it).
- analog31 8y agoQuite agreed, and it's why I deliberately chose the "revisionist" label. Before screens, there were pages, before pages there were cards... going back to plugboards, etc. Each of us got in at a certain time point. When I learned programming, we had a workstation that simulated a card punch, let you see the contents of one card at a time on a little vacuum-fluorescent display, and stored the data on an 8" floppy. You handed the floppy to the computer operator, and received a printout on green-bar paper. The other way I got into "programming" was staring at a nest of wires, reading my output on an oscilloscope, and making corrections with a soldering iron. ;-) The common theme was that a program gets too large for somebody to look at and understand it, much less for a team to work on it all at once, unless disciplines are adopted that tend to come from software engineering. When I'm mentoring colleagues who are just beginning to get into scientific programming, some of these simple techniques, such as avoiding globals and creating functions, are in fact an improvement.
- yodon 8y agoIt sounds like instead of putting the sentence in past tense and calling it a revisionist history it sounds like what you are really doing is making a present tense generalization based on years of historical observation: software engineering begins when the code gets too big to see the whole thing on one screen and intuitively understand (aka Analog31's law)
- analog31 8y agoYes, that's perfect.
- jpeloquin 8y ago> I think out-of-order or unknown-order execution is the biggest source of my own mistakes / bugs. To avoid the state problem, I start each cell (or, in org-mode, each src block) with a command to set up the environment for that cell, usually by importing a script named something like setup_abc123.py. Cells are not allowed to reference code or results from other cells except through the filesystem, which is under the watchful eye of git. In this workflow, cells exist only to write plots or tables into the notebook for human consumption. This workflow, for me, has completely eliminated errors due to hidden state dependencies. Before I adopted it I made a serious state-related error every couple of days, which was intolerable. As a bonus, using a single script to set up the analysis environment makes it trivial to start a new analysis, even with an entirely different tool chain.