4 ms·

I've been using Python for 15 years, and I also haven't gotten into Jupyter. I worked with some research scientists who used it, but even they ported their pro

by chubot 9y ago

I've been using Python for 15 years, and I also haven't gotten into Jupyter. I worked with some research scientists who used it, but even they ported their programs to plain text files in order to get version control with textual diffs (which is IMO better for collaboration.) One reason I don't use it is that I started doing data science before the Python ecosystem was viable -- before Pandas existed. I use R for data science (which I generally find superior due to Hadley Wickham's libraries). I know Jupyter supports R now, but I already had a terminal/web-based workflow by the time that happened. ----- More importantly, I think this recent blog post finally crystallized why I don't program in REPLs: Because they encourage global variables! I naturally structure my code into functions from the outset. I don't like the persistence because it can lead to "wrong" programs. I prefer to test my programs with a "clean slate", i.e. by starting a new process. http://willcrichton.net/notes/programming-in-the-debugger/ http://willcrichton.net/notes/programming-in-the-debugger/ Jupyter’s structure of delimited code cells enables a programming style where each can be treated like an atomic unit, where if it completes, then its effects are persisted in memory for other code cells to process. However, this style of programming with Jupyter has its limits. For example, Jupyter penalizes abstraction by removing this interactive debuggability. In other words, if you put all your code in functions like I do, then Jupyter doesn't add anything. It doesn't let you "step through" the function like a debugger does. Though, I think that I should somehow try to get over this because there are a lot of benefits to something like Jupyter, like having graphics inline. Or maybe Jupyter just needs an integrated debugger? And maybe the ability to clear state or tree-walk definitions? I don't like having unused definitions laying around in my workspace. Also, does Jupyter have any notion of data flow? I don't think it does, because Python doesn't. I think Observable might address some of my gripes, but I haven't tried it yet: https://beta.observablehq.com/ https://beta.observablehq.com/

celias 9y ago
Take a look at Cauldron, aka the unnotebook -http://www.unnotebook.com http://www.unnotebook.com The author discussed it on pocastinit episode 111 - https://www.podcastinit.com/episode-111-cauldron-notebook-with-scott-ernst/ https://www.podcastinit.com/episode-111-cauldron-notebook-wi...
- chubot 9y ago
  Thanks, I hadn't heard of it! Looks interesting.
FrojoS 9y ago
I agree on the global variables. At least you can define functions in blocks of the same notebook file. e.g. [1] a = 2 [2] def square(x): return a 2 print square(a) > 4 [3] b = 5 print square(b) > 25 This is something that you can't do in MATLAB, which I still use primarily. In MATLAB you have to create a new file for each function. Which nulls the readability of the script (notebook) when you publish it, because the source of the functions is not included. So, if I write a MATLAB notebook that I want to publish and share, I end up avoiding to create functions for as long as possible and instead use copy paste ... Currently, some of the main thing that keeps me with MATLAB regardless is: a) It feels more responsive, probably because its a native app and not running in the browser. b) I don't like to work in the browser. Its distracting. c) I like the profiler, debugger and workspace (constant visual inspection of global variables in a separate window) of MATLAB which comes right out of the box. For python/Jupyter I have to set this up manually. Note: MATLAB now has an 'interactive script' function that is similar to Jupyter, but it is so slow for > 100 loc, that it's completely useless. Even the Mathworks developers admit this. Instead I use %% to seperate my MATLAB scripts into executable blocks (ctrl + enter) and then use the 'publish' function to create a LaTex file which I can then compile to pdf. This creates vector based math formulas and figures (which 'publish to html / pdf' doesn't).
- chubot 9y ago
  Hm that limitation sounds severe! Functions are essential. I actually don't remember that, but I last used MATLAB like 15 years ago! I've done a lot more data science than linear algebra in the last decade. It sounds like RStudio might be a better model for what we want: https://en.wikipedia.org/wiki/RStudio https://en.wikipedia.org/wiki/RStudio (click on the screenshot) It only works with R, but a lot of people say great things about it. I saw another commenter say that Jupyter is more like Mathematica notebooks, and RStudio is more like MATLAB. I think this sounds right. In the former, the interactive experience is more central. In the latter, you are developing a program, and the IDE helps you do it interactively. But the program is central. (At least this is true for R, it sounds like it might not be as true for MATLAB. But I know that pretty large programs are written in MATLAB.) For now I'm still sticking with my highly-custom shell-based workflow. But I do want to make interactive graphics less painful. Right now I juggle a web browser, a terminal, a text editor, and an R REPL!
breatheoften 9y ago
Your complaints resonant strongly with me. I'd really love an ideally typed expression-oriented language to take notebook programming to the next level ... This is my dream for what I want from swift notebooks ... Something that: - memoized all (non-loop?) values created in the notebook file scope - automatically invalidated memoized entries after source change using control flow analysis and code coverage data from previous runs - provides an approximation of the conceptual model of re-running the whole notebook 'from the beginning' when the notebook is executed -- but pulls memoized values from the memoized cache when present to make execution fast and to avoid repeating side-effects - allowed for easy to issue interactive 'invalidate all memoized entries before/after here' operations in the notebook file ... - involving editing code normally in a regular source-file in ide (with maybe a different extension to imply the different execution semantics) - allowed for execution of code in debugger when desired without having to change anything ... - supported inline graphical representations - supported wiring custom graphical ui models into the notebook's execution context ...
- chubot 9y ago
  Yes, in short, I think we both want the "dataflow model" and not the "stateful update" model. If anything should use the dataflow model, it's data analysis!!! And yes that's why I mentioned Observable, and I'm glad jashkenas also responded. As far as I understand, it's like a spreadsheet, so when you update your inputs, the outputs become consistent automatically. It's sort of like Make (or perhaps Make in reverse). Dataflow also allows your code to be parallelized. Some scientists don't care about this, but engineers do. It's an eye-opening experience to speed up naive data analysis by 1000x or more with shell scripts and a little C++.
- jashkenas 9y ago
  Fantastic list! I'd encourage you to give https://beta.observablehq.com https://beta.observablehq.com a try. From all of your points — aside from a statically typed language and editing in your normal IDE — we try to hit that target on the nose. Every cell is only reevaluated when any of its inputs changes, inline and custom graphical representations can render your live data — and even be used as values to be passed themselves as inputs to other cells. For a very simple example, see: https://beta.observablehq.com/@mbostock/d3-brushable-scatterplot https://beta.observablehq.com/@mbostock/d3-brushable-scatter...

by chubot 9y ago

celias 9y ago

Take a look at Cauldron, aka the unnotebook -http://www.unnotebook.com http://www.unnotebook.com The author discussed it on pocastinit episode 111 - https://www.podcastinit.com/episode-111-cauldron-notebook-with-scott-ernst/ https://www.podcastinit.com/episode-111-cauldron-notebook-wi...

chubot 9y ago
Thanks, I hadn't heard of it! Looks interesting.

FrojoS 9y ago

I agree on the global variables. At least you can define functions in blocks of the same notebook file. e.g. [1] a = 2 [2] def square(x): return a 2 print square(a) > 4 [3] b = 5 print square(b) > 25 This is something that you can't do in MATLAB, which I still use primarily. In MATLAB you have to create a new file for each function. Which nulls the readability of the script (notebook) when you publish it, because the source of the functions is not included. So, if I write a MATLAB notebook that I want to publish and share, I end up avoiding to create functions for as long as possible and instead use copy paste ... Currently, some of the main thing that keeps me with MATLAB regardless is: a) It feels more responsive, probably because its a native app and not running in the browser. b) I don't like to work in the browser. Its distracting. c) I like the profiler, debugger and workspace (constant visual inspection of global variables in a separate window) of MATLAB which comes right out of the box. For python/Jupyter I have to set this up manually. Note: MATLAB now has an 'interactive script' function that is similar to Jupyter, but it is so slow for > 100 loc, that it's completely useless. Even the Mathworks developers admit this. Instead I use %% to seperate my MATLAB scripts into executable blocks (ctrl + enter) and then use the 'publish' function to create a LaTex file which I can then compile to pdf. This creates vector based math formulas and figures (which 'publish to html / pdf' doesn't).

chubot 9y ago
Hm that limitation sounds severe! Functions are essential. I actually don't remember that, but I last used MATLAB like 15 years ago! I've done a lot more data science than linear algebra in the last decade. It sounds like RStudio might be a better model for what we want: https://en.wikipedia.org/wiki/RStudio https://en.wikipedia.org/wiki/RStudio (click on the screenshot) It only works with R, but a lot of people say great things about it. I saw another commenter say that Jupyter is more like Mathematica notebooks, and RStudio is more like MATLAB. I think this sounds right. In the former, the interactive experience is more central. In the latter, you are developing a program, and the IDE helps you do it interactively. But the program is central. (At least this is true for R, it sounds like it might not be as true for MATLAB. But I know that pretty large programs are written in MATLAB.) For now I'm still sticking with my highly-custom shell-based workflow. But I do want to make interactive graphics less painful. Right now I juggle a web browser, a terminal, a text editor, and an R REPL!

breatheoften 9y ago

Your complaints resonant strongly with me. I'd really love an ideally typed expression-oriented language to take notebook programming to the next level ... This is my dream for what I want from swift notebooks ... Something that: - memoized all (non-loop?) values created in the notebook file scope - automatically invalidated memoized entries after source change using control flow analysis and code coverage data from previous runs - provides an approximation of the conceptual model of re-running the whole notebook 'from the beginning' when the notebook is executed -- but pulls memoized values from the memoized cache when present to make execution fast and to avoid repeating side-effects - allowed for easy to issue interactive 'invalidate all memoized entries before/after here' operations in the notebook file ... - involving editing code normally in a regular source-file in ide (with maybe a different extension to imply the different execution semantics) - allowed for execution of code in debugger when desired without having to change anything ... - supported inline graphical representations - supported wiring custom graphical ui models into the notebook's execution context ...

chubot 9y ago
Yes, in short, I think we both want the "dataflow model" and not the "stateful update" model. If anything should use the dataflow model, it's data analysis!!! And yes that's why I mentioned Observable, and I'm glad jashkenas also responded. As far as I understand, it's like a spreadsheet, so when you update your inputs, the outputs become consistent automatically. It's sort of like Make (or perhaps Make in reverse). Dataflow also allows your code to be parallelized. Some scientists don't care about this, but engineers do. It's an eye-opening experience to speed up naive data analysis by 1000x or more with shell scripts and a little C++.
jashkenas 9y ago
Fantastic list! I'd encourage you to give https://beta.observablehq.com https://beta.observablehq.com a try. From all of your points — aside from a statically typed language and editing in your normal IDE — we try to hit that target on the nose. Every cell is only reevaluated when any of its inputs changes, inline and custom graphical representations can render your live data — and even be used as values to be passed themselves as inputs to other cells. For a very simple example, see: https://beta.observablehq.com/@mbostock/d3-brushable-scatterplot https://beta.observablehq.com/@mbostock/d3-brushable-scatter...