6 ms·

In the same spirit as “Effective Java” and “Effective C++” we need to have a book entitled “Effective Jupyter Notebooks”. Here are some of my items below. Maybe

In the same spirit as “Effective Java” and “Effective C++” we need to have a book entitled “Effective Jupyter Notebooks”. Here are some of my items below. Maybe this sub-thread can come up with an outline for this book. Item #1 Writing a notebook is foremost an exercise in expository writing. Make sure the writing is high quality is the first objective when writing a notebook. This is the Knuth’s literate programming idea where prose takes precedence over code, which is usually the reverse of the way we usually program; code first, comments second. Item #2 Don't use notebooks for general purpose programming. Notebooks are supposed to have an audience and clearly explain something. Item #3 Keep code cells simple and clear. If needing a comment in the code block, consider putting that verbiage in a markdown cell instead and elaborating on the idea the notebook is trying to convey. Item #4 Don't make notebooks a long series of extended code cells, or even worse, just one long cell. Explain what is going on or see Item #2.

jimmar 7y ago
For #2, I have used notebooks for personal data analysis. I am my own audience and need to remember what I did and why.
arnioxux 7y ago
For example Peter Norvig does an exemplary job with jupyter notebooks for expository writings: https://github.com/norvig/pytudes https://github.com/norvig/pytudes
eoinmurray92 7y ago
This is a really good idea, and I've been planning to write said guide for a while. I think there is a lot to do with organising and formatting the notebooks, and the %run command works a treat for me when breaking a project into multiple notebooks
zimablue 7y ago
I disagree massively with this, to me Jupyter is a repl+ development environment + rapid prototyping in ETL/visualisation. There's no rule that they have to be used for exposition, or any writing at all. I don't know why you think this is a rule, or what you'd use as an alternative data-repl.
localhost 7y ago
While I generally agree with everything you say for a particular audience, i.e., someone who is writing something to explain an idea to someone else, I think that at the same time this scoping limits the utility of Jupyter too much. I believe that there is an audience for Jupyter beyond just "data scientist", or "teacher". I think that what Jupyter encourages is _experimentation_, and there are lots of folks who could benefit by using Jupyter to experiment with ideas. If you consider programmer as another audience, we all at different points in time experiment with things. Think about all of those random console applications that we've all created at one point or another to learn a new API, or to try out an idea. Jupyter can (and does) excel in these scenarios and handles the mundane task of "writing things down". This is why adding a Jupyter experience to a text editor is something that we're experimenting with as well.
rhodin 7y ago
See also "computational essay"[1]. It is written with Mathematica notebooks in mind, but many of the ideas carry over to Jupyter notebooks. [1] https://blog.stephenwolfram.com/2017/11/what-is-a-computational-essay/ https://blog.stephenwolfram.com/2017/11/what-is-a-computatio...
- jgamman 7y ago
  and a more jupyter-centric agree/reply by Tony Hirst: https://blog.ouseful.info/2017/11/15/programming-meh-lets-teach-how-to-write-computational-essays-instead/ https://blog.ouseful.info/2017/11/15/programming-meh-lets-te...
dec0dedab0de 7y ago
I disagree. While your points are valid for presentations, I believe that notebooks should first and foremost be used for exploratory computing. Notebooks are my goto tool for whenever I need to do something with a computer and any of the following apply: * I'm not quite sure what or how. * It will likely be a one-off * I need it now. * Someone is watching me, to learn how I do it. I would go a step further and say that notebooks should probably not be shared directly most of the time, and that you should wrap up functionality into modules/packages for other people to use in their own notebooks. Edit: Just thought of a corny pun to make my point: They are notebooks, not textbooks. Notebooks are personal, and while it may be useful/insightful to compare notes, it's not the primary function of a notebook.
- blueboo 7y ago
  Indeed, when the alternative is the interpreter, the structure notebooks offer and the (loose) process they encourage makes for a vastly better solution for recording and capitalizing on your work.
- bluejay2387 7y ago
  I agree with with your disagreement. I've always approached notebooks as a prototyping tool first and a presentation tool second. I see Jupyter more as an example of an early Interactive programming tool that just happens to be useful for presentation and teaching purposes. (https://en.wikipedia.org/wiki/Interactive_programming https://en.wikipedia.org/wiki/Interactive_programming) And regarding sharing -- particularly with notebooks you should be really cautious about sharing and opening them. The security model is not exactly bullet proof.
  - TeMPOraL 7y ago
    > I see Jupyter more as an example of an early Interactive programming tool Nitpick, but I hope you meant "young", not "early". Interactive programming is as old as (and today still primarily featured in) Lisp - i.e. twice as old as most of us here on HN.
- TeMPOraL 7y ago
analog31 7y ago
While I don't follow #2, I suggest two more item: Take the time to learn some good programming practices. If you're lucky and fall in love with scientific programming, your programs will grow to a point where they are unmanageable without some techniques borrowed from computer science and software engineering. Have someone show you the "out of order execution" and "hidden state" failures.
_raoulcousins 7y ago
Obligatory Joel Grus: slides from his 'I don't like notebooks' talk. He has good follow-up talks too. https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-dkAIsUXP-AL4ffI/edit https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUh...
- jasode 7y ago
  ...and a well-written response to that by Yihui Xie: https://yihui.name/en/2018/09/notebook-war/ https://yihui.name/en/2018/09/notebook-war/ Since Xie's post discusses a "meta" layer above Joel Grus, I recommend people read it before JG. This actually helps one understand the underlying thinking in JG's slides.
deleted 7y ago
[deleted]
pwrose 7y ago
Check out: Ten Simple Rules for Reproducible Research in Jupyter Notebooks, https://arxiv.org/abs/1810.08055 https://arxiv.org/abs/1810.08055 and https://github.com/jupyter-guide/ten-rules-jupyter https://github.com/jupyter-guide/ten-rules-jupyter

jimmar 7y ago

For #2, I have used notebooks for personal data analysis. I am my own audience and need to remember what I did and why.

arnioxux 7y ago

For example Peter Norvig does an exemplary job with jupyter notebooks for expository writings: https://github.com/norvig/pytudes https://github.com/norvig/pytudes

eoinmurray92 7y ago

This is a really good idea, and I've been planning to write said guide for a while. I think there is a lot to do with organising and formatting the notebooks, and the %run command works a treat for me when breaking a project into multiple notebooks

zimablue 7y ago

I disagree massively with this, to me Jupyter is a repl+ development environment + rapid prototyping in ETL/visualisation. There's no rule that they have to be used for exposition, or any writing at all. I don't know why you think this is a rule, or what you'd use as an alternative data-repl.

localhost 7y ago

While I generally agree with everything you say for a particular audience, i.e., someone who is writing something to explain an idea to someone else, I think that at the same time this scoping limits the utility of Jupyter too much. I believe that there is an audience for Jupyter beyond just "data scientist", or "teacher". I think that what Jupyter encourages is _experimentation_, and there are lots of folks who could benefit by using Jupyter to experiment with ideas. If you consider programmer as another audience, we all at different points in time experiment with things. Think about all of those random console applications that we've all created at one point or another to learn a new API, or to try out an idea. Jupyter can (and does) excel in these scenarios and handles the mundane task of "writing things down". This is why adding a Jupyter experience to a text editor is something that we're experimenting with as well.

rhodin 7y ago

See also "computational essay"[1]. It is written with Mathematica notebooks in mind, but many of the ideas carry over to Jupyter notebooks. [1] https://blog.stephenwolfram.com/2017/11/what-is-a-computational-essay/ https://blog.stephenwolfram.com/2017/11/what-is-a-computatio...

jgamman 7y ago
and a more jupyter-centric agree/reply by Tony Hirst: https://blog.ouseful.info/2017/11/15/programming-meh-lets-teach-how-to-write-computational-essays-instead/ https://blog.ouseful.info/2017/11/15/programming-meh-lets-te...

dec0dedab0de 7y ago

I disagree. While your points are valid for presentations, I believe that notebooks should first and foremost be used for exploratory computing. Notebooks are my goto tool for whenever I need to do something with a computer and any of the following apply: * I'm not quite sure what or how. * It will likely be a one-off * I need it now. * Someone is watching me, to learn how I do it. I would go a step further and say that notebooks should probably not be shared directly most of the time, and that you should wrap up functionality into modules/packages for other people to use in their own notebooks. Edit: Just thought of a corny pun to make my point: They are notebooks, not textbooks. Notebooks are personal, and while it may be useful/insightful to compare notes, it's not the primary function of a notebook.

blueboo 7y ago
Indeed, when the alternative is the interpreter, the structure notebooks offer and the (loose) process they encourage makes for a vastly better solution for recording and capitalizing on your work.
bluejay2387 7y ago
I agree with with your disagreement. I've always approached notebooks as a prototyping tool first and a presentation tool second. I see Jupyter more as an example of an early Interactive programming tool that just happens to be useful for presentation and teaching purposes. (https://en.wikipedia.org/wiki/Interactive_programming https://en.wikipedia.org/wiki/Interactive_programming) And regarding sharing -- particularly with notebooks you should be really cautious about sharing and opening them. The security model is not exactly bullet proof.
- TeMPOraL 7y ago
  > I see Jupyter more as an example of an early Interactive programming tool Nitpick, but I hope you meant "young", not "early". Interactive programming is as old as (and today still primarily featured in) Lisp - i.e. twice as old as most of us here on HN.
TeMPOraL 7y ago

analog31 7y ago

While I don't follow #2, I suggest two more item: Take the time to learn some good programming practices. If you're lucky and fall in love with scientific programming, your programs will grow to a point where they are unmanageable without some techniques borrowed from computer science and software engineering. Have someone show you the "out of order execution" and "hidden state" failures.

_raoulcousins 7y ago

Obligatory Joel Grus: slides from his 'I don't like notebooks' talk. He has good follow-up talks too. https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-dkAIsUXP-AL4ffI/edit https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUh...

jasode 7y ago
...and a well-written response to that by Yihui Xie: https://yihui.name/en/2018/09/notebook-war/ https://yihui.name/en/2018/09/notebook-war/ Since Xie's post discusses a "meta" layer above Joel Grus, I recommend people read it before JG. This actually helps one understand the underlying thinking in JG's slides.

deleted 7y ago

[deleted]

pwrose 7y ago

Check out: Ten Simple Rules for Reproducible Research in Jupyter Notebooks, https://arxiv.org/abs/1810.08055 https://arxiv.org/abs/1810.08055 and https://github.com/jupyter-guide/ten-rules-jupyter https://github.com/jupyter-guide/ten-rules-jupyter