3 ms·

Have we reach a point where there are similar guides for Jupyter notebooks? Or just for the domain of data science / exploratory statistics Like, conceptually,

Have we reach a point where there are similar guides for Jupyter notebooks? Or just for the domain of data science / exploratory statistics Like, conceptually, what should I keep, and how should I organise my notebook(s)? Most guides are focused on what you can do (Here's how you draw a graph, here's how you load a kernel), I'd rather read about what I should store in my notebooks, and how many I should have.

mxwsn 8y ago
I agree. From personal experience, here are some organizing principles I find useful for exploratory data analysis: 1. A single Jupyter notebook should either tackle a single question, or use collapsible headings (https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/collapsible_headings/readme.html https://jupyter-contrib-nbextensions.readthedocs.io/en/lates...) to organize separate questions. 2. The Jupyter notebook should be written such that all cells are executable in order, and it should produce exactly the same output every time (unless the input has changed) for reproducibility. The entire notebook should be executable on the order of seconds - if it's taking longer, this is usually a sign that plotting should be scripted instead, or the data needs to be subsetted. 3. The jupyter notebook should have a clearly annotated input file / folder of data (typically pre-processed using scripts). I usually include this in the title. The notebook must have a creation/last modified date (fortunately this is automatic, but it is crucial when the input data can change over time) 4. Observations, general conclusions, preliminary answers to exploratory questions, etc from data analysis should be written in the notebook in comments or Markdown. This is for your future self.
eoinmurray92 8y ago
This is something we think a lot about at Kyso (https://kyso.io/for-teams https://kyso.io/for-teams) - my startup - we let teams manage analytics knowledge basically by letting teams share Jupyter notebooks - we've spoken to lots of people about how the organise them and are writing a guide really soon that sounds like what you want. - Keep all your teams notebooks in one place (not on laptops) - Reuse them liberally (the %run command is excellent and often better than creating python libraries) - Most important of all is make them readable documents not just list of coding commands, that means you should title things clearly, explain what you going to do. - Label everything, axis labels are needed on almost every chart, and also make sure you put a title and description for every chart. - Separating your project into processing and presentation notebooks is really helpful. So one notebook for data prep and another for plots and explanations.
- wodenokoto 8y ago
  > Reuse them liberally (the %run command is excellent and often better than creating python libraries) Does that mean you run other notebooks from within notebooks? Can you ping me or something when the guide is out?
  - eoinmurray92 8y ago
    Happy to, but I've no way to ping you?

mxwsn 8y ago

I agree. From personal experience, here are some organizing principles I find useful for exploratory data analysis: 1. A single Jupyter notebook should either tackle a single question, or use collapsible headings (https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/collapsible_headings/readme.html https://jupyter-contrib-nbextensions.readthedocs.io/en/lates...) to organize separate questions. 2. The Jupyter notebook should be written such that all cells are executable in order, and it should produce exactly the same output every time (unless the input has changed) for reproducibility. The entire notebook should be executable on the order of seconds - if it's taking longer, this is usually a sign that plotting should be scripted instead, or the data needs to be subsetted. 3. The jupyter notebook should have a clearly annotated input file / folder of data (typically pre-processed using scripts). I usually include this in the title. The notebook must have a creation/last modified date (fortunately this is automatic, but it is crucial when the input data can change over time) 4. Observations, general conclusions, preliminary answers to exploratory questions, etc from data analysis should be written in the notebook in comments or Markdown. This is for your future self.

eoinmurray92 8y ago

This is something we think a lot about at Kyso (https://kyso.io/for-teams https://kyso.io/for-teams) - my startup - we let teams manage analytics knowledge basically by letting teams share Jupyter notebooks - we've spoken to lots of people about how the organise them and are writing a guide really soon that sounds like what you want. - Keep all your teams notebooks in one place (not on laptops) - Reuse them liberally (the %run command is excellent and often better than creating python libraries) - Most important of all is make them readable documents not just list of coding commands, that means you should title things clearly, explain what you going to do. - Label everything, axis labels are needed on almost every chart, and also make sure you put a title and description for every chart. - Separating your project into processing and presentation notebooks is really helpful. So one notebook for data prep and another for plots and explanations.

wodenokoto 8y ago
> Reuse them liberally (the %run command is excellent and often better than creating python libraries) Does that mean you run other notebooks from within notebooks? Can you ping me or something when the guide is out?
- eoinmurray92 8y ago
  Happy to, but I've no way to ping you?