4 ms·
I agree. From personal experience, here are some organizing principles I find useful for exploratory data analysis: 1. A single Jupyter notebook should either
by mxwsn 7y ago
I agree. From personal experience, here are some organizing principles I find useful for exploratory data analysis:
1. A single Jupyter notebook should either tackle a single question, or use collapsible headings (https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/collapsible_headings/readme.html https://jupyter-contrib-nbextensions.readthedocs.io/en/lates...) to organize separate questions.
2. The Jupyter notebook should be written such that all cells are executable in order, and it should produce exactly the same output every time (unless the input has changed) for reproducibility. The entire notebook should be executable on the order of seconds - if it's taking longer, this is usually a sign that plotting should be scripted instead, or the data needs to be subsetted.
3. The jupyter notebook should have a clearly annotated input file / folder of data (typically pre-processed using scripts). I usually include this in the title. The notebook must have a creation/last modified date (fortunately this is automatic, but it is crucial when the input data can change over time)
4. Observations, general conclusions, preliminary answers to exploratory questions, etc from data analysis should be written in the notebook in comments or Markdown. This is for your future self.