6 ms·

What happens to the outputs in this case? I found the outputs to be both the most useful parts of notebooks, but also the most troublesome for diffing and versi

What happens to the outputs in this case? I found the outputs to be both the most useful parts of notebooks, but also the most troublesome for diffing and versioning.

enriquto 3y ago
Why would you commit the outputs into git? That would be like committing compiled binary objects or pdfs. Of course the outputs are useful, but you just want to commit the sources. The .ipynb stores inputs and outputs together in an unholy way. It is much cleaner to separate them. The inputs are python (or markdown) files that you can edit with a text editor and version control with git. The outputs are html, pdf, or whatever you want to nbconvert to and share. The .ipynb file would only be useful if you want to share a stateful notebook, whose state cannot be easily reproduced by the people who you share it with. But that would be really bizarre and definitely in bad taste. Sharing the .ipynb is akin to sharing your .pyc files. I love working with notebooks, but as a measure of hygiene I avoid .ipynb files altogether.
- RobinL 3y ago
  Most of the time I agree, but if you want to e.g present a tutorial as a webpage, having an ipynb with both inputs and outputs becomes a feature. You can even bundle into docs e.g. https://moj-analytical-services.github.io/splink/demos/02_Exploratory_analysis.html https://moj-analytical-services.github.io/splink/demos/02_Ex...
- apwheele 3y ago
  Some people use the outputs like documentation, since github renders the notebook contents nicely in the browser. I agree it is not the best practice in many situations, I like using it on occasion though. Another alternative if you want the outputs is to use nbconvert to convert the output to markdown, https://andrewpwheeler.com/2021/09/06/using-jupyter-notebooks-to-make-nice-readmes-for-github/ https://andrewpwheeler.com/2021/09/06/using-jupyter-notebook...
- pletnes 3y ago
  Sometimes I work on software development, and this mindset («the only valuable asset is the code») makes total sense. But if I work on analytics / datascience projects, the analysis including outputs could be time consuming to run, validate, and visualize. In these cases, it might be required to version the outputs. I’ve never used jupyter for taking notes in a lab setting, but with more and more instruments being computer/network connected, I imagine this would make total sense - put your data and notes with your analytics work. Many jupyter users are not «software developers», they just use code to perform their work.
  - Zandikar 3y ago
    Precisely this. When your output is something like research data, or even just something that generally takes a long human time to complete (hours vs Microseconds) it makes a lot of sense to version and keep outputs, at least on major "versions".
    - adament 3y ago
      But would you version it by storing it as output in an ipnyb file where it is overwritten if you rerun that cell? I would store the data in a versioned database or as separate data files in the repository (possibly stored in git-lfs). And I would store results of the analysis as data files / image files / whatever else, NOT as ephemereal outputs in an ipynb file. But I am pretty far down the “ipynb files are for local use only” path.
      - tnecniv 3y ago
        Yeah if your analysis to takes hours to run, you should really split up the number crunching code and result analysis / visualization. Not only does it make version control of the code easier, you can save the output in an organized labeled manner (time-stamped, etc.) and, if you lose power or the kernel crashes, you don’t need to rerun the lengthy analysis if you want to make a change further down the pipeline.
        3y ago
- westurner 3y ago
  If you want to store (e.g. base64-encoded) outputs in Markdown, you're going to have to scroll past a lot of data to get to the next input cell in a notebook. Jupyter notebooks store which Jupyter kernel they were run with to generate the outputs. nbformat (.ipynb with inlined base64 outputs) isn't a sufficient package format: https://github.com/jupyter/enhancement-proposals/pull/103#issuecomment-1510418219 https://github.com/jupyter/enhancement-proposals/pull/103#is... Papermill is one tool for running Jupyter notebooks as reports; with the date in the filename. https://papermill.readthedocs.io/en/latest/ https://papermill.readthedocs.io/en/latest/
- sfpotter 3y ago
  "unholy" "cleaner" "stateful" "bizarre" "bad taste" "hygiene" I'm not sure whether you're unaware or just feigning ignorance, but notebooks are frequently used to share partial results, often in the context of "research", however you may interpret it. Imagine a grad student or data scientist preparing some code and plots to show during a weekly meeting. In this context, the only thing that matters is quick progress and advancing understanding of a problem. The highly loaded words you employ while blasting the idea of uploading Jupyter notebooks are not relevant here. Wasting time on these things is seen as a bad thing. It's clear why someone using notebooks this way would want the interaction with Git and GitHub to be as seamless as possible: uploading something to GitHub is a very easy way to share it, even if this isn't the platonic ideal. It will probably cause you some pain, but I've known people to commit binary objects and PDFs to git to accomplish the same ends. ;-)
  - lynndotpy 3y ago
    I agree entirely, outputs are great and I consider it best practice to provide notebooks with outputs provided. As a concrete example, this one-liner of Python code is much more interesting to those who don't recognize it when it's presented with the associated output. 4*sum([(random.random()**2 + random.random()**2)**.5 < 1 for _ in range(10**7)])/10**7 This is also useful, e.g. when viewing the read-only export of a notebook. (The one-linear above is a monte-carlo simulation which approximates Pi. On one run, this result came to 3.1410416.)
  - deleted 3y ago
    [deleted]
  - tnecniv 3y ago
    As someone who uses notebooks for research, outputs don’t play well with git. You can end up with very large commits that GitHub or wherever your repo lives may not like if you have a lot of plots and animations. Moreover, while research moves fast, reproducibility remains important. If your notebook is stateful, then when you share it I may not be able to recreate your result or you might have a bug due to something lingering in the notebook state. Having your outputs is convenient, but if I download the notebook, run it myself, and find that the code doesn’t run because there’s some variable that got defined earlier in your session but that code got deleted during iteration, that’s really not helpful. It’s the equivalent of handing someone your lab notebook but you kept erasing over early pages to make room for new content. That’s one example of a bug. You could easily introduce more subtle bugs where the state leads to invalid results.
- benrutter 3y ago
  I think it depends a lot on what your git repository is. If it's specifically source code for anything that's intended to run, then avoiding including the outputs is a smart move. But then, if that's the case, there's a good chance you'd just be committing a .py file. I like notebooks because they include output alongisde input. For example, Peter Norvig's Pytudes are all brilliant, quick notebooks that solve a particular puzzle[0]. The code itself might not be that interesting to run (unless you really want to confirm his strategy for wordle checks out) but reading through the notebooks makes for a great experience of simultaneously understanding his thought process, and seeing the solution. I do a bunch of generative art stuff and have recently been experimenting with using notebooks as quick sketches[1]. I really like the workflow and end up with something like a journal that isn't necessarily intended to be ran repeatedly, but read over, where I can see the visual output created, as well as the method for it. [0] Norvig's extremely cool pytudes, wordle example: https://github.com/norvig/pytudes/blob/main/ipynb/Wordle.ipynb https://github.com/norvig/pytudes/blob/main/ipynb/Wordle.ipy... [1] My not anywhere near as cool as Norvig's pytudes example: https://github.com/benrutter/jupyter-sketches https://github.com/benrutter/jupyter-sketches
  - WCSTombs 3y ago
    You don't lose the outputs, they just aren't committed into Git. So for each new clone, you'd need to regenerate the outputs, but on a single clone, the outputs exist and are persistent in the .ipynb form of the notebook (which is not committed). You are correct that the .py version of the notebook is exactly what Git ends up tracking, with the .ipynb being essentially a build product. (Note that the jupytext paradigm does assume that the outputs can always be recomputed as a function of the inputs. I consider that a best practice, but some might disagree.)
- chthonicdaemon 3y ago
  How do you capture things like charts or tables produced from long-running notebooks? Do you have a separate system to keep track of these? I prize notebooks with outputs in our data science repo since I can see the results of our analyses years later without having to re-run the notebook. In some cases, the notebooks don't even run anymore since our environment has moved on, but I can still see the graphs and read the text that was associated with that analysis.
kzrdude 3y ago
I use jupytext paired with ipynb files. Only store the .py files in git. The ipynb files act as a local cache of outputs. Outputs are loaded from the ipynb even if you open the .py notebook.
Helmut10001 3y ago
I use Jupytext since years. It allows me to have three types of synced notebook versions: 1) .ipynb (for opening/running), 2) .md (formatted code+comments, without outputs) and 3) *.py (python formatted, code+comments). I commit the Markdown-version, but I also use the py-version of notebooks for chained notebook imports. Allows me to split larger notebooks into multiple smaller ones. Both of these options are a blessing and Jupytext works super-robust. Finally, when I want to archive (and share) notebooks _with_ outputs once in a while, I have a cell at the end to convert (nbconvert) to HTML, and I commit this html file. The Markdown-version remains as a clean basis for commit history. The HTML file is much better suited for sharing and archiving than the ipynb file.
- Helmut10001 3y ago
  Here's another HN comment with links of an example repo [1]. [1]: https://news.ycombinator.com/item?id=36516836 https://news.ycombinator.com/item?id=36516836
bootsmann 3y ago
We use jupytext with dvc. You can generate the notebook in dvc.yaml using the jupytext cli and then push this alongside the .py file.

What happens to the outputs in this case? I found the outputs to be both the most useful parts of notebooks, but also the most troublesome for diffing and versioning.

enriquto 3y ago

Why would you commit the outputs into git? That would be like committing compiled binary objects or pdfs. Of course the outputs are useful, but you just want to commit the sources. The .ipynb stores inputs and outputs together in an unholy way. It is much cleaner to separate them. The inputs are python (or markdown) files that you can edit with a text editor and version control with git. The outputs are html, pdf, or whatever you want to nbconvert to and share. The .ipynb file would only be useful if you want to share a stateful notebook, whose state cannot be easily reproduced by the people who you share it with. But that would be really bizarre and definitely in bad taste. Sharing the .ipynb is akin to sharing your .pyc files. I love working with notebooks, but as a measure of hygiene I avoid .ipynb files altogether.

RobinL 3y ago
Most of the time I agree, but if you want to e.g present a tutorial as a webpage, having an ipynb with both inputs and outputs becomes a feature. You can even bundle into docs e.g. https://moj-analytical-services.github.io/splink/demos/02_Exploratory_analysis.html https://moj-analytical-services.github.io/splink/demos/02_Ex...
apwheele 3y ago
Some people use the outputs like documentation, since github renders the notebook contents nicely in the browser. I agree it is not the best practice in many situations, I like using it on occasion though. Another alternative if you want the outputs is to use nbconvert to convert the output to markdown, https://andrewpwheeler.com/2021/09/06/using-jupyter-notebooks-to-make-nice-readmes-for-github/ https://andrewpwheeler.com/2021/09/06/using-jupyter-notebook...
pletnes 3y ago
Sometimes I work on software development, and this mindset («the only valuable asset is the code») makes total sense. But if I work on analytics / datascience projects, the analysis including outputs could be time consuming to run, validate, and visualize. In these cases, it might be required to version the outputs. I’ve never used jupyter for taking notes in a lab setting, but with more and more instruments being computer/network connected, I imagine this would make total sense - put your data and notes with your analytics work. Many jupyter users are not «software developers», they just use code to perform their work.
- Zandikar 3y ago
  Precisely this. When your output is something like research data, or even just something that generally takes a long human time to complete (hours vs Microseconds) it makes a lot of sense to version and keep outputs, at least on major "versions".
  - adament 3y ago
    But would you version it by storing it as output in an ipnyb file where it is overwritten if you rerun that cell? I would store the data in a versioned database or as separate data files in the repository (possibly stored in git-lfs). And I would store results of the analysis as data files / image files / whatever else, NOT as ephemereal outputs in an ipynb file. But I am pretty far down the “ipynb files are for local use only” path.
    - tnecniv 3y ago
      Yeah if your analysis to takes hours to run, you should really split up the number crunching code and result analysis / visualization. Not only does it make version control of the code easier, you can save the output in an organized labeled manner (time-stamped, etc.) and, if you lose power or the kernel crashes, you don’t need to rerun the lengthy analysis if you want to make a change further down the pipeline.
      - 3y ago
westurner 3y ago
If you want to store (e.g. base64-encoded) outputs in Markdown, you're going to have to scroll past a lot of data to get to the next input cell in a notebook. Jupyter notebooks store which Jupyter kernel they were run with to generate the outputs. nbformat (.ipynb with inlined base64 outputs) isn't a sufficient package format: https://github.com/jupyter/enhancement-proposals/pull/103#issuecomment-1510418219 https://github.com/jupyter/enhancement-proposals/pull/103#is... Papermill is one tool for running Jupyter notebooks as reports; with the date in the filename. https://papermill.readthedocs.io/en/latest/ https://papermill.readthedocs.io/en/latest/
sfpotter 3y ago
"unholy" "cleaner" "stateful" "bizarre" "bad taste" "hygiene" I'm not sure whether you're unaware or just feigning ignorance, but notebooks are frequently used to share partial results, often in the context of "research", however you may interpret it. Imagine a grad student or data scientist preparing some code and plots to show during a weekly meeting. In this context, the only thing that matters is quick progress and advancing understanding of a problem. The highly loaded words you employ while blasting the idea of uploading Jupyter notebooks are not relevant here. Wasting time on these things is seen as a bad thing. It's clear why someone using notebooks this way would want the interaction with Git and GitHub to be as seamless as possible: uploading something to GitHub is a very easy way to share it, even if this isn't the platonic ideal. It will probably cause you some pain, but I've known people to commit binary objects and PDFs to git to accomplish the same ends. ;-)
- lynndotpy 3y ago
  I agree entirely, outputs are great and I consider it best practice to provide notebooks with outputs provided. As a concrete example, this one-liner of Python code is much more interesting to those who don't recognize it when it's presented with the associated output. 4*sum([(random.random()**2 + random.random()**2)**.5 < 1 for _ in range(10**7)])/10**7 This is also useful, e.g. when viewing the read-only export of a notebook. (The one-linear above is a monte-carlo simulation which approximates Pi. On one run, this result came to 3.1410416.)
- deleted 3y ago
  [deleted]
- tnecniv 3y ago
  As someone who uses notebooks for research, outputs don’t play well with git. You can end up with very large commits that GitHub or wherever your repo lives may not like if you have a lot of plots and animations. Moreover, while research moves fast, reproducibility remains important. If your notebook is stateful, then when you share it I may not be able to recreate your result or you might have a bug due to something lingering in the notebook state. Having your outputs is convenient, but if I download the notebook, run it myself, and find that the code doesn’t run because there’s some variable that got defined earlier in your session but that code got deleted during iteration, that’s really not helpful. It’s the equivalent of handing someone your lab notebook but you kept erasing over early pages to make room for new content. That’s one example of a bug. You could easily introduce more subtle bugs where the state leads to invalid results.
benrutter 3y ago
I think it depends a lot on what your git repository is. If it's specifically source code for anything that's intended to run, then avoiding including the outputs is a smart move. But then, if that's the case, there's a good chance you'd just be committing a .py file. I like notebooks because they include output alongisde input. For example, Peter Norvig's Pytudes are all brilliant, quick notebooks that solve a particular puzzle[0]. The code itself might not be that interesting to run (unless you really want to confirm his strategy for wordle checks out) but reading through the notebooks makes for a great experience of simultaneously understanding his thought process, and seeing the solution. I do a bunch of generative art stuff and have recently been experimenting with using notebooks as quick sketches[1]. I really like the workflow and end up with something like a journal that isn't necessarily intended to be ran repeatedly, but read over, where I can see the visual output created, as well as the method for it. [0] Norvig's extremely cool pytudes, wordle example: https://github.com/norvig/pytudes/blob/main/ipynb/Wordle.ipynb https://github.com/norvig/pytudes/blob/main/ipynb/Wordle.ipy... [1] My not anywhere near as cool as Norvig's pytudes example: https://github.com/benrutter/jupyter-sketches https://github.com/benrutter/jupyter-sketches
- WCSTombs 3y ago
  You don't lose the outputs, they just aren't committed into Git. So for each new clone, you'd need to regenerate the outputs, but on a single clone, the outputs exist and are persistent in the .ipynb form of the notebook (which is not committed). You are correct that the .py version of the notebook is exactly what Git ends up tracking, with the .ipynb being essentially a build product. (Note that the jupytext paradigm does assume that the outputs can always be recomputed as a function of the inputs. I consider that a best practice, but some might disagree.)
chthonicdaemon 3y ago
How do you capture things like charts or tables produced from long-running notebooks? Do you have a separate system to keep track of these? I prize notebooks with outputs in our data science repo since I can see the results of our analyses years later without having to re-run the notebook. In some cases, the notebooks don't even run anymore since our environment has moved on, but I can still see the graphs and read the text that was associated with that analysis.

kzrdude 3y ago

I use jupytext paired with ipynb files. Only store the .py files in git. The ipynb files act as a local cache of outputs. Outputs are loaded from the ipynb even if you open the .py notebook.

Helmut10001 3y ago

I use Jupytext since years. It allows me to have three types of synced notebook versions: 1) .ipynb (for opening/running), 2) .md (formatted code+comments, without outputs) and 3) *.py (python formatted, code+comments). I commit the Markdown-version, but I also use the py-version of notebooks for chained notebook imports. Allows me to split larger notebooks into multiple smaller ones. Both of these options are a blessing and Jupytext works super-robust. Finally, when I want to archive (and share) notebooks _with_ outputs once in a while, I have a cell at the end to convert (nbconvert) to HTML, and I commit this html file. The Markdown-version remains as a clean basis for commit history. The HTML file is much better suited for sharing and archiving than the ipynb file.

Helmut10001 3y ago
Here's another HN comment with links of an example repo [1]. [1]: https://news.ycombinator.com/item?id=36516836 https://news.ycombinator.com/item?id=36516836

bootsmann 3y ago

We use jupytext with dvc. You can generate the notebook in dvc.yaml using the jupytext cli and then push this alongside the .py file.