4 ms·

Beyond the cool python angle, I find the following statement quite interesting: "...James Somers argued that Jupyter notebooks may replace the traditional resea

Beyond the cool python angle, I find the following statement quite interesting: "...James Somers argued that Jupyter notebooks may replace the traditional research paper typically shared as a PDF..." I've only been exposed to jupyter notebook references here and there...but i guess i should become a little more familiar with them.

Rainymood 8y ago
I was recently asked by a company to do a take-home assignment, some data analysis stuff. I opted for a jupyter notebook as it sequentially allows you to both explore the data in an interactive manner while still programming in Python. It was great. I love notebooks.
AlanSE 8y ago
While I think this is a good thing, an additional requirement is needed. The published notebook should be sequentially executed by an automated tool. Users can execute commands out of order and have artifacts from deleted commands. Without verification, you can be publishing notebooks with bugs in them that you don't see until you re-execute them.
- dunpeal 8y ago
  Jupyter notebooks are ultimately just a text file, which can be version controlled, verified (via the VCS or independent hashing), and if necessary - digitally signed in a cryptographically secure manner. Given that Git is already secure, I'd say all the researcher needs to know is basic usage of Git. Version controlling the notebook satisfies all the requirements you mentioned: prevents accidental distortion, is verifiable, restorable, etc.
  - ashtonbaker 8y ago
    It's not just tracability and security that's the issue, though. It's an issue of reproducibility. When I give you a notebook, how do you 'run' it to verify my calculations? You can't just press "run", because I might have executed the cells out of order, or deleted cells which later cells depended on. Version control doesn't solve this problem. Good practice might solve it, but it turns out that researchers are stunningly bad at following a strict top-to-bottom execution of notebooks. And who can blame them? The non-linearity of notebooks is one of their features. But when it comes to reproducibility and just the basic mechanics of sharing code, we definitely need a better solution.
  - 21 8y ago
    You forget that a typical Python Jupyter notebook will import a dozen Python packages. Versioning that it's still a major problem in the ecosystem.
    - dunpeal 8y ago
      > Versioning that it's still a major problem in the ecosystem. There's actually a perfectly good way to solve this problem: a virtual environment with a requirements.txt file.
      - deleted 8y ago
        [deleted]
      - LyndsySimon 8y ago
        For research in particular, I'd suggest a Pipfile.lock. That will ensure that the packages installed match exactly, not just by version number.
- japanuspus 8y ago
  Tools for combining the interactive work in notebooks with a more predictable linear flow are starting to mature. See for instance jupytext [1]: a bi-directional converter between jupyter notebooks and python/R/julia/... I believe jupytext was featured here on HN a few weeks back. [1] https://github.com/mwouts/jupytext https://github.com/mwouts/jupytext
qwerty456127 8y ago
"Jupyter notebooks may replace the traditional research paper typically shared as a PDF..." - hardly a replacement as long as Jupyter notebooks are not self-sufficient files you can download and view offline everywhere. Even if you have up-to-date versions of Python and all the libraries installed you still have to run a server and open it in a browser which is a way more clumsy than just opening a PDF in Okular/Evince/AcrobatReader. I also doubt Jupyter notebooks are easy to view on mobile devices.
- sueders101 8y ago
  You can view them just as easily as you would a pdf. Running the code inside them is what takes extra effort. Which isn’t necessary if you just want to understand the processes being applied.
  - qwerty456127 8y ago
    Can they include pre-rendered code execution results to display when code execution is not available?
    - sueders101 8y ago
      Yes, they do by default if I’m not mistaken. The output “cells” are saved alongside the code.

Rainymood 8y ago

I was recently asked by a company to do a take-home assignment, some data analysis stuff. I opted for a jupyter notebook as it sequentially allows you to both explore the data in an interactive manner while still programming in Python. It was great. I love notebooks.

AlanSE 8y ago

While I think this is a good thing, an additional requirement is needed. The published notebook should be sequentially executed by an automated tool. Users can execute commands out of order and have artifacts from deleted commands. Without verification, you can be publishing notebooks with bugs in them that you don't see until you re-execute them.

dunpeal 8y ago
Jupyter notebooks are ultimately just a text file, which can be version controlled, verified (via the VCS or independent hashing), and if necessary - digitally signed in a cryptographically secure manner. Given that Git is already secure, I'd say all the researcher needs to know is basic usage of Git. Version controlling the notebook satisfies all the requirements you mentioned: prevents accidental distortion, is verifiable, restorable, etc.
- ashtonbaker 8y ago
  It's not just tracability and security that's the issue, though. It's an issue of reproducibility. When I give you a notebook, how do you 'run' it to verify my calculations? You can't just press "run", because I might have executed the cells out of order, or deleted cells which later cells depended on. Version control doesn't solve this problem. Good practice might solve it, but it turns out that researchers are stunningly bad at following a strict top-to-bottom execution of notebooks. And who can blame them? The non-linearity of notebooks is one of their features. But when it comes to reproducibility and just the basic mechanics of sharing code, we definitely need a better solution.
- 21 8y ago
  You forget that a typical Python Jupyter notebook will import a dozen Python packages. Versioning that it's still a major problem in the ecosystem.
  - dunpeal 8y ago
    > Versioning that it's still a major problem in the ecosystem. There's actually a perfectly good way to solve this problem: a virtual environment with a requirements.txt file.
    - deleted 8y ago
      [deleted]
    - LyndsySimon 8y ago
      For research in particular, I'd suggest a Pipfile.lock. That will ensure that the packages installed match exactly, not just by version number.
japanuspus 8y ago
Tools for combining the interactive work in notebooks with a more predictable linear flow are starting to mature. See for instance jupytext [1]: a bi-directional converter between jupyter notebooks and python/R/julia/... I believe jupytext was featured here on HN a few weeks back. [1] https://github.com/mwouts/jupytext https://github.com/mwouts/jupytext

qwerty456127 8y ago

"Jupyter notebooks may replace the traditional research paper typically shared as a PDF..." - hardly a replacement as long as Jupyter notebooks are not self-sufficient files you can download and view offline everywhere. Even if you have up-to-date versions of Python and all the libraries installed you still have to run a server and open it in a browser which is a way more clumsy than just opening a PDF in Okular/Evince/AcrobatReader. I also doubt Jupyter notebooks are easy to view on mobile devices.

sueders101 8y ago
You can view them just as easily as you would a pdf. Running the code inside them is what takes extra effort. Which isn’t necessary if you just want to understand the processes being applied.
- qwerty456127 8y ago
  Can they include pre-rendered code execution results to display when code execution is not available?
  - sueders101 8y ago
    Yes, they do by default if I’m not mistaken. The output “cells” are saved alongside the code.