31 ms·

Jupyter Receives the ACM Software System Award

714 points by williamstein 8y ago

monkeydust 8y ago
I am non programmer business guy who 'excels' a lot. Been teaching myself pandas through notebook. Awesome combination.
criddell 8y ago
Do you have a favorite tutorial or simple use case that helped you grok it?
zitterbewegung 8y ago
Wow this is awesome! It got the Award quickly also I think Juypter hasn't been around as long as other recipients.
- carreau 8y ago
  As we say in the blog post, the Jupyter Name has been around only since 2014, but the work started in 2001. So 17 years is a good check of time !
  - filmor 8y ago
    Though the part that people mostly identify with the project, the Jupyter Notebook was introduced only in 2011. I remember that vividly as I was writing my thesises at that time and spending quite some time playing around with it :)
    - carreau 8y ago
      Though the current notebook is the 6th prototype, so there've been quite some work before actually being made "public". Hope the notebook didn't distract you too much from your PhD !
      - po84 8y ago
        If open source didn't distract from ph.d's, we wouldn't have had ipython! ;)
      - pgbovine 8y ago
        overnight successes are usually 10-15 years in the making; so much goes on before the public release
angrygoat 8y ago
So well deserved. Jupyter is critical infrastructure, helping the scientific community address the reproducibility crisis. It's so great that it's free and open source, and can be shared, used and contributed to by scientists all around the world.
- scaryclam 8y ago
  Not only that, it's great for tutorial material as well!
msravi 8y ago
Trivia: Ju-py-ter = Julia Python Terminal Although it's gone far beyond just Julia and Python now. Edit: Ahurmazda is right. ...the core programming languages supported by Jupyter are Julia, Python and R. While the name Jupyter is not a direct acronym for these languages, it nods its head in those directions. In particular, the "y" in the middle of Jupyter was chosen to honor our Python heritage. https://github.com/jupyter/design/wiki/Jupyter-Logo https://github.com/jupyter/design/wiki/Jupyter-Logo
- osteele 8y ago
  Also worth noting: Hydrogen, the Jupyter plugin for Atom[1][2]. Hydrogen is a fantastically well-executed and useful piece of software. (As is Jupyter.) I’ve used it in my own programming, and also in teaching introductory software development, where it’s a helpful transition from Jupyter notebooks (which are used in the early part of the course, for problem sets / reading journals) to text files and code editors (which are introduced later, and used for team projects, and for projects that use Flask or PyGame). But also — relevant to this sub-thread — “Hydrogen” (it relates “Jupyter” to “Atom”) has to be one of the best project names ever. It’s right up there with “Pyramid Scheme”[3]. [1] https://nteract.io/atom https://nteract.io/atom [2] https://atom.io/packages/hydrogen https://atom.io/packages/hydrogen [3] http://www.michaelburge.us/2017/11/28/write-your-next-ethereum-contract-in-pyramid-scheme.html http://www.michaelburge.us/2017/11/28/write-your-next-ethere...
- ahurmazda 8y ago
  Julia Python R
- takluyver 8y ago
  ahurmazda pointed out that you're thinking of 'Jupyter Python R'. And this is a reference, not an 'equals'. The name is also in homage to Galileo's notebooks recording the discovery of the moons of Jupiter (the four we now call the 'Galilean moons'), and also to a bar called Jupiter in Berkeley, which the core team has visited quite often. The last one's more like a funny coincidence, though.
  - 8y ago
sytse 8y ago
Jupyter is awesome, well deserved award. Our CTO is working to integrate JupyterLab and JupyterHub directly into GitLab.
- evanb 8y ago
  Is there a way to associate and store large data sets, as well? I don't necessarily want them in git (hundreds of GB, for example), but I do want my notebooks to see them...
- carreau 8y ago
  Thanks, we love GitLab as well ! Would appreciate to have your feedback on how to make integration easier and get productive and constructive discussion. Will we see GitLab at JupyterCon this august in NY ?
  - sytse 8y ago
    We probably won't be a JupyterCon, but my CTO would love to have a call. Is it OK if Eliran, responsible for partnerships, reaches out to you for a video call?
    - carreau 8y ago
      Sure. We have team member across the bay in Berkeley, we can also cross over to see you or vice versa.
      - sytse 8y ago
        Great, we'll make it happen. It will probably be a video call since our CTO and co-founder is living in the Ukraine.
        sls 8y ago
        fyi - Why Ukrainians Hate When You Say 'The Ukraine' http://www.newsweek.com/why-ukrainians-hate-when-you-say-ukraine-392903 http://www.newsweek.com/why-ukrainians-hate-when-you-say-ukr...
nmca 8y ago
Thoroughly deserved - Jupyter is helping push science forwards.
oh-kumudo 8y ago
Congratulations! Jupyter is such a gem. It has become such a critical piece in the whole data analytics/ML landscape that I can't think of living without.
gbrown 8y ago
Am I the only person in the universe who doesn't like Jupyter? I much prefer tools like Rmarkdown and Sweave.
- tincholio 8y ago
  Nope... you're not the only one
- jasongrout 8y ago
  We would love to hear what you like about Rmarkdown and Sweave. Jupyter tooling is always improving, and we are very interested in engaging with users about their needs, and helping grow the ecosystem to be able to address those.
  - goerz 8y ago
    The number one feature request by far at this point would be collaborative features in the notebook. Not really because I actually want to collaboratively edit a notebook with others, but because I would like to open the same notebook in two separate browser windows (side-by-side), and edit a cell at the top of the notebook in one, and a cell at the bottom in the other, and have things get merged automatically. Or, not having to worry that I left a (remote) notebook open on my work computer when I connect to the same server from home, causing parts of the notebook to be accidentally overwritten.
    - targafarian 8y ago
      JupyterLab has the ability to open multiple "views" on the same file. http://jupyterlab.readthedocs.io/en/latest/user/notebook.html?highlight=views http://jupyterlab.readthedocs.io/en/latest/user/notebook.htm...
      - goerz 8y ago
        That's amazing! I'll have to try out JupyterLab sometime very soon. I was holding out because I'm using a whole bunch of plugins for the classic notebook from https://github.com/ipython-contrib/jupyter_contrib_nbextensions https://github.com/ipython-contrib/jupyter_contrib_nbextensi... that I think have't been ported yet
wlll 8y ago
I just started using Jupyter a couple of days ago as I'm starting to learn ML. I'm really impressed by it. Would love something similar for Ruby.
- gnulinux 8y ago
  There are jupyter kernels for dozens of languages. Search it in github.
- carreau 8y ago
  Good news ! It exists ! It's called Jupyter ! Just install one of the non-python kernels[1], for example the Ruby one [2], and create a new Ruby Notebook ! 1: https://github.com/jupyter/jupyter/wiki/Jupyter-kernels https://github.com/jupyter/jupyter/wiki/Jupyter-kernels 2: https://github.com/SciRuby/iruby https://github.com/SciRuby/iruby
Karishma1234 8y ago
Well deserved. There are few softwares that totally amazed me when I first used them. (I used Jupyter for the first time last week). I was like "Did they actually achieve this?"
- gnulinux 8y ago
  I've been using Jupyter for some time now since it was strongly recommended/almost-required in my school's classes. One thing I think Jupyter team achieved is reliability of common interface. Like, regardless what my data is, white noise, music, image, matrix, human face... I know I can easily output it and get some sane representation.
kelvin0 8y ago
I started using Jupyter + Python recently, can't say enough good things about the project. Sometimes you want to present data, graphics and have a bit of interactivity. The notebooks make it easy to share your code/data/graphics. And it beats a PowerPoint any day (for this use case anyway). Thanks Jupyter Team!
robohamburger 8y ago
I really like jupyter and I am looking forward to jupyter lab and where it takes computing. It is really great for solving one off problems or learning. The jupyter code itself while verbose is pretty extensible also. I just put together something that lets me connect to my spark kubernetes pod. I think being able to customize jupyter and add new kernels (languages) is where it becomes really powerful and awesome.
- dswalter 8y ago
  Jupyterlab is here. I've been using it basically full-time since August, but it recently went to Beta, so it's ready for use. The major downside is that it disables JS by default within notebooks, so if you're using Bokeh you'll have to install the jupyterlab extension, but the innovations around files, downloads, views on notebooks, etc. are worth the price of admission.
uptownfunk 8y ago
Wish they would add variable inspection to Jupiterlab!
rishabhparikh 8y ago
Cannot imagine liking data science nearly as much as I do if Jupyter didn't make it so easy to quickly test new ideas. Well-deserved.
ecesena 8y ago
I'm excited for this award, it's certainly well deserved. In my heart, I wish the "data community" would show more care for security (and, related, privacy), with deeper focus on features that simplify access control, and guidelines on how to enforce "reasonable defaults". I fear that Jupyter in many companies is becoming the next Jenkins, with unconstrained access to all data vs all infra, and this will lead to more and more incidents and leaks. I very much hope that recognitions like this one will foster not only better tools and support, but also best practice and security considerations. But, back to the focus of the post, congrats on this success!
- mistrial9 8y ago
  a CTO said recently .. we think of the base community edition as the place where we enable people to do things, and the corporate enterprise edition as where we prevent people from doing things. All the hardened security goes into the Enterprise Edition, for which they can pay.
- carreau 8y ago
  Thanks for your comment. One of the next focus area, where we are looking for funding and help, is making sure the right restrictions and permissions are in place. We probably will put that in the context of GDPR/HIPPA/FERPA and follow these guidelines to make Jupyter "Ready" for these framework. We can't say that Jupyter it itself compliant, as you need to see in which context it is deployed, but we want to make it as easy as possible for a team of researcher with low budget, or a companies with 1000+ user to make it easy to deploy a secure, auditable and safe Jupyter environement.
cozzyd 8y ago
Tools like Jupyter (and Mathematica) don't really match my mental model. I'm fine using them as a REPL (in which case they're like a bloated ipython or ROOT prompt), but as soon as I go back and change something, I get confused about the internal state.
- analog31 8y ago
  I don't know if it's exactly the same as what you're getting at, but I'm a heavy Jupyter user, and find that as notebooks get more elaborate, I have to work with great care to avoid problems with everything having global scope. So I have to refactor my code once in a while, get things into subroutines, etc. All of my good and bad coding habits are laid bare in Jupyter. Also, this is not for the faint of heart, but before closing a session, I do a "restart kernel and run all." This ensures that my notebook is running the way I'd think if I open it up later and try to re-run it.
  - dswalter 8y ago
    It's one of those difficult things. You have to restart and run all to make sure your code is working and your state hasn't gone fudgy, but you also don't want to re-run that cell that took 3 hours to run unless you have to. My personal best practice is to 'restart and run all' to run the entire thing before I commit it to github. Jupyter git integration is a whole separate bag of worms, as the combination of presentation/code violates some core git assumptions.
    - goerz 8y ago
      Any 3-hour calculation should be cached to disk, so that when you re-rerun the notebook you get the result instantaneously
- carreau 8y ago
  Internal state is often a problem, look at project like stichfix nodebook, and dataflow kernels, they make things a bit easier by re-executing cells. On of the issue is you always have internal state as soon as you interact with a data source or sink. If you read/write from a API, then rest is stateful. Your file system is stateful... etc. It's an interesting but hard problem, we'll be happy to have more help with. https://github.com/dataflownb/dfkernel https://github.com/dataflownb/dfkernel https://multithreaded.stitchfix.com/blog/2017/07/26/nodebook/ https://multithreaded.stitchfix.com/blog/2017/07/26/nodebook...
smortaz 8y ago
A huge congrats from Azure Notebooks which is entirely built around Jupyter. We actually started way before that by connecting the visual studio REPL to Jupyter and the whole experience (technically and people) has been delightful.
pilchardbreath 8y ago
The other "innovations" that ACM lists alongside were real innovations " Unix, TeX, S (R’s predecessor), the Web, Mosaic, Java, INGRES " now they are handing out awards for "copy commercial software but make it free" projects. Its funny how copying is a bad thing in an essay but applauded in software (as long as it has the right license).
vienno 8y ago
This is great. Without Jupyter, I doubt I would have started learning data science a few years ago.
cosmic_ape 8y ago
IPython's interactive shell is way more convenient than using the default python command line when trying things out. But I just don't get why one would need the whole Jupiter thing on top of that, except for maybe making presentations. One thing that I think is missing from IPython though is the ability to save a given state of the interpreter, with all the variables in it. So that one could preform a time consuming data loading/parsing once, and restart from that point if some variables get messed up. Jupiter cann't do that either, afaik.
- mistrial9 8y ago
  you get a 'narrative' with explanatory text, charting and code together, interactively building until you get something you like, as a single document. That document can then can be exported to HTML or pdf to share with others.
- mkl 8y ago
  Anything graphical or interactive: maths, science, data, web, machine learning, etc. Interactive includes interactive widgets, but also almost any kind of exploratory programming. Even stuff that's technically plain text is easier when you can display tables and other formatted text. E.g. I have a tiny little notebook that generates LaTeX code for a normal distribution table; it's a notebook because then I can display an HTML preview in a few lines of code. Jupyter can't save interpreter state - that seems essentially impossible without adding state-saving and -loading code to every single library and dependency.
  - cosmic_ape 8y ago
    >>Anything graphical or interactive but ipython is already well integrated with matplotlib in the --pylab mode, and is pretty interactive. Thats how I use it. re:saving state - its technically a nightmare, I agree. I though it might be possible at OS level - just dumping the whole process. There still would be issues of what to do with open files, network connections, etc, but somehow it seems that in many use cases that would be enough.
    - 8y ago
carljv 8y ago
The Jupyter team deserves every accolade they get and more. The console, notebook, and now JupyterLab are some of the key reasons why Python's data ecosystem thrives. I think Jupyter notebooks are quite useful as "rich display" shells. I often use them to set up simple interactive demos or tutorials to show folks or keep notes or scratch for myself. That being said, I do think the "reproducibility" aspect of the notebook is overblown for the reasons other comments cite. Notebooks are hard to version control and diff, and are easy to "corrupt." I often see Jupyter notebooks described as "literate programs," and I really don't think that's an apt description. The notebook is basically the IPython shell exposed to the browser where you can display rich output. This is where I think the R ecosystem's approach to the problem is better (a bit like org-mode & org-babel). For them, there is a literate program in plain text. Code blocks can be executed interactively and results displayed inline by a "viewer" on the document (like that provided by RStudio), but executing code doesn't change the source code of the program, and diffs/versions are only created by editing the source. At any point, the file can be "compiled" or processed into a static output document like HTML or PDF. This is essentially literate programming but with an intermediate "interactive" feature facilitated by an external program. RMarkdown source doesn't know its being interacted with or executed, and you can edit it like any other literate program. Interaction, reproducibility, and publication have fundamental tensions with each other. Jupyter notebooks are trying to do all three in the same software/format, and my sense is that they're starting to strain against those tensions.
- your-nanny 8y ago
  I agree, 120%. I like the r approach so much more.
  - carljv 8y ago
    I mean, as a medium for interactive exploration where you might want graphs and widgets or other rich/dynamic output, I still think the notebook is superior. But as a medium for developing complete, share-able, reproducible data analyses, I do think R has the upper hand.
    - 8y ago

714 points by williamstein 8y ago

monkeydust 8y ago

I am non programmer business guy who 'excels' a lot. Been teaching myself pandas through notebook. Awesome combination.

criddell 8y ago

Do you have a favorite tutorial or simple use case that helped you grok it?

zitterbewegung 8y ago

Wow this is awesome! It got the Award quickly also I think Juypter hasn't been around as long as other recipients.

carreau 8y ago
As we say in the blog post, the Jupyter Name has been around only since 2014, but the work started in 2001. So 17 years is a good check of time !
- filmor 8y ago
  Though the part that people mostly identify with the project, the Jupyter Notebook was introduced only in 2011. I remember that vividly as I was writing my thesises at that time and spending quite some time playing around with it :)
  - carreau 8y ago
    Though the current notebook is the 6th prototype, so there've been quite some work before actually being made "public". Hope the notebook didn't distract you too much from your PhD !
    - po84 8y ago
      If open source didn't distract from ph.d's, we wouldn't have had ipython! ;)
    - pgbovine 8y ago
      overnight successes are usually 10-15 years in the making; so much goes on before the public release

angrygoat 8y ago

So well deserved. Jupyter is critical infrastructure, helping the scientific community address the reproducibility crisis. It's so great that it's free and open source, and can be shared, used and contributed to by scientists all around the world.

scaryclam 8y ago
Not only that, it's great for tutorial material as well!

msravi 8y ago

Trivia: Ju-py-ter = Julia Python Terminal Although it's gone far beyond just Julia and Python now. Edit: Ahurmazda is right. ...the core programming languages supported by Jupyter are Julia, Python and R. While the name Jupyter is not a direct acronym for these languages, it nods its head in those directions. In particular, the "y" in the middle of Jupyter was chosen to honor our Python heritage. https://github.com/jupyter/design/wiki/Jupyter-Logo https://github.com/jupyter/design/wiki/Jupyter-Logo

osteele 8y ago
Also worth noting: Hydrogen, the Jupyter plugin for Atom[1][2]. Hydrogen is a fantastically well-executed and useful piece of software. (As is Jupyter.) I’ve used it in my own programming, and also in teaching introductory software development, where it’s a helpful transition from Jupyter notebooks (which are used in the early part of the course, for problem sets / reading journals) to text files and code editors (which are introduced later, and used for team projects, and for projects that use Flask or PyGame). But also — relevant to this sub-thread — “Hydrogen” (it relates “Jupyter” to “Atom”) has to be one of the best project names ever. It’s right up there with “Pyramid Scheme”[3]. [1] https://nteract.io/atom https://nteract.io/atom [2] https://atom.io/packages/hydrogen https://atom.io/packages/hydrogen [3] http://www.michaelburge.us/2017/11/28/write-your-next-ethereum-contract-in-pyramid-scheme.html http://www.michaelburge.us/2017/11/28/write-your-next-ethere...
ahurmazda 8y ago
Julia Python R
takluyver 8y ago
ahurmazda pointed out that you're thinking of 'Jupyter Python R'. And this is a reference, not an 'equals'. The name is also in homage to Galileo's notebooks recording the discovery of the moons of Jupiter (the four we now call the 'Galilean moons'), and also to a bar called Jupiter in Berkeley, which the core team has visited quite often. The last one's more like a funny coincidence, though.
- 8y ago

sytse 8y ago

Jupyter is awesome, well deserved award. Our CTO is working to integrate JupyterLab and JupyterHub directly into GitLab.

evanb 8y ago
Is there a way to associate and store large data sets, as well? I don't necessarily want them in git (hundreds of GB, for example), but I do want my notebooks to see them...
carreau 8y ago
Thanks, we love GitLab as well ! Would appreciate to have your feedback on how to make integration easier and get productive and constructive discussion. Will we see GitLab at JupyterCon this august in NY ?
- sytse 8y ago
  We probably won't be a JupyterCon, but my CTO would love to have a call. Is it OK if Eliran, responsible for partnerships, reaches out to you for a video call?
  - carreau 8y ago
    Sure. We have team member across the bay in Berkeley, we can also cross over to see you or vice versa.
    - sytse 8y ago
      Great, we'll make it happen. It will probably be a video call since our CTO and co-founder is living in the Ukraine.
      - sls 8y ago
        fyi - Why Ukrainians Hate When You Say 'The Ukraine' http://www.newsweek.com/why-ukrainians-hate-when-you-say-ukraine-392903 http://www.newsweek.com/why-ukrainians-hate-when-you-say-ukr...

nmca 8y ago

Thoroughly deserved - Jupyter is helping push science forwards.

oh-kumudo 8y ago

Congratulations! Jupyter is such a gem. It has become such a critical piece in the whole data analytics/ML landscape that I can't think of living without.

gbrown 8y ago

Am I the only person in the universe who doesn't like Jupyter? I much prefer tools like Rmarkdown and Sweave.

tincholio 8y ago
Nope... you're not the only one
jasongrout 8y ago
We would love to hear what you like about Rmarkdown and Sweave. Jupyter tooling is always improving, and we are very interested in engaging with users about their needs, and helping grow the ecosystem to be able to address those.
- goerz 8y ago
  The number one feature request by far at this point would be collaborative features in the notebook. Not really because I actually want to collaboratively edit a notebook with others, but because I would like to open the same notebook in two separate browser windows (side-by-side), and edit a cell at the top of the notebook in one, and a cell at the bottom in the other, and have things get merged automatically. Or, not having to worry that I left a (remote) notebook open on my work computer when I connect to the same server from home, causing parts of the notebook to be accidentally overwritten.
  - targafarian 8y ago
    JupyterLab has the ability to open multiple "views" on the same file. http://jupyterlab.readthedocs.io/en/latest/user/notebook.html?highlight=views http://jupyterlab.readthedocs.io/en/latest/user/notebook.htm...
    - goerz 8y ago
      That's amazing! I'll have to try out JupyterLab sometime very soon. I was holding out because I'm using a whole bunch of plugins for the classic notebook from https://github.com/ipython-contrib/jupyter_contrib_nbextensions https://github.com/ipython-contrib/jupyter_contrib_nbextensi... that I think have't been ported yet

wlll 8y ago

I just started using Jupyter a couple of days ago as I'm starting to learn ML. I'm really impressed by it. Would love something similar for Ruby.

gnulinux 8y ago
There are jupyter kernels for dozens of languages. Search it in github.
carreau 8y ago
Good news ! It exists ! It's called Jupyter ! Just install one of the non-python kernels[1], for example the Ruby one [2], and create a new Ruby Notebook ! 1: https://github.com/jupyter/jupyter/wiki/Jupyter-kernels https://github.com/jupyter/jupyter/wiki/Jupyter-kernels 2: https://github.com/SciRuby/iruby https://github.com/SciRuby/iruby

Karishma1234 8y ago

Well deserved. There are few softwares that totally amazed me when I first used them. (I used Jupyter for the first time last week). I was like "Did they actually achieve this?"

gnulinux 8y ago
I've been using Jupyter for some time now since it was strongly recommended/almost-required in my school's classes. One thing I think Jupyter team achieved is reliability of common interface. Like, regardless what my data is, white noise, music, image, matrix, human face... I know I can easily output it and get some sane representation.

kelvin0 8y ago

I started using Jupyter + Python recently, can't say enough good things about the project. Sometimes you want to present data, graphics and have a bit of interactivity. The notebooks make it easy to share your code/data/graphics. And it beats a PowerPoint any day (for this use case anyway). Thanks Jupyter Team!

robohamburger 8y ago

I really like jupyter and I am looking forward to jupyter lab and where it takes computing. It is really great for solving one off problems or learning. The jupyter code itself while verbose is pretty extensible also. I just put together something that lets me connect to my spark kubernetes pod. I think being able to customize jupyter and add new kernels (languages) is where it becomes really powerful and awesome.

dswalter 8y ago
Jupyterlab is here. I've been using it basically full-time since August, but it recently went to Beta, so it's ready for use. The major downside is that it disables JS by default within notebooks, so if you're using Bokeh you'll have to install the jupyterlab extension, but the innovations around files, downloads, views on notebooks, etc. are worth the price of admission.

uptownfunk 8y ago

Wish they would add variable inspection to Jupiterlab!

rishabhparikh 8y ago

Cannot imagine liking data science nearly as much as I do if Jupyter didn't make it so easy to quickly test new ideas. Well-deserved.

ecesena 8y ago

I'm excited for this award, it's certainly well deserved. In my heart, I wish the "data community" would show more care for security (and, related, privacy), with deeper focus on features that simplify access control, and guidelines on how to enforce "reasonable defaults". I fear that Jupyter in many companies is becoming the next Jenkins, with unconstrained access to all data vs all infra, and this will lead to more and more incidents and leaks. I very much hope that recognitions like this one will foster not only better tools and support, but also best practice and security considerations. But, back to the focus of the post, congrats on this success!

mistrial9 8y ago
a CTO said recently .. we think of the base community edition as the place where we enable people to do things, and the corporate enterprise edition as where we prevent people from doing things. All the hardened security goes into the Enterprise Edition, for which they can pay.
carreau 8y ago
Thanks for your comment. One of the next focus area, where we are looking for funding and help, is making sure the right restrictions and permissions are in place. We probably will put that in the context of GDPR/HIPPA/FERPA and follow these guidelines to make Jupyter "Ready" for these framework. We can't say that Jupyter it itself compliant, as you need to see in which context it is deployed, but we want to make it as easy as possible for a team of researcher with low budget, or a companies with 1000+ user to make it easy to deploy a secure, auditable and safe Jupyter environement.

cozzyd 8y ago

Tools like Jupyter (and Mathematica) don't really match my mental model. I'm fine using them as a REPL (in which case they're like a bloated ipython or ROOT prompt), but as soon as I go back and change something, I get confused about the internal state.

analog31 8y ago
I don't know if it's exactly the same as what you're getting at, but I'm a heavy Jupyter user, and find that as notebooks get more elaborate, I have to work with great care to avoid problems with everything having global scope. So I have to refactor my code once in a while, get things into subroutines, etc. All of my good and bad coding habits are laid bare in Jupyter. Also, this is not for the faint of heart, but before closing a session, I do a "restart kernel and run all." This ensures that my notebook is running the way I'd think if I open it up later and try to re-run it.
- dswalter 8y ago
  It's one of those difficult things. You have to restart and run all to make sure your code is working and your state hasn't gone fudgy, but you also don't want to re-run that cell that took 3 hours to run unless you have to. My personal best practice is to 'restart and run all' to run the entire thing before I commit it to github. Jupyter git integration is a whole separate bag of worms, as the combination of presentation/code violates some core git assumptions.
  - goerz 8y ago
    Any 3-hour calculation should be cached to disk, so that when you re-rerun the notebook you get the result instantaneously
carreau 8y ago
Internal state is often a problem, look at project like stichfix nodebook, and dataflow kernels, they make things a bit easier by re-executing cells. On of the issue is you always have internal state as soon as you interact with a data source or sink. If you read/write from a API, then rest is stateful. Your file system is stateful... etc. It's an interesting but hard problem, we'll be happy to have more help with. https://github.com/dataflownb/dfkernel https://github.com/dataflownb/dfkernel https://multithreaded.stitchfix.com/blog/2017/07/26/nodebook/ https://multithreaded.stitchfix.com/blog/2017/07/26/nodebook...

smortaz 8y ago

A huge congrats from Azure Notebooks which is entirely built around Jupyter. We actually started way before that by connecting the visual studio REPL to Jupyter and the whole experience (technically and people) has been delightful.

pilchardbreath 8y ago

The other "innovations" that ACM lists alongside were real innovations " Unix, TeX, S (R’s predecessor), the Web, Mosaic, Java, INGRES " now they are handing out awards for "copy commercial software but make it free" projects. Its funny how copying is a bad thing in an essay but applauded in software (as long as it has the right license).

vienno 8y ago

This is great. Without Jupyter, I doubt I would have started learning data science a few years ago.

cosmic_ape 8y ago

IPython's interactive shell is way more convenient than using the default python command line when trying things out. But I just don't get why one would need the whole Jupiter thing on top of that, except for maybe making presentations. One thing that I think is missing from IPython though is the ability to save a given state of the interpreter, with all the variables in it. So that one could preform a time consuming data loading/parsing once, and restart from that point if some variables get messed up. Jupiter cann't do that either, afaik.

mistrial9 8y ago
you get a 'narrative' with explanatory text, charting and code together, interactively building until you get something you like, as a single document. That document can then can be exported to HTML or pdf to share with others.
mkl 8y ago
Anything graphical or interactive: maths, science, data, web, machine learning, etc. Interactive includes interactive widgets, but also almost any kind of exploratory programming. Even stuff that's technically plain text is easier when you can display tables and other formatted text. E.g. I have a tiny little notebook that generates LaTeX code for a normal distribution table; it's a notebook because then I can display an HTML preview in a few lines of code. Jupyter can't save interpreter state - that seems essentially impossible without adding state-saving and -loading code to every single library and dependency.
- cosmic_ape 8y ago
  >>Anything graphical or interactive but ipython is already well integrated with matplotlib in the --pylab mode, and is pretty interactive. Thats how I use it. re:saving state - its technically a nightmare, I agree. I though it might be possible at OS level - just dumping the whole process. There still would be issues of what to do with open files, network connections, etc, but somehow it seems that in many use cases that would be enough.
  - 8y ago

carljv 8y ago

The Jupyter team deserves every accolade they get and more. The console, notebook, and now JupyterLab are some of the key reasons why Python's data ecosystem thrives. I think Jupyter notebooks are quite useful as "rich display" shells. I often use them to set up simple interactive demos or tutorials to show folks or keep notes or scratch for myself. That being said, I do think the "reproducibility" aspect of the notebook is overblown for the reasons other comments cite. Notebooks are hard to version control and diff, and are easy to "corrupt." I often see Jupyter notebooks described as "literate programs," and I really don't think that's an apt description. The notebook is basically the IPython shell exposed to the browser where you can display rich output. This is where I think the R ecosystem's approach to the problem is better (a bit like org-mode & org-babel). For them, there is a literate program in plain text. Code blocks can be executed interactively and results displayed inline by a "viewer" on the document (like that provided by RStudio), but executing code doesn't change the source code of the program, and diffs/versions are only created by editing the source. At any point, the file can be "compiled" or processed into a static output document like HTML or PDF. This is essentially literate programming but with an intermediate "interactive" feature facilitated by an external program. RMarkdown source doesn't know its being interacted with or executed, and you can edit it like any other literate program. Interaction, reproducibility, and publication have fundamental tensions with each other. Jupyter notebooks are trying to do all three in the same software/format, and my sense is that they're starting to strain against those tensions.

your-nanny 8y ago
I agree, 120%. I like the r approach so much more.
- carljv 8y ago
  I mean, as a medium for interactive exploration where you might want graphs and widgets or other rich/dynamic output, I still think the notebook is superior. But as a medium for developing complete, share-able, reproducible data analyses, I do think R has the upper hand.
  - 8y ago