7 ms·

Jupyter Notebook 4.2 Released

JesseAldridge 10y ago
I had my first experience with Jupyter last weekend when I was trying to learn about document clustering with Python. It seems like a cool idea, but in practice ended up being kind of annoying: https://github.com/brandomr/document_cluster/issues/7 https://github.com/brandomr/document_cluster/issues/7
- entee 10y ago
  Jupyter is great for prototyping and playing with ideas, essentially its great it you want persistent data. But after you more or less know what you want and loading the initial dataset isn't a constant annoyance you're usually better off in an IDE or other real programming environment. I use it to ask a whole bunch of exploratory questions about a dataset then productionize the result in PyCharm (my preference, other ways work great too :)
  - zmmmmm 10y ago
    This is actually my main problem with the whole idea of these "notebooks". They explicitly encourage exactly the kind of ad hoc coding and practises that plague a lot of scientific work. It's nearly impossible to practise good software engineering while inside one of these things. I know the rationale will be that this is for ad-hoc exploration and the code should be rewritten / redesigned when it's moved into an app, but just like all prototype code that has ever been written, that is not what happens. I would love something that combines this style with support for good software practices. For example, that let's you seamlessly move snippets of code into functions, classes, modules, and then create tests for them. RStudio is actually the closest I have found, which is ironic since as a language R is horrible for encouraging good software practices.
- jimktrains2 10y ago
  I teach an internal class on python at my office, and the notebook makes it significantly easier to work and play with code. It's a step between a REPL and a file you load and run each time you make a chance. It speeds up the dev/test cycle significantly.
- minimaxir 10y ago
  Jupyter notebooks are for running ad hoc blocks of code. The biggest advantage of doing so is being able to check the output of each block to make sure the output is expected. This feature in particular makes for a great tool for tutorials. (Example of mine: https://github.com/minimaxir/facebook-page-post-scraper/blob/master/how_to_build_facebook_scraper.ipynb https://github.com/minimaxir/facebook-page-post-scraper/blob...) It is definitely not a tool to replace a typical Python workflow.
- makecheck 10y ago
  It works really well in a class scenario. A professor can just show a notebook on the screen “live”: walk through it, or even change things and demonstrate the effects. And when finished, the whole thing can be posted as a file for students to download and try themselves.
vhost- 10y ago
While working on a data science team some months ago, these notebooks helped me build something that explained, in detail and at a high level, the implementation details of an algorithm to sales and others not familiar with data science techniques. It was awesome and so easy. I also used them when we did a capture the flag contest to help explain visually how a multi time pad vulnerability works.
- vhost- 10y ago
  Here is a link to an html export of the jupyter notebook I made for the many-time-pad experiment: https://kyleterry.com/natas11.html https://kyleterry.com/natas11.html
jypepin 10y ago
One of our data analyst had fun drawing all the shots from Kobe during his whole career. He did everything on jupyter, which I discovered at that moment, and was mind blown on how well it worked and how powerful it is. I don't think I would ever use this tool as a seasoned software engineer, but I can definitely see the power it has for newer people who want to learn, or simply people like him who know a little bit of code and just wanted to run it. Congrats to the team building this tool!
- aabaker99 10y ago
  can you comment on why you wouldn't use it? I am somewhere in the middle of software engineering and (biological) data science. I have seen some jupyter notebooks as companions to genomics software and I enjoyed those presentations. I was considering using it in the future based on that; so, I wanted to hear your thoughts.
  - delluminatus 10y ago
    As another "software engineer", I can say that we simply don't ever need a tool like Jupyter for our jobs. It's excellent for exploratory programming, research, and publishing -- but it's not really meant for software development.
  - jypepin 10y ago
    I kind of missed the point of how notebook are used (and should be) and it seems actually very interesting :) So you should definitely look into it and see if you like doing whatever you do with them!
    - aeroevan 10y ago
      Yeah, the coolest thing I just found out is that nbconvert can be used to turn your python/matplotlib/markdown to reveal.js presentations.
- minimaxir 10y ago
  Your comment implies that using a Jupyter notebook is a programming crutch for those who are not skilled. That's far from the case; Jupyter notebooks allow for logical separation of code and output, which is important for comprehension. It also allows for reproducibility of results, which is arguably even more important, especially in the data science case.
benatkin 10y ago
There's also a newer alternative called Zeppelin: https://zeppelin.incubator.apache.org https://zeppelin.incubator.apache.org Comparison: https://www.linkedin.com/pulse/comprehensive-comparison-jupyter-vs-zeppelin-hoc-q-phan-mba- https://www.linkedin.com/pulse/comprehensive-comparison-jupy...
- bpchaps 10y ago
  Thanks for the links! I've been having a lot of frustrations with jupyter and since it's almost impossible to find anything technical on google these days, I haven't been able to find any good alternatives.
  - asimuvPR 10y ago
    I've found that searching through duckduckgo with the !g command improves results. Try it out. :)
    - bpchaps 10y ago
      That's exactly what I do, actually. :p
      - asimuvPR 10y ago
        Heh, weird how you have to use another engine to improve results. D:
- nhaliday 10y ago
  Another competitor (with a strong focus on polyglot use): http://beakernotebook.com/ http://beakernotebook.com/. Made by the people at https://www.twosigma.com/ https://www.twosigma.com/.
  - zmmmmm 10y ago
    I love beaker. I spent a week or so trying to force myself to use Jupyter since others in my team use it, but I bailed out and went back to beaker eventually - it's got everything Jupyter has but is all round nicer to use. It's not without it's problems, but neither is Jupyter by a long shot. edit: fixed a word
tdaltonc 10y ago
Every time I try to do some data crunching in a python notebook, I find that I really miss the variable explorer from spyder.
- aeroevan 10y ago
  You might like Yhat's rodeo: https://github.com/yhat/rodeo https://github.com/yhat/rodeo I haven't used it too much, but it looks interesting.
  - tdaltonc 10y ago
    I don't care for their python implementation of ggplot, but that looks really shinny. I will definitely check that out.
jnpatel 10y ago
I find Jupyter great for the single-user use case, but I'm often frustrated with how it "breaks" with version control, when collaborating.
- anonetal 10y ago
  There are several startups in that space, e.g., Domino Data Labs, that have been trying to make it easy to do collaborative/versioned notebooks. I have only seen the product videos so don't know how well they work etc.
- williamstein 10y ago
  SageMathCloud has collaborative Jupyter notebooks (like Google docs) with TimeTravel and multiple cursors.
- jre 10y ago
  I think the jupyter devs are aware of this and trying to find a fix. There is an enhancement proposal on how to get diff to work with notebooks : https://github.com/jupyter/enhancement-proposals/blob/master/notebook-diff/notebook-diff.md https://github.com/jupyter/enhancement-proposals/blob/master... And I think that's the project that should implement this : https://github.com/jupyter/nbdime https://github.com/jupyter/nbdime
- filmor 10y ago
  As long as you strip outputs and prompt numbers, diffing works quite well. I've been using this commit hook for quite a while: http://stackoverflow.com/questions/25178118/opening-ipython-notebook-without-output/25178426#25178426 http://stackoverflow.com/questions/25178118/opening-ipython-...
deleted 10y ago
[deleted]
jamesdutc 10y ago
For some background on the direction of the Jupyter project, check out this recent talk at PyData Amsterdam (http://pydata.org/amsterdam2016 http://pydata.org/amsterdam2016) by Min Ragan-Kelley & Thomas Kluyver. They talk about how Jupyter has "evolved from a Python-specific tool to a general data science tool that supports many different languages." https://www.youtube.com/watch?v=T385txAYSt8 https://www.youtube.com/watch?v=T385txAYSt8
thom 10y ago
Probably the longest time-to-knowing-what-the-hell-this-thing-actually-is I've ever seen on HN. Clicked on the link, clicked on a Github link, clicked to the Github root, clicked on the link to the project's site, clicked on the first item in the table of contents, got a vague idea what it was. I feel grumpy and old. :(
- tptacek 10y ago
  "Jupyter is the extraction of the language-agnostic stuff from IPython; IPython is a really important web-based scientific computing environment."
- kasbah 10y ago
  > The IPython Notebook is now known as the Jupyter Notebook. It is an interactive computational environment, in which you can combine code execution, rich text, mathematics, plots and rich media. The best place to start is probably https://try.jupyter.org/ https://try.jupyter.org/
  - rasz_pl 10y ago
    /opt/conda/lib/python3.4/site-packages/IPython/kernel/__init__.py:13: ShimWarning: The `IPython.kernel` package has been deprecated. You should import from ipykernel or jupyter_client instead. "You should import from ipykernel or jupyter_client instead.", ShimWarning)
- nxzero 10y ago
  I explain notebooks as just how-to's with embedded coded/graphics/data that are editable in realtime; possible that's just as vague, but notebook sounds like a plaintext editor to me; term comes from logs, kept in notebooks, used in science labs to note/reproduce prior work.
- alyandon 10y ago
  It's really annoying to get caught in a blog.domain.com subdomain without any obvious links back to the main domain.
KKKKkkkk1 10y ago
Anyone knows why Debian/Ubuntu is still on version 2.3? Is it because the Jupyter devs are all Mac aficionados?
- rafunzi 10y ago
  It's not. You're probably doing it wrong. http://jupyter.readthedocs.org/en/latest/install.html#new-to-python-and-jupyter http://jupyter.readthedocs.org/en/latest/install.html#new-to... use something like: >pip install jupyter -U
e0m 10y ago
If you use Atom, I would highly recommend looking at https://atom.io/packages/hydrogen https://atom.io/packages/hydrogen It uses Jupyter under the hood to do some serious inline magic.
- ced 10y ago
  Is Hydrogen a notebook replacement? The demo makes it look like a fancy REPL. Does it have section headers, latex support, etc.?
  - deleted 10y ago
    [deleted]
  - lisnake 10y ago
    No, Hydrogen is not a jupyter notebook replacement, it cannot open .ipynb files
enobrev 10y ago
"The Jupyter Notebook is a web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, machine learning and much more." http://jupyter.org/ http://jupyter.org/
- KKKKkkkk1 10y ago
  Thanks for explaining what Jupyter is. This is certainly useful since it's hard to find this information from the original link.
jxy 10y ago
It's fun to play with code and try out new ideas in such a rich interactive environment. When you want to get the work done in production scenario, the shortcomings of unable to use version control and the overhead of interactive environment just kill it. What works is that you get a subset of your data and try to develop some code to process it and generate a handful of graphs. You can then save the code in its true text form and edit with your favorite editor, and run it on your real data.
analog31 10y ago
Jupyter/Python has been game changing for me. I'm not a commercial software developer, but a scientist working on technology development. I've been programming for 30+ years. Jupyter has become my lab notebook. In the past, I always had illegible, disorganized notebooks, files, and program code, all over the place. A Jupyter notebook lets me organize all of that stuff in one place, in a narrative fashion, allowing me to reconstruct what I did, long after I've forgotten the details. The reasons for open communication of methods and results to the public, also apply to internal work. My notebooks become my reports. I've abandoned PowerPoint, and my colleagues, including managers, don't seem to mind. Seeing the actual work might actually give them a feeling of involvement, like inviting them into the lab. They're also a good way of communicating a prototype of a process to the software development team, when an idea ends up in a product. Even if they don't like Python, the programmers can read and understand it. I can actually run some of my data acquisition code directly within Jupyter. A code cell that spits out an inline graph is practically the default interface for a lot of this kind of work, so I don't have to build a unique GUI for every kind of test. This speeds up incremental refinement of an experimental technique, even if the routines that I write end up in a "straight" Python program when it's time to let an experiment run for a few hours or days. Granted, Jupyter won't turn bad programmers into good. Learning good programming methods is still a gap in the education of scientists.
jasongrout 10y ago
Thanks for the feedback on the announcement blog post. I added a short description and link to the project home page at the top of the blog post after reading the feedback here. Thanks!

I had my first experience with Jupyter last weekend when I was trying to learn about document clustering with Python. It seems like a cool idea, but in practice ended up being kind of annoying: https://github.com/brandomr/document_cluster/issues/7 https://github.com/brandomr/document_cluster/issues/7

entee 10y ago
Jupyter is great for prototyping and playing with ideas, essentially its great it you want persistent data. But after you more or less know what you want and loading the initial dataset isn't a constant annoyance you're usually better off in an IDE or other real programming environment. I use it to ask a whole bunch of exploratory questions about a dataset then productionize the result in PyCharm (my preference, other ways work great too :)
- zmmmmm 10y ago
  This is actually my main problem with the whole idea of these "notebooks". They explicitly encourage exactly the kind of ad hoc coding and practises that plague a lot of scientific work. It's nearly impossible to practise good software engineering while inside one of these things. I know the rationale will be that this is for ad-hoc exploration and the code should be rewritten / redesigned when it's moved into an app, but just like all prototype code that has ever been written, that is not what happens. I would love something that combines this style with support for good software practices. For example, that let's you seamlessly move snippets of code into functions, classes, modules, and then create tests for them. RStudio is actually the closest I have found, which is ironic since as a language R is horrible for encouraging good software practices.
jimktrains2 10y ago
I teach an internal class on python at my office, and the notebook makes it significantly easier to work and play with code. It's a step between a REPL and a file you load and run each time you make a chance. It speeds up the dev/test cycle significantly.
minimaxir 10y ago
Jupyter notebooks are for running ad hoc blocks of code. The biggest advantage of doing so is being able to check the output of each block to make sure the output is expected. This feature in particular makes for a great tool for tutorials. (Example of mine: https://github.com/minimaxir/facebook-page-post-scraper/blob/master/how_to_build_facebook_scraper.ipynb https://github.com/minimaxir/facebook-page-post-scraper/blob...) It is definitely not a tool to replace a typical Python workflow.
makecheck 10y ago
It works really well in a class scenario. A professor can just show a notebook on the screen “live”: walk through it, or even change things and demonstrate the effects. And when finished, the whole thing can be posted as a file for students to download and try themselves.

vhost- 10y ago

While working on a data science team some months ago, these notebooks helped me build something that explained, in detail and at a high level, the implementation details of an algorithm to sales and others not familiar with data science techniques. It was awesome and so easy. I also used them when we did a capture the flag contest to help explain visually how a multi time pad vulnerability works.

vhost- 10y ago
Here is a link to an html export of the jupyter notebook I made for the many-time-pad experiment: https://kyleterry.com/natas11.html https://kyleterry.com/natas11.html

jypepin 10y ago

One of our data analyst had fun drawing all the shots from Kobe during his whole career. He did everything on jupyter, which I discovered at that moment, and was mind blown on how well it worked and how powerful it is. I don't think I would ever use this tool as a seasoned software engineer, but I can definitely see the power it has for newer people who want to learn, or simply people like him who know a little bit of code and just wanted to run it. Congrats to the team building this tool!

aabaker99 10y ago
can you comment on why you wouldn't use it? I am somewhere in the middle of software engineering and (biological) data science. I have seen some jupyter notebooks as companions to genomics software and I enjoyed those presentations. I was considering using it in the future based on that; so, I wanted to hear your thoughts.
- delluminatus 10y ago
  As another "software engineer", I can say that we simply don't ever need a tool like Jupyter for our jobs. It's excellent for exploratory programming, research, and publishing -- but it's not really meant for software development.
- jypepin 10y ago
  I kind of missed the point of how notebook are used (and should be) and it seems actually very interesting :) So you should definitely look into it and see if you like doing whatever you do with them!
  - aeroevan 10y ago
    Yeah, the coolest thing I just found out is that nbconvert can be used to turn your python/matplotlib/markdown to reveal.js presentations.
minimaxir 10y ago
Your comment implies that using a Jupyter notebook is a programming crutch for those who are not skilled. That's far from the case; Jupyter notebooks allow for logical separation of code and output, which is important for comprehension. It also allows for reproducibility of results, which is arguably even more important, especially in the data science case.

benatkin 10y ago

There's also a newer alternative called Zeppelin: https://zeppelin.incubator.apache.org https://zeppelin.incubator.apache.org Comparison: https://www.linkedin.com/pulse/comprehensive-comparison-jupyter-vs-zeppelin-hoc-q-phan-mba- https://www.linkedin.com/pulse/comprehensive-comparison-jupy...

bpchaps 10y ago
Thanks for the links! I've been having a lot of frustrations with jupyter and since it's almost impossible to find anything technical on google these days, I haven't been able to find any good alternatives.
- asimuvPR 10y ago
  I've found that searching through duckduckgo with the !g command improves results. Try it out. :)
  - bpchaps 10y ago
    That's exactly what I do, actually. :p
    - asimuvPR 10y ago
      Heh, weird how you have to use another engine to improve results. D:
nhaliday 10y ago
Another competitor (with a strong focus on polyglot use): http://beakernotebook.com/ http://beakernotebook.com/. Made by the people at https://www.twosigma.com/ https://www.twosigma.com/.
- zmmmmm 10y ago
  I love beaker. I spent a week or so trying to force myself to use Jupyter since others in my team use it, but I bailed out and went back to beaker eventually - it's got everything Jupyter has but is all round nicer to use. It's not without it's problems, but neither is Jupyter by a long shot. edit: fixed a word

tdaltonc 10y ago

Every time I try to do some data crunching in a python notebook, I find that I really miss the variable explorer from spyder.

aeroevan 10y ago
You might like Yhat's rodeo: https://github.com/yhat/rodeo https://github.com/yhat/rodeo I haven't used it too much, but it looks interesting.
- tdaltonc 10y ago
  I don't care for their python implementation of ggplot, but that looks really shinny. I will definitely check that out.

jnpatel 10y ago

I find Jupyter great for the single-user use case, but I'm often frustrated with how it "breaks" with version control, when collaborating.

anonetal 10y ago
There are several startups in that space, e.g., Domino Data Labs, that have been trying to make it easy to do collaborative/versioned notebooks. I have only seen the product videos so don't know how well they work etc.
williamstein 10y ago
SageMathCloud has collaborative Jupyter notebooks (like Google docs) with TimeTravel and multiple cursors.
jre 10y ago
I think the jupyter devs are aware of this and trying to find a fix. There is an enhancement proposal on how to get diff to work with notebooks : https://github.com/jupyter/enhancement-proposals/blob/master/notebook-diff/notebook-diff.md https://github.com/jupyter/enhancement-proposals/blob/master... And I think that's the project that should implement this : https://github.com/jupyter/nbdime https://github.com/jupyter/nbdime
filmor 10y ago
As long as you strip outputs and prompt numbers, diffing works quite well. I've been using this commit hook for quite a while: http://stackoverflow.com/questions/25178118/opening-ipython-notebook-without-output/25178426#25178426 http://stackoverflow.com/questions/25178118/opening-ipython-...

deleted 10y ago

[deleted]

jamesdutc 10y ago

For some background on the direction of the Jupyter project, check out this recent talk at PyData Amsterdam (http://pydata.org/amsterdam2016 http://pydata.org/amsterdam2016) by Min Ragan-Kelley & Thomas Kluyver. They talk about how Jupyter has "evolved from a Python-specific tool to a general data science tool that supports many different languages." https://www.youtube.com/watch?v=T385txAYSt8 https://www.youtube.com/watch?v=T385txAYSt8

thom 10y ago

Probably the longest time-to-knowing-what-the-hell-this-thing-actually-is I've ever seen on HN. Clicked on the link, clicked on a Github link, clicked to the Github root, clicked on the link to the project's site, clicked on the first item in the table of contents, got a vague idea what it was. I feel grumpy and old. :(

tptacek 10y ago
"Jupyter is the extraction of the language-agnostic stuff from IPython; IPython is a really important web-based scientific computing environment."
kasbah 10y ago
> The IPython Notebook is now known as the Jupyter Notebook. It is an interactive computational environment, in which you can combine code execution, rich text, mathematics, plots and rich media. The best place to start is probably https://try.jupyter.org/ https://try.jupyter.org/
- rasz_pl 10y ago
  /opt/conda/lib/python3.4/site-packages/IPython/kernel/__init__.py:13: ShimWarning: The `IPython.kernel` package has been deprecated. You should import from ipykernel or jupyter_client instead. "You should import from ipykernel or jupyter_client instead.", ShimWarning)
nxzero 10y ago
I explain notebooks as just how-to's with embedded coded/graphics/data that are editable in realtime; possible that's just as vague, but notebook sounds like a plaintext editor to me; term comes from logs, kept in notebooks, used in science labs to note/reproduce prior work.
alyandon 10y ago
It's really annoying to get caught in a blog.domain.com subdomain without any obvious links back to the main domain.

KKKKkkkk1 10y ago

Anyone knows why Debian/Ubuntu is still on version 2.3? Is it because the Jupyter devs are all Mac aficionados?

rafunzi 10y ago
It's not. You're probably doing it wrong. http://jupyter.readthedocs.org/en/latest/install.html#new-to-python-and-jupyter http://jupyter.readthedocs.org/en/latest/install.html#new-to... use something like: >pip install jupyter -U

e0m 10y ago

If you use Atom, I would highly recommend looking at https://atom.io/packages/hydrogen https://atom.io/packages/hydrogen It uses Jupyter under the hood to do some serious inline magic.

ced 10y ago
Is Hydrogen a notebook replacement? The demo makes it look like a fancy REPL. Does it have section headers, latex support, etc.?
- deleted 10y ago
  [deleted]
- lisnake 10y ago
  No, Hydrogen is not a jupyter notebook replacement, it cannot open .ipynb files

enobrev 10y ago

"The Jupyter Notebook is a web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, machine learning and much more." http://jupyter.org/ http://jupyter.org/

KKKKkkkk1 10y ago
Thanks for explaining what Jupyter is. This is certainly useful since it's hard to find this information from the original link.

jxy 10y ago

It's fun to play with code and try out new ideas in such a rich interactive environment. When you want to get the work done in production scenario, the shortcomings of unable to use version control and the overhead of interactive environment just kill it. What works is that you get a subset of your data and try to develop some code to process it and generate a handful of graphs. You can then save the code in its true text form and edit with your favorite editor, and run it on your real data.

analog31 10y ago

Jupyter/Python has been game changing for me. I'm not a commercial software developer, but a scientist working on technology development. I've been programming for 30+ years. Jupyter has become my lab notebook. In the past, I always had illegible, disorganized notebooks, files, and program code, all over the place. A Jupyter notebook lets me organize all of that stuff in one place, in a narrative fashion, allowing me to reconstruct what I did, long after I've forgotten the details. The reasons for open communication of methods and results to the public, also apply to internal work. My notebooks become my reports. I've abandoned PowerPoint, and my colleagues, including managers, don't seem to mind. Seeing the actual work might actually give them a feeling of involvement, like inviting them into the lab. They're also a good way of communicating a prototype of a process to the software development team, when an idea ends up in a product. Even if they don't like Python, the programmers can read and understand it. I can actually run some of my data acquisition code directly within Jupyter. A code cell that spits out an inline graph is practically the default interface for a lot of this kind of work, so I don't have to build a unique GUI for every kind of test. This speeds up incremental refinement of an experimental technique, even if the routines that I write end up in a "straight" Python program when it's time to let an experiment run for a few hours or days. Granted, Jupyter won't turn bad programmers into good. Learning good programming methods is still a gap in the education of scientists.

jasongrout 10y ago

Thanks for the feedback on the announcement blog post. I added a short description and link to the project home page at the top of the blog post after reading the feedback here. Thanks!