18 ms·

Project Jupyter Celebrates 20 Years, Fernando Pérez Reflects on How It Started

118 points by infodocket 5y ago

Causality1 5y ago
the Value of Diversity in Coding Strange title on the article itself. There is nothing on this topic in the article or the interview.
- leephillips 5y ago
  “if you only rely on volunteers or on people who perhaps their job allows them to do the one thing they like, you will exclude parts of the population that don't have those affordances. By building tools without a complete slice of the society you want to reach, you can't possibly have the impact that you want in that society. It's not just a technical mission. It’s an ethical mission of building things that have a positive impact in the world. That impact isn't going to be realized if we're building things only by a few, because when you build by a few, you build for a few. If we want to build things that really are for everyone, we need to build them with everyone.”
DataCrayon 5y ago
I'm very grateful for Jupyter Lab and Jupyter Notebooks. I work with them more than anything else, and even use them with Rust [1], my data visualisations [2], and to write my books [3]. [1] https://datacrayon.com/posts/programming/rust-notebooks/setup-anaconda-jupyter-and-rust/ https://datacrayon.com/posts/programming/rust-notebooks/setu... [2] https://plotapi.com/gallery/posts/showcase/pokemon-types-with-plotapi-chord/ https://plotapi.com/gallery/posts/showcase/pokemon-types-wit... [3] https://datacrayon.com/shop/ https://datacrayon.com/shop/
dfee 5y ago
> Twenty years ago, UC Berkeley Associate Statistics Professor Fernando Pérez(link is external) started one of the foundational tools for analyzing large amounts of data in a transparent and collaborative way. That project, IPython, evolved into Project Jupyter(link is external). I just used iPython as a REPL because it was so much nicer than regular ole python (the unwrapped variant). Ipdb was nice too.
gammarator 5y ago
Glad to see Berkeley finally gave him a faculty position.
mark_l_watson 5y ago
He also co-wrote a fantastic NLP demo in Prolog for geographic queries. I have played with this demo and read the code many times. Cool stuff.
diskzero 5y ago
Jupyter Notebooks have been been so amazingly valuable to me. Would anyone consider them to be an example of literate programming? If so, are they the most successful example out there?
- leephillips 5y ago
  In my article about Pluto¹, which is an evolution of the Jupyter concept specific to Julia, I said that these notebooks “may be a way to realize Donald Knuth's concept of literate programming.”. [1] https://lwn.net/Articles/835930/ https://lwn.net/Articles/835930/
  - diskzero 5y ago
    OK Lee, Pluto is one of the cooler things I have seen recently. I am going to dive in more deeply. Gilad Bracha has also been going in this direction with his recent work on Newspeak and literate programming. Is this how Smalltalk is finally going to gain adoption?
  - O_H_E 5y ago
    Oh wow, I just realized that Pluto's ability to track dependencies—and thus cells could be written "out of order"—would also fulfill Knuth's idea of code being presented in an order more ideal for humans to read, that is different from execution order.
- mumblemumble 5y ago
  They're a possible vehicle for literate programming, but I don't know that I'd call them the best vehicle. Using notebooks to build a stand-alone application would be awkward at best. The current space to watch for literate programming seems to be Racket.
  - diskzero 5y ago
    Good point. I have taught classes using DrRacket, so I suppose what might be needed is a web-based interface as well. I see similar limitations with Swift Playgrounds. They are great as long as you have a Mac, installed XCode, etc. I don't want to say it was a struggle to get students to install DrRacket, but a web interface like Observable, maybe Pluto, etc. is hard to compete against
hatmatrix 5y ago
Jupyter has been revolutionary - Steve Yegge predicted that emacs's greatest competition will be the web browser and this has been a partial fulfillment of that. I'm personally a big fan of org-babel due its text-based nature but the accessibility of Jupyter has made it the great equalizer.
- sunsetSamurai 5y ago
  lol emacs? vscode probably have a million time more user's than emacs, hell probably vim is more pupular than emacs
  - xxpor 5y ago
    I'm 99% sure that's not what he means. He's talking about the power of emacs.
  - fooofw 5y ago
    VS Code has a lot in common with webbrowsers. VS Code is (as e.g. the older Atom) mostly implemented using web technologies HTML, CSS and JavaScript that execute in a runtime forked from a browser (Chromium). I assume the grand parent is, at least partially, referring to this essay [1]. Here, the browser and HTML/CSS/JS is complimented for its dynamic nature and low barrier to entry, similarly to Emacs. In that sense, VS Code, like Jupyter, is a perfect example of the webbrowser gaining functionality of Emacs. [1] https://steve-yegge.blogspot.com/2008/04/xemacs-is-dead-long-live-xemacs.html https://steve-yegge.blogspot.com/2008/04/xemacs-is-dead-long... (the section: "The bad news: the competition isn't the IDEs")
simonw 5y ago
I've been using the iPython CLI interface for well over a decade, but I'm embarrassed at how long it took me to understand how transformational "iPython notebooks" (now Jupyter) would be. I remember seeing new releases of iPython that enhanced the weird web interface feature and being utterly baffled as to why anyone would want that. I finally got on board with Jupyter notebooks a few years ago and I've used them multiple times a week ever since.
- kristianp 5y ago
  I've been using ipython as a repl in the terminal, I've been struck how good it is for editing multi-line functions or loops.
jwilber 5y ago
I did my undergrad at Cal and worked in a research lab Fernando worked in as well. I remember some of the postdocs were very nice and approachable. Others were very pretentious and cold. One thing the other undergrads and myself would always talk about was how nice Fernando was. He knew everyone’s name, would take the time to give advice, casually chat, etc. - overall a great person. Off topic: A big takeaway from working there was that those who accomplished the most were often the nicest. We’d have Nobel laureates come to and from the lab fairly often. They always seemed very grounded and passionate. The rudest people were overwhelmingly the ‘important’ bureaucrats and admin, with the occasional postdoc trying to make a name for themself. tl;dr Fernando Pérez is an awesome person all around.
jjtheblunt 5y ago
How does that entire article not mention Theo Gray's 1988 Mathematica notebooks as the motivation? Citation: https://en.wikipedia.org/wiki/Wolfram_Mathematica https://en.wikipedia.org/wiki/Wolfram_Mathematica
- maliker 5y ago
  It was clearly the original invention. I think what Fernando added was an open source implementation of the same ideas. And now I see Jupyter notebooks everywhere and never see a Mathematica notebook.
  - taeric 5y ago
    Yeah... This is tough. I think Mathematica and Matlab are both far more powerful than most in our industry realize. But competing with open source is getting harder every day.
- spoonjim 5y ago
  You get the money or the legacy. Two men, two different choices. The world needs both types.
abxm 5y ago
I've never understood this project. It's like Powerpoint: You can produce flashy presentations that are largely meaningless and don't advance science. I guess that is why it's popular.
- BiteCode_dev 5y ago
  jupyter does allow to produce presentations, and I used it with great success at conferences where a mix of slideshow and interactive programming suits the format perfectly. But honestly it's only a tiny small part of what it's for. jupyter is mostly about explorating ideas, code and data. It gives you a comfortable env to quickly try something out. Sharing it with other is just the cherry on the cake, not the main goal. I mostly never share my notebooks, but I will fire one as soon as I want to try a new library, or need to extract data from a file.
eyeball 5y ago
What’s the best way to learn to use Jupyter? Should one start with notebooks or lab?
- analog31 5y ago
  I'm an avid Jupyter user. In my view it doesn't matter. The basic notebook concept is the same in both cases. Once you get more experienced with Python and notebooks, then you can try both and form an opinion of which one you prefer. I still prefer notebooks because I get distracted by extra "stuff" on the screen. I haven't found a case where I have to commit to one or the other. My notebooks that work in one environment, tend to work in the other.
- teruakohatu 5y ago
  I think there is no good reason not to use Lab.
maliker 5y ago
Associate professor… ipython 20 years ago… jeeez what do you have to do to make full professor? Clearly creating the interface to 90% of data science doesn’t cut it.
- maliker 5y ago
  Also noticed his first PHD advisor kicked him out and only his second advisor let him work on ipython. What a horrible system.
  - humanistbot 5y ago
    Well, if you're in a particle physics Ph.D program and halfway through you decide that instead of doing particle physics you want to be a software engineer and build a clone of Mathematica Notebooks.... obviously the system is not going to support you all the way. I'm a big fan of Jupyter and am thankful to Perez for all the work he did for all scientific fields. But you shouldn't expect to get a particle physics Ph.D for it.
BiteCode_dev 5y ago
Such a wonderful tool, but hardly accessible to beginners. First, you have to install it, which if you don't use anaconda is going to be full of the usual gotchas. Then you need to launch it, which because "-m" is broken on jupyter for windows, means people will have an extra hard time. And then, once everything works, new commers still have to understand they need to create a notebook before entering any python code, find it back in their weird FS browser implementation and get how to execute and create cells. Not a problems for any dev, but in class, it can quickly become a burden. Thanks to pyiodine, you can now use jupyter directly in the browser with no install at all: https://notebook.basthon.fr/ https://notebook.basthon.fr/ Even using sqlite works :) This is going to open so many doors for education. I still think there is a room for a python scratchpad tool that would: - be provided with a standard installer (exe, deb, .app) as well as with pip - start with a fully operational shell (no need to create a file) but you can save it later - show big buttons for common notebook operations near each cell such as "run", "add", "run and add", "delete", "duplicate" and "move" - gui to deal install packages (and deal with venv when you decide to) - use the currently activated virtualenv automatically
- deleted 5y ago
  [deleted]
- nerdponx 5y ago
  In a teaching environment, maybe it makes sense to use JupyterHub to give every student their own notebook in the browser. You can teach them how to set the thing up later, once they understand what the heck a kernel is and how the system actually works.
  - BiteCode_dev 5y ago
    An online service causes other problems. Baston can run locally.

118 points by infodocket 5y ago

the Value of Diversity in Coding Strange title on the article itself. There is nothing on this topic in the article or the interview.

leephillips 5y ago
“if you only rely on volunteers or on people who perhaps their job allows them to do the one thing they like, you will exclude parts of the population that don't have those affordances. By building tools without a complete slice of the society you want to reach, you can't possibly have the impact that you want in that society. It's not just a technical mission. It’s an ethical mission of building things that have a positive impact in the world. That impact isn't going to be realized if we're building things only by a few, because when you build by a few, you build for a few. If we want to build things that really are for everyone, we need to build them with everyone.”

DataCrayon 5y ago

I'm very grateful for Jupyter Lab and Jupyter Notebooks. I work with them more than anything else, and even use them with Rust [1], my data visualisations [2], and to write my books [3]. [1] https://datacrayon.com/posts/programming/rust-notebooks/setup-anaconda-jupyter-and-rust/ https://datacrayon.com/posts/programming/rust-notebooks/setu... [2] https://plotapi.com/gallery/posts/showcase/pokemon-types-with-plotapi-chord/ https://plotapi.com/gallery/posts/showcase/pokemon-types-wit... [3] https://datacrayon.com/shop/ https://datacrayon.com/shop/

dfee 5y ago

> Twenty years ago, UC Berkeley Associate Statistics Professor Fernando Pérez(link is external) started one of the foundational tools for analyzing large amounts of data in a transparent and collaborative way. That project, IPython, evolved into Project Jupyter(link is external). I just used iPython as a REPL because it was so much nicer than regular ole python (the unwrapped variant). Ipdb was nice too.

gammarator 5y ago

Glad to see Berkeley finally gave him a faculty position.

mark_l_watson 5y ago

He also co-wrote a fantastic NLP demo in Prolog for geographic queries. I have played with this demo and read the code many times. Cool stuff.

diskzero 5y ago

Jupyter Notebooks have been been so amazingly valuable to me. Would anyone consider them to be an example of literate programming? If so, are they the most successful example out there?

leephillips 5y ago
In my article about Pluto¹, which is an evolution of the Jupyter concept specific to Julia, I said that these notebooks “may be a way to realize Donald Knuth's concept of literate programming.”. [1] https://lwn.net/Articles/835930/ https://lwn.net/Articles/835930/
- diskzero 5y ago
  OK Lee, Pluto is one of the cooler things I have seen recently. I am going to dive in more deeply. Gilad Bracha has also been going in this direction with his recent work on Newspeak and literate programming. Is this how Smalltalk is finally going to gain adoption?
- O_H_E 5y ago
  Oh wow, I just realized that Pluto's ability to track dependencies—and thus cells could be written "out of order"—would also fulfill Knuth's idea of code being presented in an order more ideal for humans to read, that is different from execution order.
mumblemumble 5y ago
They're a possible vehicle for literate programming, but I don't know that I'd call them the best vehicle. Using notebooks to build a stand-alone application would be awkward at best. The current space to watch for literate programming seems to be Racket.
- diskzero 5y ago
  Good point. I have taught classes using DrRacket, so I suppose what might be needed is a web-based interface as well. I see similar limitations with Swift Playgrounds. They are great as long as you have a Mac, installed XCode, etc. I don't want to say it was a struggle to get students to install DrRacket, but a web interface like Observable, maybe Pluto, etc. is hard to compete against

hatmatrix 5y ago

Jupyter has been revolutionary - Steve Yegge predicted that emacs's greatest competition will be the web browser and this has been a partial fulfillment of that. I'm personally a big fan of org-babel due its text-based nature but the accessibility of Jupyter has made it the great equalizer.

sunsetSamurai 5y ago
lol emacs? vscode probably have a million time more user's than emacs, hell probably vim is more pupular than emacs
- xxpor 5y ago
  I'm 99% sure that's not what he means. He's talking about the power of emacs.
- fooofw 5y ago
  VS Code has a lot in common with webbrowsers. VS Code is (as e.g. the older Atom) mostly implemented using web technologies HTML, CSS and JavaScript that execute in a runtime forked from a browser (Chromium). I assume the grand parent is, at least partially, referring to this essay [1]. Here, the browser and HTML/CSS/JS is complimented for its dynamic nature and low barrier to entry, similarly to Emacs. In that sense, VS Code, like Jupyter, is a perfect example of the webbrowser gaining functionality of Emacs. [1] https://steve-yegge.blogspot.com/2008/04/xemacs-is-dead-long-live-xemacs.html https://steve-yegge.blogspot.com/2008/04/xemacs-is-dead-long... (the section: "The bad news: the competition isn't the IDEs")

simonw 5y ago

I've been using the iPython CLI interface for well over a decade, but I'm embarrassed at how long it took me to understand how transformational "iPython notebooks" (now Jupyter) would be. I remember seeing new releases of iPython that enhanced the weird web interface feature and being utterly baffled as to why anyone would want that. I finally got on board with Jupyter notebooks a few years ago and I've used them multiple times a week ever since.

kristianp 5y ago
I've been using ipython as a repl in the terminal, I've been struck how good it is for editing multi-line functions or loops.

jwilber 5y ago

I did my undergrad at Cal and worked in a research lab Fernando worked in as well. I remember some of the postdocs were very nice and approachable. Others were very pretentious and cold. One thing the other undergrads and myself would always talk about was how nice Fernando was. He knew everyone’s name, would take the time to give advice, casually chat, etc. - overall a great person. Off topic: A big takeaway from working there was that those who accomplished the most were often the nicest. We’d have Nobel laureates come to and from the lab fairly often. They always seemed very grounded and passionate. The rudest people were overwhelmingly the ‘important’ bureaucrats and admin, with the occasional postdoc trying to make a name for themself. tl;dr Fernando Pérez is an awesome person all around.

jjtheblunt 5y ago

How does that entire article not mention Theo Gray's 1988 Mathematica notebooks as the motivation? Citation: https://en.wikipedia.org/wiki/Wolfram_Mathematica https://en.wikipedia.org/wiki/Wolfram_Mathematica

maliker 5y ago
It was clearly the original invention. I think what Fernando added was an open source implementation of the same ideas. And now I see Jupyter notebooks everywhere and never see a Mathematica notebook.
- taeric 5y ago
  Yeah... This is tough. I think Mathematica and Matlab are both far more powerful than most in our industry realize. But competing with open source is getting harder every day.
spoonjim 5y ago
You get the money or the legacy. Two men, two different choices. The world needs both types.

abxm 5y ago

I've never understood this project. It's like Powerpoint: You can produce flashy presentations that are largely meaningless and don't advance science. I guess that is why it's popular.

BiteCode_dev 5y ago
jupyter does allow to produce presentations, and I used it with great success at conferences where a mix of slideshow and interactive programming suits the format perfectly. But honestly it's only a tiny small part of what it's for. jupyter is mostly about explorating ideas, code and data. It gives you a comfortable env to quickly try something out. Sharing it with other is just the cherry on the cake, not the main goal. I mostly never share my notebooks, but I will fire one as soon as I want to try a new library, or need to extract data from a file.

eyeball 5y ago

What’s the best way to learn to use Jupyter? Should one start with notebooks or lab?

analog31 5y ago
I'm an avid Jupyter user. In my view it doesn't matter. The basic notebook concept is the same in both cases. Once you get more experienced with Python and notebooks, then you can try both and form an opinion of which one you prefer. I still prefer notebooks because I get distracted by extra "stuff" on the screen. I haven't found a case where I have to commit to one or the other. My notebooks that work in one environment, tend to work in the other.
teruakohatu 5y ago
I think there is no good reason not to use Lab.

maliker 5y ago

Associate professor… ipython 20 years ago… jeeez what do you have to do to make full professor? Clearly creating the interface to 90% of data science doesn’t cut it.

maliker 5y ago
Also noticed his first PHD advisor kicked him out and only his second advisor let him work on ipython. What a horrible system.
- humanistbot 5y ago
  Well, if you're in a particle physics Ph.D program and halfway through you decide that instead of doing particle physics you want to be a software engineer and build a clone of Mathematica Notebooks.... obviously the system is not going to support you all the way. I'm a big fan of Jupyter and am thankful to Perez for all the work he did for all scientific fields. But you shouldn't expect to get a particle physics Ph.D for it.

BiteCode_dev 5y ago

Such a wonderful tool, but hardly accessible to beginners. First, you have to install it, which if you don't use anaconda is going to be full of the usual gotchas. Then you need to launch it, which because "-m" is broken on jupyter for windows, means people will have an extra hard time. And then, once everything works, new commers still have to understand they need to create a notebook before entering any python code, find it back in their weird FS browser implementation and get how to execute and create cells. Not a problems for any dev, but in class, it can quickly become a burden. Thanks to pyiodine, you can now use jupyter directly in the browser with no install at all: https://notebook.basthon.fr/ https://notebook.basthon.fr/ Even using sqlite works :) This is going to open so many doors for education. I still think there is a room for a python scratchpad tool that would: - be provided with a standard installer (exe, deb, .app) as well as with pip - start with a fully operational shell (no need to create a file) but you can save it later - show big buttons for common notebook operations near each cell such as "run", "add", "run and add", "delete", "duplicate" and "move" - gui to deal install packages (and deal with venv when you decide to) - use the currently activated virtualenv automatically

deleted 5y ago
[deleted]
nerdponx 5y ago
In a teaching environment, maybe it makes sense to use JupyterHub to give every student their own notebook in the browser. You can teach them how to set the thing up later, once they understand what the heck a kernel is and how the system actually works.
- BiteCode_dev 5y ago
  An online service causes other problems. Baston can run locally.