20 ms·

Advanced computing with IPython

326 points by leephillips 8y ago

carc1n0gen 8y ago
For the last couple of year I had wrongly thought that IPython was short for Iron Python. Now that I know it is not, I understand the hype around it more
- Waterluvian 8y ago
  Early in my career I thought CPython and Cython were the same. And then Pypy and Pypi confused me. And Python the language vs. Python the implementations and how when someone says "Python" in some contexts they assume "CPython". Landmines everywhere!
  - deleted 8y ago
    [deleted]
  - leephillips 8y ago
    Yup, whenever I write one of these I need to spend some time on terminology. It's even more confusing now with Jupyter, as there is still much overlap between that and IPython.
    - Waterluvian 8y ago
      Yep. I honestly thought Jupyter is just a rename of IPython until you said that. Now I have to go learn the difference.
      - Asooka 8y ago
        Summary: Jupyter is the evolution of IPython. IPython is deprecated.
        detaro 8y ago
        Jupyter is the language-agnostic parts of IPython (UI, notebook format, protocols to talk to notebooks, ...) extracted out to be used with many languages. IPython remains as the project maintaining the Python-specific parts of that stack. It's not deprecated, but has been limited in scope.
        Phrodo_00 8y ago
        IPython still makes nicer-repl-for-python. I don't know if it has a name beyond just IPython, which is a bit confusing.
      - wenc 8y ago
        Also Jupyter Notebook (pure notebook interface) is evolving into Jupyter Lab (a notebook-oriented IDE). The latter is much more IDE-like than the former, though the former is still really good. Technology evolves...
  - carc1n0gen 8y ago
    Well there's another two I thought meant the same thing. Thanks for the lesson!
mlthoughts2018 8y ago
If interested, I spent some time on a comment thread a few days ago describing how my experience leads me to believe the Notebook environment (not all of Jupyter / IPython, just the Notebook part) is actually only appropriate for a tiny subset of pedagogical or throw-away situations, and should be avoided most of the time and avoided in most of the cases it’s marketed for (especially anything having to do with ‘reproducibility’ or ‘exploratory analysis’). < https://news.ycombinator.com/item?id=17202704 https://news.ycombinator.com/item?id=17202704 >
- bunderbunder 8y ago
  You're not wrong, though I think the value of pedagogical and throw-away situations in your work may not be a universal experience. For me, at least, pedagogic and throw-away situations aren't a tiny subset. They're most of what I do. It's exploratory work, figuring out how the data behaves, if the data behaves, where it needs to be cleaned, churning through great heaps of experiments and iterations before hitting on the ultimate plan, and putting together a presentation to help explain what I finally settled on to colleagues and stakeholders. Only after sinking a whole lot of sweat into that process do I go on to start building anything that we intend to keep. At which point, forget Jupyter notebooks, I'm typically not even working in Python anymore for that part of the job.
  - na85 8y ago
    What are you working in at that later part of the job?
  - wenc 8y ago
    > At which point, forget Jupyter notebooks, I'm typically not even working in Python anymore for that part of the job. This is what is typically done out there but I suggest it breaks the feedback loop between the scientist roles and the developer roles. In rapidly changing environments those feedback loops could be crucial. It's similar to what Wall Street folks did (still do?)--quants write models in Excel/VBA and pass them over to developers who would rewrite them in Java for production. There's a natural impedance mismatch, and back-and-forths are difficult. I think a better approach would be for data scientists to write somewhat production-ready code, send it to prod (with the help of devs), get feedback from the production environment as well as get a sense of what tricks are needed for prod, and then iterate on that code. It also helps to remove the insulation between data scientists and the real world.
bitexploder 8y ago
Don't overlook all of the % commands, such as %edit. If you are familiar with emacs keybindings, it has a very good built in editor as well. You can also load snippets from saved files, and save your history to a file. Or individual lines to files using range type expressions. In short it is very easy to get code in and out of IPython. Another great trick. Anywhere you want to debug or play in your scripts run `import IPython` and the run `IPython.embed()` and your program at that point with all its locals drops into an IPython session, which is nice.
- IanCal 8y ago
  As of 3.7 I think, you can use breakpoint() in your code and select the debugger you want with an environment variable, or enable/disable it without changing the code. https://www.python.org/dev/peps/pep-0553/ https://www.python.org/dev/peps/pep-0553/
- taeric 8y ago
  I'm not sure I would compare it favorably to emacs in pretty much anything other than mindshare. And, don't get me wrong, that is huge and not intended as a vanity comment. The keyboard shortcuts it has are superficial. My number one shortcuts in emacs are compile, jump to next error, grep/occur, index, and magic. Just up and down? Obviously u use them a lot, but the arrow keys do work fine. Beginning of line and begging of text are both huge. But really, the screwy tab behavior kills me in Jupyter.
  - bitexploder 8y ago
    Sure. The keybindings for the line editor are emacs like, but only surface level. I can often fix up a line quickly, whereas I notice the vi folks just use %edit. Jupyter, I only use when making plots and such. The one other super handy shortcut is CTRL-R for shell like reverse search. That is pretty sweet.
- kennedyp 8y ago
  You can also leverage embed() within ipdb too and it can be nested. Very useful. Especially when combined with the ultratb or set_trace() from ipdb (or pdb): syntax is a little different dependant on whether you're in py3 or 2.7 so just google it but the jist of it is you replace the sys.excepthook with this colortb then when your script hits an uncaught exception say, you'll be presented with an ipdb colorised traceback. You can step up and down the stack frames, embedding at any point with `from IPython import embed` then `embed()` to get a fully functional ipython repl. game changer
chrxr 8y ago
At Harvard we've built out an infrastructure to allow us to deploy JupyterHub to courses with authentication managed by Canvas. It has allowed us to easily deploy complex set-ups to students so they can do really cool stuff without having to spend hours walking them through setup. Instructors are writing their lectures as IPython notebooks, and distributing them to students, who then work through them in their JupyterHub environment. Our most ambitious so far has been setting up each student in the course with a p2.xlarge machine with cuda and TensorFlow so they could do deep learning work for their final projects. We supported 15 courses last year, and got deployment time for an implementation down to only 2-3 hours. In conclusion, IPython good, JupyterHub good. Edit: surfacing the link to the open source repo on GitHub https://github.com/harvard/cloudJHub https://github.com/harvard/cloudJHub
- chrxr 8y ago
  It is worth noting that there is an argument that it is a worthwhile task for students to learn how to setup complex computing environments, as it better prepares them for the real world. However, in reality, there just isn't time within a single semester to do this for a class of 100+ students. So implementations such as this one trade-off that learning for a greater focus on computational theory and its implementations.
  - cirgue 8y ago
    It’s a good experience, but a motivated student who’s good at charging through docs can do it on their own. Having to manage that for a class that’s intended to teach conceptual material would be a big time-sink.
  - w0m 8y ago
    Agreed. At the beginning of class, walk students through the setup. Then for every project after, let them use the pre-rolled systems.
    - killaken2000 8y ago
      I think at the very least they should be able to see someone do it so they know what steps are involved. Possibly a video with steps but stated that it isn't the focus of the class.
giu 8y ago
The linked slides "Python in HPC" [0] are quite awesome, from which I learned about mpi4py's [1] existance. Definitely going to give it a try in the future; using the MPI API with an "easy-to-follow" syntax should be fun. [0] https://hpc.nih.gov/training/handouts/171121_python_in_hpc.pdf https://hpc.nih.gov/training/handouts/171121_python_in_hpc.p... [1] http://mpi4py.scipy.org/docs/ http://mpi4py.scipy.org/docs/
- skierscott 8y ago
  mpi4py is a solid implementation, but the docs aren’t the greatest, especially if you’re not familiar with MPI.
  - ianhowson 8y ago
    And there are different versions with different APIs. Best to just read the source code for whatever you're running.
biridir 8y ago
The content in the article is a bit dated. Jupyter + Numba + Dask is the direction scientific computing (in Python) is taking. Ipyparallel is not really scalable in my experience.
- commandlinefan 8y ago
  I thought so, too - enough that I checked the date, but the date on the article is yesterday. Maybe it was written a while ago and just now uploaded. I thought it was strange to refer to IPython as a standalone interpreter outside of Jupyter... I mean, you can, but I don't think anybody does anymore.
  - xapata 8y ago
    > I don't think anybody does anymore Counter-example: I use IPython in the terminal, outside of Jupyter.
    - leephillips 8y ago
      So do I, and many others. Nowadays you can type "jupyter console", which gets you almost the same thing; and JupyterLab has console components.
    - newen 8y ago
      Me too! I actually hook up an Ipython terminal to a Jupyter notebook and in order to do REPL type work in the terminal. Much better to work in a REPL terminal and also save my work in the notebook.
- semi-extrinsic 8y ago
  This is an extremely limited view of "scientific computing" that seems to only focus on analytics, which is a tiiiny part of sci comp. Your "stack" does nothing for solving/including ODEs, PDEs, DAEs, Fourier analysis, numerical integration, automatic differentiation, linear equation system solvers, preconditioners, nonlinear equation system solvers, the entire field of optimization, inverse problems, statistical methods, Monte Carlo simulations, molecular dynamics, PIC methods, geometric integration, lattice quantum field theory, molecular dynamics, ab initio methods, density functional theory, finite difference/volume/element methods, lattice Boltzmann methods, boundary integral methods, mesh generation methods, error estimation, uncertainty quantification... Those are just off the top of my head, the list goes on and on.
AdmiralAsshat 8y ago
Deep research uses aside, I often prefer to use IPython because it's simply a better shell than the default Python shell. You get basic niceties like tab completion and being able to up-arrow to revise an earlier multi-line command (like a function) without it being an exercise in frustration.
- deleted 8y ago
  [deleted]
- Bromskloss 8y ago
  Indeed. If I could only figure out how to have it automatically run `from math import *` (and then present me with the interactive shell, I could use it as a calculator too.
  - spot 8y ago
    http://ipython.readthedocs.io/en/stable/config/options/terminal.html#configtrait-InteractiveShellApp.exec_lines http://ipython.readthedocs.io/en/stable/config/options/termi...
    - Bromskloss 8y ago
      My problem is that I don't know how to make it accept spaces in the command to be run. This works: ipython3 --InteractiveShellApp.exec_lines='["print(2)"]' This does not work: ipython3 --InteractiveShellApp.exec_lines='["from math import *"]' The latter command results in the following error: [TerminalIPythonApp] CRITICAL | The 'exec_lines' trait of a TerminalIPythonApp instance must be a list, but a value of class 'str' (i.e. '["from') was specified.
      - takluyver 8y ago
        Rather than fiddling around with quoting bash, you can put the import line in a startup file: http://ipython.readthedocs.io/en/stable/interactive/tutorial.html#startup-files http://ipython.readthedocs.io/en/stable/interactive/tutorial...
pankajkumar229 8y ago
Disclaimer: I am one of the founders of DataCabinet. We built an online service on top of Jupyter which takes away the effort of handling JupyterHub. It would be great if we could hear some feedback in the context of this conversation. We feel that DataCabinet is better in ways because it provides: a. Autoscaling according to number of users b. Sharing full containers easily between people. You can install pip/conda binaries and share with students/users. c. Shared storage so nbgrader works seamlessly. Here is a full comparison: https://datacabinet.info/pricing.html https://datacabinet.info/pricing.html Please excuse our landing page, it just got created today and we are fixing it.
happy-go-lucky 8y ago
It's a pdf file, which I am trying to open but cannot. Can you guys read it? Edit: The link to the file: https://hpc.nih.gov/training/handouts/171121_python_in_hpc.pdf https://hpc.nih.gov/training/handouts/171121_python_in_hpc.p... Edit: I can read it now.
vaibhavsagar 8y ago
There are solutions available for hosted Jupyter notebooks, such as https://cocalc.com/ https://cocalc.com/ (which is collaborative!) and https://mybinder.org/ https://mybinder.org/.

326 points by leephillips 8y ago

For the last couple of year I had wrongly thought that IPython was short for Iron Python. Now that I know it is not, I understand the hype around it more

Waterluvian 8y ago
Early in my career I thought CPython and Cython were the same. And then Pypy and Pypi confused me. And Python the language vs. Python the implementations and how when someone says "Python" in some contexts they assume "CPython". Landmines everywhere!
- deleted 8y ago
  [deleted]
- leephillips 8y ago
  Yup, whenever I write one of these I need to spend some time on terminology. It's even more confusing now with Jupyter, as there is still much overlap between that and IPython.
  - Waterluvian 8y ago
    Yep. I honestly thought Jupyter is just a rename of IPython until you said that. Now I have to go learn the difference.
    - Asooka 8y ago
      Summary: Jupyter is the evolution of IPython. IPython is deprecated.
      - detaro 8y ago
        Jupyter is the language-agnostic parts of IPython (UI, notebook format, protocols to talk to notebooks, ...) extracted out to be used with many languages. IPython remains as the project maintaining the Python-specific parts of that stack. It's not deprecated, but has been limited in scope.
        Phrodo_00 8y ago
        IPython still makes nicer-repl-for-python. I don't know if it has a name beyond just IPython, which is a bit confusing.
    - wenc 8y ago
      Also Jupyter Notebook (pure notebook interface) is evolving into Jupyter Lab (a notebook-oriented IDE). The latter is much more IDE-like than the former, though the former is still really good. Technology evolves...
- carc1n0gen 8y ago
  Well there's another two I thought meant the same thing. Thanks for the lesson!

mlthoughts2018 8y ago

If interested, I spent some time on a comment thread a few days ago describing how my experience leads me to believe the Notebook environment (not all of Jupyter / IPython, just the Notebook part) is actually only appropriate for a tiny subset of pedagogical or throw-away situations, and should be avoided most of the time and avoided in most of the cases it’s marketed for (especially anything having to do with ‘reproducibility’ or ‘exploratory analysis’). < https://news.ycombinator.com/item?id=17202704 https://news.ycombinator.com/item?id=17202704 >

bunderbunder 8y ago
You're not wrong, though I think the value of pedagogical and throw-away situations in your work may not be a universal experience. For me, at least, pedagogic and throw-away situations aren't a tiny subset. They're most of what I do. It's exploratory work, figuring out how the data behaves, if the data behaves, where it needs to be cleaned, churning through great heaps of experiments and iterations before hitting on the ultimate plan, and putting together a presentation to help explain what I finally settled on to colleagues and stakeholders. Only after sinking a whole lot of sweat into that process do I go on to start building anything that we intend to keep. At which point, forget Jupyter notebooks, I'm typically not even working in Python anymore for that part of the job.
- na85 8y ago
  What are you working in at that later part of the job?
- wenc 8y ago
  > At which point, forget Jupyter notebooks, I'm typically not even working in Python anymore for that part of the job. This is what is typically done out there but I suggest it breaks the feedback loop between the scientist roles and the developer roles. In rapidly changing environments those feedback loops could be crucial. It's similar to what Wall Street folks did (still do?)--quants write models in Excel/VBA and pass them over to developers who would rewrite them in Java for production. There's a natural impedance mismatch, and back-and-forths are difficult. I think a better approach would be for data scientists to write somewhat production-ready code, send it to prod (with the help of devs), get feedback from the production environment as well as get a sense of what tricks are needed for prod, and then iterate on that code. It also helps to remove the insulation between data scientists and the real world.

bitexploder 8y ago

Don't overlook all of the % commands, such as %edit. If you are familiar with emacs keybindings, it has a very good built in editor as well. You can also load snippets from saved files, and save your history to a file. Or individual lines to files using range type expressions. In short it is very easy to get code in and out of IPython. Another great trick. Anywhere you want to debug or play in your scripts run `import IPython` and the run `IPython.embed()` and your program at that point with all its locals drops into an IPython session, which is nice.

IanCal 8y ago
As of 3.7 I think, you can use breakpoint() in your code and select the debugger you want with an environment variable, or enable/disable it without changing the code. https://www.python.org/dev/peps/pep-0553/ https://www.python.org/dev/peps/pep-0553/
taeric 8y ago
I'm not sure I would compare it favorably to emacs in pretty much anything other than mindshare. And, don't get me wrong, that is huge and not intended as a vanity comment. The keyboard shortcuts it has are superficial. My number one shortcuts in emacs are compile, jump to next error, grep/occur, index, and magic. Just up and down? Obviously u use them a lot, but the arrow keys do work fine. Beginning of line and begging of text are both huge. But really, the screwy tab behavior kills me in Jupyter.
- bitexploder 8y ago
  Sure. The keybindings for the line editor are emacs like, but only surface level. I can often fix up a line quickly, whereas I notice the vi folks just use %edit. Jupyter, I only use when making plots and such. The one other super handy shortcut is CTRL-R for shell like reverse search. That is pretty sweet.
kennedyp 8y ago
You can also leverage embed() within ipdb too and it can be nested. Very useful. Especially when combined with the ultratb or set_trace() from ipdb (or pdb): syntax is a little different dependant on whether you're in py3 or 2.7 so just google it but the jist of it is you replace the sys.excepthook with this colortb then when your script hits an uncaught exception say, you'll be presented with an ipdb colorised traceback. You can step up and down the stack frames, embedding at any point with `from IPython import embed` then `embed()` to get a fully functional ipython repl. game changer

chrxr 8y ago

At Harvard we've built out an infrastructure to allow us to deploy JupyterHub to courses with authentication managed by Canvas. It has allowed us to easily deploy complex set-ups to students so they can do really cool stuff without having to spend hours walking them through setup. Instructors are writing their lectures as IPython notebooks, and distributing them to students, who then work through them in their JupyterHub environment. Our most ambitious so far has been setting up each student in the course with a p2.xlarge machine with cuda and TensorFlow so they could do deep learning work for their final projects. We supported 15 courses last year, and got deployment time for an implementation down to only 2-3 hours. In conclusion, IPython good, JupyterHub good. Edit: surfacing the link to the open source repo on GitHub https://github.com/harvard/cloudJHub https://github.com/harvard/cloudJHub

chrxr 8y ago
It is worth noting that there is an argument that it is a worthwhile task for students to learn how to setup complex computing environments, as it better prepares them for the real world. However, in reality, there just isn't time within a single semester to do this for a class of 100+ students. So implementations such as this one trade-off that learning for a greater focus on computational theory and its implementations.
- cirgue 8y ago
  It’s a good experience, but a motivated student who’s good at charging through docs can do it on their own. Having to manage that for a class that’s intended to teach conceptual material would be a big time-sink.
- w0m 8y ago
  Agreed. At the beginning of class, walk students through the setup. Then for every project after, let them use the pre-rolled systems.
  - killaken2000 8y ago
    I think at the very least they should be able to see someone do it so they know what steps are involved. Possibly a video with steps but stated that it isn't the focus of the class.

giu 8y ago

The linked slides "Python in HPC" [0] are quite awesome, from which I learned about mpi4py's [1] existance. Definitely going to give it a try in the future; using the MPI API with an "easy-to-follow" syntax should be fun. [0] https://hpc.nih.gov/training/handouts/171121_python_in_hpc.pdf https://hpc.nih.gov/training/handouts/171121_python_in_hpc.p... [1] http://mpi4py.scipy.org/docs/ http://mpi4py.scipy.org/docs/

skierscott 8y ago
mpi4py is a solid implementation, but the docs aren’t the greatest, especially if you’re not familiar with MPI.
- ianhowson 8y ago
  And there are different versions with different APIs. Best to just read the source code for whatever you're running.

biridir 8y ago

The content in the article is a bit dated. Jupyter + Numba + Dask is the direction scientific computing (in Python) is taking. Ipyparallel is not really scalable in my experience.

commandlinefan 8y ago
I thought so, too - enough that I checked the date, but the date on the article is yesterday. Maybe it was written a while ago and just now uploaded. I thought it was strange to refer to IPython as a standalone interpreter outside of Jupyter... I mean, you can, but I don't think anybody does anymore.
- xapata 8y ago
  > I don't think anybody does anymore Counter-example: I use IPython in the terminal, outside of Jupyter.
  - leephillips 8y ago
    So do I, and many others. Nowadays you can type "jupyter console", which gets you almost the same thing; and JupyterLab has console components.
  - newen 8y ago
    Me too! I actually hook up an Ipython terminal to a Jupyter notebook and in order to do REPL type work in the terminal. Much better to work in a REPL terminal and also save my work in the notebook.
semi-extrinsic 8y ago
This is an extremely limited view of "scientific computing" that seems to only focus on analytics, which is a tiiiny part of sci comp. Your "stack" does nothing for solving/including ODEs, PDEs, DAEs, Fourier analysis, numerical integration, automatic differentiation, linear equation system solvers, preconditioners, nonlinear equation system solvers, the entire field of optimization, inverse problems, statistical methods, Monte Carlo simulations, molecular dynamics, PIC methods, geometric integration, lattice quantum field theory, molecular dynamics, ab initio methods, density functional theory, finite difference/volume/element methods, lattice Boltzmann methods, boundary integral methods, mesh generation methods, error estimation, uncertainty quantification... Those are just off the top of my head, the list goes on and on.

AdmiralAsshat 8y ago

Deep research uses aside, I often prefer to use IPython because it's simply a better shell than the default Python shell. You get basic niceties like tab completion and being able to up-arrow to revise an earlier multi-line command (like a function) without it being an exercise in frustration.

deleted 8y ago
[deleted]
Bromskloss 8y ago
Indeed. If I could only figure out how to have it automatically run `from math import *` (and then present me with the interactive shell, I could use it as a calculator too.
- spot 8y ago
  http://ipython.readthedocs.io/en/stable/config/options/terminal.html#configtrait-InteractiveShellApp.exec_lines http://ipython.readthedocs.io/en/stable/config/options/termi...
  - Bromskloss 8y ago
    My problem is that I don't know how to make it accept spaces in the command to be run. This works: ipython3 --InteractiveShellApp.exec_lines='["print(2)"]' This does not work: ipython3 --InteractiveShellApp.exec_lines='["from math import *"]' The latter command results in the following error: [TerminalIPythonApp] CRITICAL | The 'exec_lines' trait of a TerminalIPythonApp instance must be a list, but a value of class 'str' (i.e. '["from') was specified.
    - takluyver 8y ago
      Rather than fiddling around with quoting bash, you can put the import line in a startup file: http://ipython.readthedocs.io/en/stable/interactive/tutorial.html#startup-files http://ipython.readthedocs.io/en/stable/interactive/tutorial...

pankajkumar229 8y ago

Disclaimer: I am one of the founders of DataCabinet. We built an online service on top of Jupyter which takes away the effort of handling JupyterHub. It would be great if we could hear some feedback in the context of this conversation. We feel that DataCabinet is better in ways because it provides: a. Autoscaling according to number of users b. Sharing full containers easily between people. You can install pip/conda binaries and share with students/users. c. Shared storage so nbgrader works seamlessly. Here is a full comparison: https://datacabinet.info/pricing.html https://datacabinet.info/pricing.html Please excuse our landing page, it just got created today and we are fixing it.

happy-go-lucky 8y ago

It's a pdf file, which I am trying to open but cannot. Can you guys read it? Edit: The link to the file: https://hpc.nih.gov/training/handouts/171121_python_in_hpc.pdf https://hpc.nih.gov/training/handouts/171121_python_in_hpc.p... Edit: I can read it now.

vaibhavsagar 8y ago

There are solutions available for hosted Jupyter notebooks, such as https://cocalc.com/ https://cocalc.com/ (which is collaborative!) and https://mybinder.org/ https://mybinder.org/.