5 ms·

Ahh, but PyPy is not being written by web developers -- or for them. The main guy behind the project is a research computer scientist (http://cfbolz.de/ http://

Ahh, but PyPy is not being written by web developers -- or for them. The main guy behind the project is a research computer scientist (http://cfbolz.de/ http://cfbolz.de/) and most of the people who are excited about it are also doing scientific computing. I am one of them, and the ability to write scientific code in Python that runs as fast (or faster!) than C would be so awesome that I can hardly put it into words. Unlike web development, performance is still a huge bottleneck in this field; people still mill about for 3 weeks waiting for their job to finish just like it was the 1970s. Consequently, most code written for research projects consists of lot of arcane, difficult-to-read C, C++, or even (gasp!) FORTRAN. The benefits of switching to Python in terms of development time, ease of debugging, reusability, potential for collaboration, reproducibility, etc. are immense, but the performance penalty is such that it rarely happens. Even in spite of that, Python already has a large academic following; if PyPy manages to pull off its stated goals, it would be revolutionary.

joe_the_user 16y ago
A lot of scientific code also runs on the (crufty, spiffy) Matlab, which is fast but not super-fast, is less portable than most things but has good usability and lots of history. A python that could run incrementally, with robust threads, in the ground might, just possibly, make SAGE a better environment and so increase migration from MATLAB (my personal scientific computing experience was with Matlab and anything that escape this monster is great in my book). I mean, Python won't become faster than optimized C so however convincing or unconvincing the faster iteration and usability argument is, that, in the end is going have to be what Python makes.
justincormack 16y ago
Right now it looks like LuaJIT is your best option for this type of performance. PyPy might get there eventually though...
- ominous_prime 16y ago
  LuaJIT is awesome, but python already has a large volume of scientific libraries. I haven't heard mention of scientific researchers using Lua (yet).
cdavid 16y ago
The big issue here is that most scientific code in python runs on top of numpy, and getting numpy to work on top of pypy would be challenging. I understand that relying on a C extension compatibility layer would make things slower at first, but the alternative of re-implementing numpy from scratch in python does not seem realistic to me. Even if numpy on top of pypy is itself say 2 or 3x slower than on top of cpython, this could be quite acceptable if you can get speed back on top of it for things for which numpy does not help. This would also enables an iterative process of getting away from cpython internals (which has already started in numpy a couple of years ago, albeit more slowly than everybody would have liked).
- ianb 16y ago
  I hear there is effort to get Numpy working on Cython, and work to get Cython to target PyPy as a backend. Given the two efforts... maybe doable.
  - joeforker 16y ago
    Enough with the Cython nonsense! Cython has excellent marketing but it is a really naive compiler. Just figure out how to translate Cython to ctypes+Python for backwards compatibility and use pypy.
    - glenjamin 16y ago
      Cython is supposed to be a naive compiler, it's primary use-case is for speeding up specific functions by having them compiled. That it uses a python-like language to do this is great. I'm reasonably certain the C parts of numpy rely more on implementation optimisations than compiler optimisations for their increased efficiency.
    - Estragon 16y ago
      You don't know what you're talking about. Using something like Cython (I'm still using Pyrex) to make C extensions dramatically speeds development, if only because it largely solves the reference count problem.
  - 16y ago
ced 16y ago
I also do scientific computing, with scipy. The weave module allows me to write C code inline directly, it's great for optimizing away the bottlenecks. On the minus side, documentation is scarce, the error messages are worthy of template C++, and the idiosyncrasies suck. Upside: I get 10-100X performance improvements! http://www.scipy.org/PerformancePython http://www.scipy.org/PerformancePython
- joelthelion 16y ago
  Now imagine the same kind of performance improvement, without any complicated inline C, wrapping or any of that! I really have high hopes for PyPy :)
  - StavrosK 16y ago
    You can already do that, albeit in a somewhat more limited scope, with shedskin. I've had great success with it.
  - ced 16y ago
    That would be awesome. Right now, with Scipy arrays, you get pretty good performance if you use matrix operations and built-in functions. That can be unnatural, besides it would be more efficient to do a plain for loop in an ideal world. Is PyPy optimized for matrix-style code, for loops, or both? What about functional programming, say, finding the maximum of a user-supplied function? Will the maximizing and maximized functions be inlined for performance? Thank you to any answering PyPy developer :)
  - pwang 16y ago
    Use Cython, then. It's my preferred way of writing C.
kingkilr 16y ago
Well, some of us (the PyPy developers) are web developers, but I'll grant you I'm the exception, not the rule :)
BobKabob 16y ago
It seems to me that there could be a version of Python that is solely optimized for speed, and allows full compilation. I haven't given this a great deal of thought, but I bet that if you defined a subset of the language that is Python-like, but doesn't allow the full introspection and fancy class manipulations, and other cool features, you could have a fully compilable language that would rival C in performance, but kick C's butt in readability. Maybe it's not a binary switch (performance exclusions switch, for example). Maybe it's a setting with various degrees of performance. You need introspection? We'll give you that, but now you can only compile at the 2x performance level (or whatever). You need certain libraries that aren't written for performance? That's going to cost you. Then again, it's quite possible that I don't know what I am talking about!
- ianb 16y ago
  PyPy itself has RPython, which is a restricted Python that can be compiled down to something fast. Cython allows you to annotate Python code (inline or externally) to allow compiling it down. ShedSkin compiles some Python directly to C++. So there's a bunch of options (all of which PyPy has to compete with!)

joe_the_user 16y ago

A lot of scientific code also runs on the (crufty, spiffy) Matlab, which is fast but not super-fast, is less portable than most things but has good usability and lots of history. A python that could run incrementally, with robust threads, in the ground might, just possibly, make SAGE a better environment and so increase migration from MATLAB (my personal scientific computing experience was with Matlab and anything that escape this monster is great in my book). I mean, Python won't become faster than optimized C so however convincing or unconvincing the faster iteration and usability argument is, that, in the end is going have to be what Python makes.

justincormack 16y ago

Right now it looks like LuaJIT is your best option for this type of performance. PyPy might get there eventually though...

ominous_prime 16y ago
LuaJIT is awesome, but python already has a large volume of scientific libraries. I haven't heard mention of scientific researchers using Lua (yet).

cdavid 16y ago

The big issue here is that most scientific code in python runs on top of numpy, and getting numpy to work on top of pypy would be challenging. I understand that relying on a C extension compatibility layer would make things slower at first, but the alternative of re-implementing numpy from scratch in python does not seem realistic to me. Even if numpy on top of pypy is itself say 2 or 3x slower than on top of cpython, this could be quite acceptable if you can get speed back on top of it for things for which numpy does not help. This would also enables an iterative process of getting away from cpython internals (which has already started in numpy a couple of years ago, albeit more slowly than everybody would have liked).

ianb 16y ago
I hear there is effort to get Numpy working on Cython, and work to get Cython to target PyPy as a backend. Given the two efforts... maybe doable.
- joeforker 16y ago
  Enough with the Cython nonsense! Cython has excellent marketing but it is a really naive compiler. Just figure out how to translate Cython to ctypes+Python for backwards compatibility and use pypy.
  - glenjamin 16y ago
    Cython is supposed to be a naive compiler, it's primary use-case is for speeding up specific functions by having them compiled. That it uses a python-like language to do this is great. I'm reasonably certain the C parts of numpy rely more on implementation optimisations than compiler optimisations for their increased efficiency.
  - Estragon 16y ago
    You don't know what you're talking about. Using something like Cython (I'm still using Pyrex) to make C extensions dramatically speeds development, if only because it largely solves the reference count problem.
- 16y ago

ced 16y ago

I also do scientific computing, with scipy. The weave module allows me to write C code inline directly, it's great for optimizing away the bottlenecks. On the minus side, documentation is scarce, the error messages are worthy of template C++, and the idiosyncrasies suck. Upside: I get 10-100X performance improvements! http://www.scipy.org/PerformancePython http://www.scipy.org/PerformancePython

joelthelion 16y ago
Now imagine the same kind of performance improvement, without any complicated inline C, wrapping or any of that! I really have high hopes for PyPy :)
- StavrosK 16y ago
  You can already do that, albeit in a somewhat more limited scope, with shedskin. I've had great success with it.
- ced 16y ago
  That would be awesome. Right now, with Scipy arrays, you get pretty good performance if you use matrix operations and built-in functions. That can be unnatural, besides it would be more efficient to do a plain for loop in an ideal world. Is PyPy optimized for matrix-style code, for loops, or both? What about functional programming, say, finding the maximum of a user-supplied function? Will the maximizing and maximized functions be inlined for performance? Thank you to any answering PyPy developer :)
- pwang 16y ago
  Use Cython, then. It's my preferred way of writing C.

kingkilr 16y ago

Well, some of us (the PyPy developers) are web developers, but I'll grant you I'm the exception, not the rule :)

BobKabob 16y ago

It seems to me that there could be a version of Python that is solely optimized for speed, and allows full compilation. I haven't given this a great deal of thought, but I bet that if you defined a subset of the language that is Python-like, but doesn't allow the full introspection and fancy class manipulations, and other cool features, you could have a fully compilable language that would rival C in performance, but kick C's butt in readability. Maybe it's not a binary switch (performance exclusions switch, for example). Maybe it's a setting with various degrees of performance. You need introspection? We'll give you that, but now you can only compile at the 2x performance level (or whatever). You need certain libraries that aren't written for performance? That's going to cost you. Then again, it's quite possible that I don't know what I am talking about!

ianb 16y ago
PyPy itself has RPython, which is a restricted Python that can be compiled down to something fast. Cython allows you to annotate Python code (inline or externally) to allow compiling it down. ShedSkin compiles some Python directly to C++. So there's a bunch of options (all of which PyPy has to compete with!)