16 ms·

Python C API, PyPy and the road into the future

178 points by mattip 11y ago

tomrod 11y ago
I'm afraid I'm not up to speed on pypy. What is the benefit here?
- giaour 11y ago
  It tends to considerably outperform CPython on performance benchmarks.
- deleted 11y ago
  [deleted]
- x0x0 11y ago
  Speed. In many common cases, pypy is significantly faster than the python C runtime. Unfortunately, the python C api was not designed with a jit in mind.
  - p4wnc6 11y ago
    But as Numba has shown, it's achievable to write a CPython-to-LLVM-IR compiler for subsets of the language, then add NumPy-awareness, and add multiple targets like GPUs, and achieve better-than-PyPy speed ups jitting pure Python (CPython) code. I think it's an interesting question as to whether PyPy "is the future" or rather something more like Numba, or even a whole ecosystem of specialized variants of Numba that may be tweaked or tuned to take advantage of different hardware optimizations, work better for different domain problems (e.g. image processing vs. network programming). In a lot of cases where PyPy improves over some standard CPython code, it's exactly the sort of code you really wouldn't care about optimizing in the context of a larger application. In any sort of scientific application, the pure CPython parts of it are usually small and insignificant compared to what's written in NumPy/SciPy/pandas and science libraries written on top of those, and Numba can be seen as the first "specializing compiler" for this sort of application -- with a lot of room for people to write other kinds of specializing compilers that compile different genres of Python code for domain-specific purposes, the way that Numba compiles generally scientific Python for numerical computing purposes.
    - duaneb 11y ago
      > But as Numba has shown, it's achievable to write a CPython-to-LLVM-IR compiler for subsets of the language, then add NumPy-awareness, and add multiple targets like GPUs, and achieve better-than-PyPy speed ups jitting pure Python (CPython) code. Most python code I write would get 0 speedup from that. I think numeric optimization is a very different target than compiling and optimizing general purpose python code. I don't see either strategies "winning" in the near future.
    - Veedrac 11y ago
      > But as Numba has shown, it's achievable to write a CPython-to-LLVM-IR compiler for subsets of the language, then add NumPy-awareness, and add multiple targets like GPUs, and achieve better-than-PyPy speed ups jitting pure Python (CPython) code. Other than "to-LLVM" and "targets like GPUs", jitpy does exactly that for PyPy-in-CPython. http://jitpy.readthedocs.org/en/latest/ http://jitpy.readthedocs.org/en/latest/ It's not quite as fast for numeric code (though it's close and conformant) but it's also not a language subset nor does it degrade to CPython performance on dynamic parts of the language. It should be a terrific option if you're able to section out discrete workloads that CPython's struggling with, even in non-numeric code. Few people know of it though.
      - p4wnc6 11y ago
        Thanks for the link to JitPy. I will definitely read more about it. Just from the first parts of the docs, though, it looks very limited compared with Numba. In Numba, there is a distinction between 'python mode' and 'nopython mode' -- meaning that even at the final LLVM IR emission, if Numba is forced to punt on type inference (e.g. because an untyped Python list object is an argument or something), Numba can still fall back directly to the CPython API, and in general even this has speed benefits as a lot of intermediate calls or intermediate variables can be lowered or typed. So even though JitPy won't "degrade to CPython performance", it's slightly moot since you can only handle a limited set of types in JitPy. I guess one area where JitPy should win in theory is if there is a dynamic Python object inside of the function, like a dynamic list. Then JitPy is OK since it's not part of the signature, and when it can't optimize due to indeterminate type inside the function, the fallback will be PyPy instead of CPython, and hence should be faster. But Numba offers a lot on the array computing side, which is what it is specialized for. For example, from the Numba docs < http://numba.pydata.org/numba-doc/0.21.0/developer/architecture.html#stage-5-rewrite-typed-ir http://numba.pydata.org/numba-doc/0.21.0/developer/architect... >: > Numba implements a user-extensible rewriting pass that reads and possibly rewrites Numba IR. This pass’s purpose is to perform any high-level optimizations that still require, or could at least benefit from, Numba IR type information. > One example of a problem domain that isn’t as easily optimized once lowered is the domain of multidimensional array operations. When Numba lowers an array operation, Numba treats the operation like a full ufunc kernel. During lowering a single array operation, Numba generates an inline broadcasting loop that creates a new result array. Then Numba generates an application loop that applies the operator over the array inputs. Recognizing and rewriting these loops once they are lowered into LLVM is hard, if not impossible. > An example pair of optimizations in the domain of array operators is loop fusion and shortcut deforestation. When the optimizer recognizes that the output of one array operator is being fed into another array operator, and only to that array operator, it can fuse the two loops into a single loop. The optimizer can further eliminate the temporary array allocated for the initial operation by directly feeding the result of the first operation into the second, skipping the store and load to the intermediate array. This elimination is known as shortcut deforestation. Numba currently uses the rewrite pass to implement these array optimizations. For more information, please consult the “Case study: Array Expressions” subsection, later in this document. One reason why I see the Numba route as more effective than the PyPy route is that these kinds of highly-specific optimizations seem likely to occur all over the place, and to be very related to the genre of computing you are doing. One person may not want Numba to perform these optimizations, because loop fusion isn't important to their codebase which doesn't do a lot of array computing. Instead, they might want some kind of domain-specific optimization that assists with a kernel bypass for some low-latency algorithm. This could all be exposed in Numba, but as separate domain-specific sub-modules or via user-defined IR optimization passes or something. Or it could exist as totally separate JIT compiler projects, of which JitPy/PyPy's JIT compiler and Numba's JIT compiler would each just be genre-specific examples. I tend to see this as the more likely future, and that apart from cases where someone is really just taking a bunch of pure Python code and running it with PyPy interpreter instead of CPython interpreter (which I forecast would be a rare case), the difference between PyPy and CPython won't matter nearly as much as the difference between domain-specific JIT compilers.
- hedgehog 11y ago
  PyPy is much faster for many workloads. It's a garbage collected just-in-time compiler and mostly compatible with the CPython runtime (the one you get from python.org). It also happens to be written in a subset of Python which is pretty neat. The CPython API exposes a bunch of details specific to the CPython implementation (like object reference counts) and so hasn't historically been supported in PyPy (which supports the whole language but internally is very different). It sounds like it would be a pain to do but the PyPy team thinks it's feasible to build.
jboy 11y ago
This post highlights an interesting dichotomy in the Python scientific computing community. Everyone knows that PyPy runs faster than CPython for many common tasks [1]. [1] PyPy is on average 7x faster than CPython: http://speed.pypy.org/ http://speed.pypy.org/ But those in the Python community who are serious about scientific computing (or image processing, like my startup) are already using Numpy & Scipy, which provide hand-coded C implementations of most matrix-related operations. Everyone knows that Python for-loops are "slow" [2], and storing a large 2-D matrix as a list-of-list-of-Python-int-object would require a huge amount of memory & indirection. So, Numpy offers an N-dimensional array type, implemented in C: C arrays of densely-packed C primitive types, with for-loops in C to iterate over the matrix elements. Then Scipy builds a lot of Matlab-like functionality as modules on top of this fundamental Numpy array type. [2] Python for-loops are slow: https://jakevdp.github.io/blog/2014/05/09/why-python-is-slow/ https://jakevdp.github.io/blog/2014/05/09/why-python-is-slow... So the most expensive operations in a Python number-crunching program are likely already implemented using Numpy & Scipy operations, which run in compiled C (and additionally, often make use of Blas/Atlas/LAPACK/etc, for even greater speedups in sustained number-crunching). But unfortunately, Numpy & PyPy do not naturally work together. Being written in C, Numpy makes substantial use of the CPython C-API -- and in fact, Numpy provides its own C-API [3]! The official Numpy package doesn't work with PyPy; the PyPy project very thoughtfully provides its own PyPy-compatible Numpy package [4]. [3] Numpy C-API: http://docs.scipy.org/doc/numpy-1.10.0/reference/c-api.html http://docs.scipy.org/doc/numpy-1.10.0/reference/c-api.html [4] PyPy-compatible Numpy package: http://pypy.org/download.html#installing-numpy http://pypy.org/download.html#installing-numpy Furthermore, Numpy is fantastic, but it can't offer all possible permutations of matrix operations. In particular, there are certain image-processing operations that are awkward (and thus, computationally-inefficient) to express using Numpy operations. So you might ultimately need to go to the Numpy C-API anyway. This is why we created (and, just a few days ago, open-sourced) Pymod: https://github.com/jboy/nim-pymod https://github.com/jboy/nim-pymod Pymod is a Nim+Python project that auto-generates all the Python C-API boilerplate & auto-compiles a Python C extension module that wraps the functions in a Nim module. Pymod enables us to write our Numpy array-processing code in Nim, then compile it (for C++-like speeds) as a well-behaved Python module. Nim made this very easy, because it compiles to C. After considering our Python-integration options (CPython C-API, `ctypes` and `cffi`), we decided to go with the CPython C-API & Numpy C-API. We explained this decision in greater detail in the "Implementation details" section of the Pymod README [5]; the executive summary is that `ctypes` seems better suited to wrapping C types in Python, rather than exposing existing Python types in C, while the CPython C-API code could be generated & compiled with the C code that Nim was going to produce anyway. [5] Pymod implementation details: https://github.com/jboy/nim-pymod#implementation-details https://github.com/jboy/nim-pymod#implementation-details That said, we would be delighted for Pymod-produced Python modules to be able to run under PyPy. We've been strongly considering implementing a `cffi` back-end for Pymod, but this won't necessarily solve the Numpy issue. It would be even better if PyPy could support all the CPython C-API extension modules in the world in one fell swoop.
_RPM 11y ago
One of the great things about Python is the C API and how well documented it is.
- aardvark179 11y ago
  APIs like that are both good and bad because they let you do almost anything so make many optimisations either hard or impossible, and have a nasty habit of exposing implementation details which constrain future changes to the language implementation. Even APIs such as JNI that do their best to abstract over such details tend to block a JIT's view of what is going on and end up being a real pain.
tych0 11y ago
I have nothing but good things to say about the pypy/cffi team. I have been maintaining a project using it for about a year and a half, and we've had great experiences reporting bugs to and getting help from Armin (the maintainer of cffi, a pypy guy). Distributing cffi libraries that depend on each other can still be somewhat of a snarl, but last I looked there are a lot of remedies for this being discussed. Chapeau to the team!
binarycrusader 11y ago
I strongly disagree with the author's assertions that the need for better C API support is strictly a need for integrating with "legacy applications". The fact of the matter is that a vast array of systems-level software and high-performance software is only available via a C API interface. New or old, that's still true today. I've used ctypes and I've used cffi; I'm glad they exist, but authors of popular packages often don't have the luxury of supporting both of them and the CPython API. If by legacy, they're attempting to paint the CPython API as the legacy one, I also find that a misleading argument at best. PyPy is not yet the official future of Python, and attempting to paint themselves as the successor without any clear indicator of support for that future from the Python leadership seems odd. I'm all for them getting financial support they need to do the work, but attempting to justify that work by claiming only "legacy" applications need it seems uninformed at best.
- frozenport 11y ago
  C APIs also offer significantly easier distribution, as you don't need to worry about things like name mangling or support for other languages. Much better than alternatives like COM.
- DasIch 11y ago
  The author only claims that better CPython C API support is needed for legacy applications. Of course there are a vast number of applications that have C API themselves that one might want to accesses. You can interact with those through cffi in a way that's simple and performant already.
  - bpicolo 11y ago
    The bigger issue is the larger number of packages using the C Api, not the applications themselves
  - stuaxo 11y ago
    GUI apps is an area that comes to mind. Gtk3 uses the C API, although you can try and use PGI it is not fully ready. In fact, lots of old graphics and sound APIs are the same... basically anything that is not hooked up to the web.
BuckRogers 11y ago
I've been using PyPy as my main development platform for quite some time now. So for me if I were to move to Python3, it would have to be Python 3.2 since that's all PyPy3 supports. I've also been told that PyPy3 is not production-ready. While PyPy4 is, so it's a no brainer for me to develop for PyPy4 then fallback to CPython2.7 if I run into problems. Fingers crossed, but so far my home business is running off pure PyPy4. No C-extensions, no interpreters. :)
ericfrederich 11y ago
Personally I'd be in favor of a more general solution... Some kind of cffi/CPython bridge.

178 points by mattip 11y ago

tomrod 11y ago

I'm afraid I'm not up to speed on pypy. What is the benefit here?

giaour 11y ago
It tends to considerably outperform CPython on performance benchmarks.
deleted 11y ago
[deleted]
x0x0 11y ago
Speed. In many common cases, pypy is significantly faster than the python C runtime. Unfortunately, the python C api was not designed with a jit in mind.
- p4wnc6 11y ago
  But as Numba has shown, it's achievable to write a CPython-to-LLVM-IR compiler for subsets of the language, then add NumPy-awareness, and add multiple targets like GPUs, and achieve better-than-PyPy speed ups jitting pure Python (CPython) code. I think it's an interesting question as to whether PyPy "is the future" or rather something more like Numba, or even a whole ecosystem of specialized variants of Numba that may be tweaked or tuned to take advantage of different hardware optimizations, work better for different domain problems (e.g. image processing vs. network programming). In a lot of cases where PyPy improves over some standard CPython code, it's exactly the sort of code you really wouldn't care about optimizing in the context of a larger application. In any sort of scientific application, the pure CPython parts of it are usually small and insignificant compared to what's written in NumPy/SciPy/pandas and science libraries written on top of those, and Numba can be seen as the first "specializing compiler" for this sort of application -- with a lot of room for people to write other kinds of specializing compilers that compile different genres of Python code for domain-specific purposes, the way that Numba compiles generally scientific Python for numerical computing purposes.
  - duaneb 11y ago
    > But as Numba has shown, it's achievable to write a CPython-to-LLVM-IR compiler for subsets of the language, then add NumPy-awareness, and add multiple targets like GPUs, and achieve better-than-PyPy speed ups jitting pure Python (CPython) code. Most python code I write would get 0 speedup from that. I think numeric optimization is a very different target than compiling and optimizing general purpose python code. I don't see either strategies "winning" in the near future.
  - Veedrac 11y ago
    > But as Numba has shown, it's achievable to write a CPython-to-LLVM-IR compiler for subsets of the language, then add NumPy-awareness, and add multiple targets like GPUs, and achieve better-than-PyPy speed ups jitting pure Python (CPython) code. Other than "to-LLVM" and "targets like GPUs", jitpy does exactly that for PyPy-in-CPython. http://jitpy.readthedocs.org/en/latest/ http://jitpy.readthedocs.org/en/latest/ It's not quite as fast for numeric code (though it's close and conformant) but it's also not a language subset nor does it degrade to CPython performance on dynamic parts of the language. It should be a terrific option if you're able to section out discrete workloads that CPython's struggling with, even in non-numeric code. Few people know of it though.
    - p4wnc6 11y ago
      Thanks for the link to JitPy. I will definitely read more about it. Just from the first parts of the docs, though, it looks very limited compared with Numba. In Numba, there is a distinction between 'python mode' and 'nopython mode' -- meaning that even at the final LLVM IR emission, if Numba is forced to punt on type inference (e.g. because an untyped Python list object is an argument or something), Numba can still fall back directly to the CPython API, and in general even this has speed benefits as a lot of intermediate calls or intermediate variables can be lowered or typed. So even though JitPy won't "degrade to CPython performance", it's slightly moot since you can only handle a limited set of types in JitPy. I guess one area where JitPy should win in theory is if there is a dynamic Python object inside of the function, like a dynamic list. Then JitPy is OK since it's not part of the signature, and when it can't optimize due to indeterminate type inside the function, the fallback will be PyPy instead of CPython, and hence should be faster. But Numba offers a lot on the array computing side, which is what it is specialized for. For example, from the Numba docs < http://numba.pydata.org/numba-doc/0.21.0/developer/architecture.html#stage-5-rewrite-typed-ir http://numba.pydata.org/numba-doc/0.21.0/developer/architect... >: > Numba implements a user-extensible rewriting pass that reads and possibly rewrites Numba IR. This pass’s purpose is to perform any high-level optimizations that still require, or could at least benefit from, Numba IR type information. > One example of a problem domain that isn’t as easily optimized once lowered is the domain of multidimensional array operations. When Numba lowers an array operation, Numba treats the operation like a full ufunc kernel. During lowering a single array operation, Numba generates an inline broadcasting loop that creates a new result array. Then Numba generates an application loop that applies the operator over the array inputs. Recognizing and rewriting these loops once they are lowered into LLVM is hard, if not impossible. > An example pair of optimizations in the domain of array operators is loop fusion and shortcut deforestation. When the optimizer recognizes that the output of one array operator is being fed into another array operator, and only to that array operator, it can fuse the two loops into a single loop. The optimizer can further eliminate the temporary array allocated for the initial operation by directly feeding the result of the first operation into the second, skipping the store and load to the intermediate array. This elimination is known as shortcut deforestation. Numba currently uses the rewrite pass to implement these array optimizations. For more information, please consult the “Case study: Array Expressions” subsection, later in this document. One reason why I see the Numba route as more effective than the PyPy route is that these kinds of highly-specific optimizations seem likely to occur all over the place, and to be very related to the genre of computing you are doing. One person may not want Numba to perform these optimizations, because loop fusion isn't important to their codebase which doesn't do a lot of array computing. Instead, they might want some kind of domain-specific optimization that assists with a kernel bypass for some low-latency algorithm. This could all be exposed in Numba, but as separate domain-specific sub-modules or via user-defined IR optimization passes or something. Or it could exist as totally separate JIT compiler projects, of which JitPy/PyPy's JIT compiler and Numba's JIT compiler would each just be genre-specific examples. I tend to see this as the more likely future, and that apart from cases where someone is really just taking a bunch of pure Python code and running it with PyPy interpreter instead of CPython interpreter (which I forecast would be a rare case), the difference between PyPy and CPython won't matter nearly as much as the difference between domain-specific JIT compilers.
hedgehog 11y ago
PyPy is much faster for many workloads. It's a garbage collected just-in-time compiler and mostly compatible with the CPython runtime (the one you get from python.org). It also happens to be written in a subset of Python which is pretty neat. The CPython API exposes a bunch of details specific to the CPython implementation (like object reference counts) and so hasn't historically been supported in PyPy (which supports the whole language but internally is very different). It sounds like it would be a pain to do but the PyPy team thinks it's feasible to build.

jboy 11y ago

This post highlights an interesting dichotomy in the Python scientific computing community. Everyone knows that PyPy runs faster than CPython for many common tasks [1]. [1] PyPy is on average 7x faster than CPython: http://speed.pypy.org/ http://speed.pypy.org/ But those in the Python community who are serious about scientific computing (or image processing, like my startup) are already using Numpy & Scipy, which provide hand-coded C implementations of most matrix-related operations. Everyone knows that Python for-loops are "slow" [2], and storing a large 2-D matrix as a list-of-list-of-Python-int-object would require a huge amount of memory & indirection. So, Numpy offers an N-dimensional array type, implemented in C: C arrays of densely-packed C primitive types, with for-loops in C to iterate over the matrix elements. Then Scipy builds a lot of Matlab-like functionality as modules on top of this fundamental Numpy array type. [2] Python for-loops are slow: https://jakevdp.github.io/blog/2014/05/09/why-python-is-slow/ https://jakevdp.github.io/blog/2014/05/09/why-python-is-slow... So the most expensive operations in a Python number-crunching program are likely already implemented using Numpy & Scipy operations, which run in compiled C (and additionally, often make use of Blas/Atlas/LAPACK/etc, for even greater speedups in sustained number-crunching). But unfortunately, Numpy & PyPy do not naturally work together. Being written in C, Numpy makes substantial use of the CPython C-API -- and in fact, Numpy provides its own C-API [3]! The official Numpy package doesn't work with PyPy; the PyPy project very thoughtfully provides its own PyPy-compatible Numpy package [4]. [3] Numpy C-API: http://docs.scipy.org/doc/numpy-1.10.0/reference/c-api.html http://docs.scipy.org/doc/numpy-1.10.0/reference/c-api.html [4] PyPy-compatible Numpy package: http://pypy.org/download.html#installing-numpy http://pypy.org/download.html#installing-numpy Furthermore, Numpy is fantastic, but it can't offer all possible permutations of matrix operations. In particular, there are certain image-processing operations that are awkward (and thus, computationally-inefficient) to express using Numpy operations. So you might ultimately need to go to the Numpy C-API anyway. This is why we created (and, just a few days ago, open-sourced) Pymod: https://github.com/jboy/nim-pymod https://github.com/jboy/nim-pymod Pymod is a Nim+Python project that auto-generates all the Python C-API boilerplate & auto-compiles a Python C extension module that wraps the functions in a Nim module. Pymod enables us to write our Numpy array-processing code in Nim, then compile it (for C++-like speeds) as a well-behaved Python module. Nim made this very easy, because it compiles to C. After considering our Python-integration options (CPython C-API, `ctypes` and `cffi`), we decided to go with the CPython C-API & Numpy C-API. We explained this decision in greater detail in the "Implementation details" section of the Pymod README [5]; the executive summary is that `ctypes` seems better suited to wrapping C types in Python, rather than exposing existing Python types in C, while the CPython C-API code could be generated & compiled with the C code that Nim was going to produce anyway. [5] Pymod implementation details: https://github.com/jboy/nim-pymod#implementation-details https://github.com/jboy/nim-pymod#implementation-details That said, we would be delighted for Pymod-produced Python modules to be able to run under PyPy. We've been strongly considering implementing a `cffi` back-end for Pymod, but this won't necessarily solve the Numpy issue. It would be even better if PyPy could support all the CPython C-API extension modules in the world in one fell swoop.

_RPM 11y ago

One of the great things about Python is the C API and how well documented it is.

aardvark179 11y ago
APIs like that are both good and bad because they let you do almost anything so make many optimisations either hard or impossible, and have a nasty habit of exposing implementation details which constrain future changes to the language implementation. Even APIs such as JNI that do their best to abstract over such details tend to block a JIT's view of what is going on and end up being a real pain.

tych0 11y ago

I have nothing but good things to say about the pypy/cffi team. I have been maintaining a project using it for about a year and a half, and we've had great experiences reporting bugs to and getting help from Armin (the maintainer of cffi, a pypy guy). Distributing cffi libraries that depend on each other can still be somewhat of a snarl, but last I looked there are a lot of remedies for this being discussed. Chapeau to the team!

binarycrusader 11y ago

I strongly disagree with the author's assertions that the need for better C API support is strictly a need for integrating with "legacy applications". The fact of the matter is that a vast array of systems-level software and high-performance software is only available via a C API interface. New or old, that's still true today. I've used ctypes and I've used cffi; I'm glad they exist, but authors of popular packages often don't have the luxury of supporting both of them and the CPython API. If by legacy, they're attempting to paint the CPython API as the legacy one, I also find that a misleading argument at best. PyPy is not yet the official future of Python, and attempting to paint themselves as the successor without any clear indicator of support for that future from the Python leadership seems odd. I'm all for them getting financial support they need to do the work, but attempting to justify that work by claiming only "legacy" applications need it seems uninformed at best.

frozenport 11y ago
C APIs also offer significantly easier distribution, as you don't need to worry about things like name mangling or support for other languages. Much better than alternatives like COM.
DasIch 11y ago
The author only claims that better CPython C API support is needed for legacy applications. Of course there are a vast number of applications that have C API themselves that one might want to accesses. You can interact with those through cffi in a way that's simple and performant already.
- bpicolo 11y ago
  The bigger issue is the larger number of packages using the C Api, not the applications themselves
- stuaxo 11y ago
  GUI apps is an area that comes to mind. Gtk3 uses the C API, although you can try and use PGI it is not fully ready. In fact, lots of old graphics and sound APIs are the same... basically anything that is not hooked up to the web.

BuckRogers 11y ago

I've been using PyPy as my main development platform for quite some time now. So for me if I were to move to Python3, it would have to be Python 3.2 since that's all PyPy3 supports. I've also been told that PyPy3 is not production-ready. While PyPy4 is, so it's a no brainer for me to develop for PyPy4 then fallback to CPython2.7 if I run into problems. Fingers crossed, but so far my home business is running off pure PyPy4. No C-extensions, no interpreters. :)

ericfrederich 11y ago

Personally I'd be in favor of a more general solution... Some kind of cffi/CPython bridge.