6 ms·

Is it possible to get cython to build C extensions that are compatible with pypy? If not, is that something the pypy developers are interested in? A lot of the

Is it possible to get cython to build C extensions that are compatible with pypy? If not, is that something the pypy developers are interested in? A lot of the code I use depends heavily on C extensions, but it would be nice if I could the power of pypy's JIT compiler on the pure-python portions of my codebase.

dman 11y ago
You might want to edit that comment. I suspect "depends heavily on pypy" should be "depends heavily on C extensions".
msellout 11y ago
AFAIK it's partially supported and under development. http://pypy.readthedocs.org/en/latest/faq.html#do-cpython-extension-modules-work-with-pypy http://pypy.readthedocs.org/en/latest/faq.html#do-cpython-ex...
- CuttlefishXXX 11y ago
  Your link mentions CPython (the reference/official Python interpreter), not Cython (optimizing static compiler and extension of the Python language) so I'm not sure it answers the question raised by parent.
  - msellout 11y ago
    You're right. The parent didn't provide enough information on the use of Cython. And frankly, there's no better way to find out whether PyPy supports a particular C extension than to try it.
wirrbel 11y ago
wouldn't it be feasible to write a Cython compiler that doesn't compile to CPython API but to the ffi that PyPy likes more?
- srean 11y ago
  ...or to RPython to expose more surface area to the JIT optimizer. BTW very curious of our login name. There used to be this type inferred language called Wirbel that compiled a subset of Python to native binary. Are you anyway associated ?
  - BlackFingolfin 11y ago
    "Wirbel" ist german for whirl, swirl, vortex, ... -- and "wirr" means roughly "confused" or "light in the head" -- my guess is that this user speaks German and the nick is a pun ;-).
  - wirrbel 11y ago
    not associated. It is indeed a kind of a pun. wirbel is whirl/swirl and also metaphorically "fuss", wirr means confused. So I put those words together, as BlackFingolfin described.
- fijal 11y ago
  see my comment above - in short, very feasible, but someone has to care enough to do (I'm not volunteering)
fijal 11y ago
Currently it should work via cpyext (CPython C API legacy infrastructure), but that's gonna be very slow It's possible to write a Cython backend that would generate cffi bindings instead of C, but the work has not been done yet, so it's a bit unclear when it'll happen. Would be a good summer of code project for example.
- empyrical 11y ago
  Some more info/caveats of Cython's cpyext support can be found here: http://docs.cython.org/src/userguide/pypy.html http://docs.cython.org/src/userguide/pypy.html https://github.com/cython/cython/wiki/enhancements-pypy https://github.com/cython/cython/wiki/enhancements-pypy
- TheCondor 11y ago
  Off topic, but you might consider flagging some bugs as good starter bugs for joining the project. You have great technology but it's a challenge diving in to it, even with the docs.
Animats 11y ago
If PyPy gets good enough, some of the need for C extensions should go away. Things such as "numpy" are written in C because numbers in Python are always boxed, that is, contained in a object. PyPy tries to figure out when that's not necessary, so it can use hardware arithmetic. PyPy has to work very hard to do this, because you can muck with almost anything from anywhere with "setattr()" Even from other threads. So any guess PyPy makes has to be un-doable at any time. Python, the language, is what a naive interpreter such as CPython can do easily. In CPython, almost everything is a dictionary, and there are a lot of lookups. PyPy gets rid of much of that overhead. I'm enormously impressed that the PyPy crowd got it to work as well as they did.
- DasIch 11y ago
  These libraries are not just used because of performance. Some of them are quite old, large and partly written in Fortran. Unless you can do better in other ways than performance, you have to be quite a bit faster to make the effort of porting all these libraries worthwhile.
- JulianWasTaken 11y ago
  PyPy already is good enough for the need for C extensions to go away in a very large majority of cases.
chrisseaton 11y ago
Another option is to write an interpreter for the C extension source code that works like the Python interpreter in PyPy. This is what we do to support C extensions in JRuby+Truffle http://chrisseaton.com/rubytruffle/cext/ http://chrisseaton.com/rubytruffle/cext/.

dman 11y ago

You might want to edit that comment. I suspect "depends heavily on pypy" should be "depends heavily on C extensions".

msellout 11y ago

AFAIK it's partially supported and under development. http://pypy.readthedocs.org/en/latest/faq.html#do-cpython-extension-modules-work-with-pypy http://pypy.readthedocs.org/en/latest/faq.html#do-cpython-ex...

CuttlefishXXX 11y ago
Your link mentions CPython (the reference/official Python interpreter), not Cython (optimizing static compiler and extension of the Python language) so I'm not sure it answers the question raised by parent.
- msellout 11y ago
  You're right. The parent didn't provide enough information on the use of Cython. And frankly, there's no better way to find out whether PyPy supports a particular C extension than to try it.

wirrbel 11y ago

wouldn't it be feasible to write a Cython compiler that doesn't compile to CPython API but to the ffi that PyPy likes more?

srean 11y ago
...or to RPython to expose more surface area to the JIT optimizer. BTW very curious of our login name. There used to be this type inferred language called Wirbel that compiled a subset of Python to native binary. Are you anyway associated ?
- BlackFingolfin 11y ago
  "Wirbel" ist german for whirl, swirl, vortex, ... -- and "wirr" means roughly "confused" or "light in the head" -- my guess is that this user speaks German and the nick is a pun ;-).
- wirrbel 11y ago
  not associated. It is indeed a kind of a pun. wirbel is whirl/swirl and also metaphorically "fuss", wirr means confused. So I put those words together, as BlackFingolfin described.
fijal 11y ago
see my comment above - in short, very feasible, but someone has to care enough to do (I'm not volunteering)

fijal 11y ago

Currently it should work via cpyext (CPython C API legacy infrastructure), but that's gonna be very slow It's possible to write a Cython backend that would generate cffi bindings instead of C, but the work has not been done yet, so it's a bit unclear when it'll happen. Would be a good summer of code project for example.

empyrical 11y ago
Some more info/caveats of Cython's cpyext support can be found here: http://docs.cython.org/src/userguide/pypy.html http://docs.cython.org/src/userguide/pypy.html https://github.com/cython/cython/wiki/enhancements-pypy https://github.com/cython/cython/wiki/enhancements-pypy
TheCondor 11y ago
Off topic, but you might consider flagging some bugs as good starter bugs for joining the project. You have great technology but it's a challenge diving in to it, even with the docs.

Animats 11y ago

If PyPy gets good enough, some of the need for C extensions should go away. Things such as "numpy" are written in C because numbers in Python are always boxed, that is, contained in a object. PyPy tries to figure out when that's not necessary, so it can use hardware arithmetic. PyPy has to work very hard to do this, because you can muck with almost anything from anywhere with "setattr()" Even from other threads. So any guess PyPy makes has to be un-doable at any time. Python, the language, is what a naive interpreter such as CPython can do easily. In CPython, almost everything is a dictionary, and there are a lot of lookups. PyPy gets rid of much of that overhead. I'm enormously impressed that the PyPy crowd got it to work as well as they did.

DasIch 11y ago
These libraries are not just used because of performance. Some of them are quite old, large and partly written in Fortran. Unless you can do better in other ways than performance, you have to be quite a bit faster to make the effort of porting all these libraries worthwhile.
JulianWasTaken 11y ago
PyPy already is good enough for the need for C extensions to go away in a very large majority of cases.

chrisseaton 11y ago

Another option is to write an interpreter for the C extension source code that works like the Python interpreter in PyPy. This is what we do to support C extensions in JRuby+Truffle http://chrisseaton.com/rubytruffle/cext/ http://chrisseaton.com/rubytruffle/cext/.