6 ms·

Can someone knowledgeable compare PyPy, Numba and Cython? I mostly use Cython. Tried Numba also, it has very nice workflow when it works with autojit (when it d

Can someone knowledgeable compare PyPy, Numba and Cython? I mostly use Cython. Tried Numba also, it has very nice workflow when it works with autojit (when it doesn't error messages are pretty cryptic). With PyPy I don't understand why for example all that type information obtained from jit wouldn't be used to make something like Numba specialized functions or Cython modules (noob question but please answer).

DasIch 13y ago
Cython is a compiler not an interpreter. Numba requires explicit hinting and can only optimize some undefined (?) subset of Python. PyPy on the other hand is simply a fast Python interpreter. Not sure what you mean regarding the type information gathered by the JIT.
- MichailP 13y ago
  For example does PyPy always warm up, or can it store "warmed up" version of some function (where type of objects is inferred)? I know that this is not in accordance with highly dynamic nature of Python, but not all functions are highly dynamic.
  - cglace 13y ago
    Do you mean can you save the state of the JIT for future use?
    - rubinelli 13y ago
      Yeah. Can you? Considering CPython already generates .pyc and checks timestamps, keeping some kind of JIT cache around wouldn't be a stretch.
      - kingkilr 13y ago
        No, the state of the JIT cannot be saved between executions.
      - gcr 13y ago
        I can't speak for python, but I do know that the impact of translating code for Racket's JIT (provided by GNU lightning, not LLVM) is negligable (ie. takes less time than the time you gain by JITting
  - rcthompson 13y ago
    My understanding (from a previous HN discussion about javascript) is that since the output from a "just-in-time compiler" is actually machine code generated on the fly, it includes direct references to memory locations that are only valid for the lifetime of the process. So the output of a JIT is simply not in a format that can be saved and reloaded later.
    - nickik 13y ago
      Actually its possible to have a JIT like that but it is uncommon.
- andreasvc 13y ago
  Cython is not actually a compiler. It is essentially a more convenient way of writing C/C++ extension modules. It produces C/C++ glue code which is then compiled by an actual compiler like gcc. By tapping into the existing C API of cpython it is simpler than PyPy & NumPy which work on a lower level, but on the other hand Cython does little optimization for you.
Thrymr 13y ago
PyPy still doesn't work with Numpy, though they're working on it.
- MostAwesomeDude 13y ago
  Numba is not Numpy.
- apendleton 13y ago
  That's not quite right. Stock numpy doesn't install, and nobody's working on that as far as I know, but they've reimplemented part of the numpy API (at least the Python API, not the C one), and the subset they've implemented works fine. I have an application in production that uses it.
- robert-zaremba 13y ago
  PyPy has it's own implementation of Numpy (called numpypy). It's not 100% complete but quiet functional. http://morepypy.blogspot.com/search?q=numpypy http://morepypy.blogspot.com/search?q=numpypy
Derbasti 13y ago
Have you tried the cffi? I used to use cython, but since I found the cffi, I haven't looked back.
- MichailP 13y ago
  No, I got scared away by Simple example and Real example from the docs :) Do you maybe have some hints/links for cffi?
- robert-zaremba 13y ago
  There are a lot of cffi c bindings. I used with success: * https://github.com/amauryfa/lxml/tree/lxml-cffi https://github.com/amauryfa/lxml/tree/lxml-cffi * https://github.com/chtd/psycopg2cffi https://github.com/chtd/psycopg2cffi

DasIch 13y ago

Cython is a compiler not an interpreter. Numba requires explicit hinting and can only optimize some undefined (?) subset of Python. PyPy on the other hand is simply a fast Python interpreter. Not sure what you mean regarding the type information gathered by the JIT.

MichailP 13y ago
For example does PyPy always warm up, or can it store "warmed up" version of some function (where type of objects is inferred)? I know that this is not in accordance with highly dynamic nature of Python, but not all functions are highly dynamic.
- cglace 13y ago
  Do you mean can you save the state of the JIT for future use?
  - rubinelli 13y ago
    Yeah. Can you? Considering CPython already generates .pyc and checks timestamps, keeping some kind of JIT cache around wouldn't be a stretch.
    - kingkilr 13y ago
      No, the state of the JIT cannot be saved between executions.
    - gcr 13y ago
      I can't speak for python, but I do know that the impact of translating code for Racket's JIT (provided by GNU lightning, not LLVM) is negligable (ie. takes less time than the time you gain by JITting
- rcthompson 13y ago
  My understanding (from a previous HN discussion about javascript) is that since the output from a "just-in-time compiler" is actually machine code generated on the fly, it includes direct references to memory locations that are only valid for the lifetime of the process. So the output of a JIT is simply not in a format that can be saved and reloaded later.
  - nickik 13y ago
    Actually its possible to have a JIT like that but it is uncommon.
andreasvc 13y ago
Cython is not actually a compiler. It is essentially a more convenient way of writing C/C++ extension modules. It produces C/C++ glue code which is then compiled by an actual compiler like gcc. By tapping into the existing C API of cpython it is simpler than PyPy & NumPy which work on a lower level, but on the other hand Cython does little optimization for you.

Thrymr 13y ago

PyPy still doesn't work with Numpy, though they're working on it.

MostAwesomeDude 13y ago
Numba is not Numpy.
apendleton 13y ago
That's not quite right. Stock numpy doesn't install, and nobody's working on that as far as I know, but they've reimplemented part of the numpy API (at least the Python API, not the C one), and the subset they've implemented works fine. I have an application in production that uses it.
robert-zaremba 13y ago
PyPy has it's own implementation of Numpy (called numpypy). It's not 100% complete but quiet functional. http://morepypy.blogspot.com/search?q=numpypy http://morepypy.blogspot.com/search?q=numpypy

Derbasti 13y ago

Have you tried the cffi? I used to use cython, but since I found the cffi, I haven't looked back.

MichailP 13y ago
No, I got scared away by Simple example and Real example from the docs :) Do you maybe have some hints/links for cffi?
robert-zaremba 13y ago
There are a lot of cffi c bindings. I used with success: * https://github.com/amauryfa/lxml/tree/lxml-cffi https://github.com/amauryfa/lxml/tree/lxml-cffi * https://github.com/chtd/psycopg2cffi https://github.com/chtd/psycopg2cffi