4 ms·

If performance is your only motivation, I recommend you have a look at cython (previously pyrex). It basically makes writing C modules a breeze for a python pro

by rplnt 9y ago

If performance is your only motivation, I recommend you have a look at cython (previously pyrex). It basically makes writing C modules a breeze for a python programmer because you write them in type annotated python. Regular python is transpiled into mess of python calls, your typed tight loop will be "raw" C.

dec0dedab0de 9y ago
cython (previously pyrex) Ive never used either outside of a toy implementation, but Cython is based on Pyrex. They are different projects.
3pt14159 9y ago
I've been using cython for years, it is indeed great. I don't recommend anyone write multi-platform raw C any more unless you have a really good reason to. You can even make C-extensions for things like Ruby! But I have a new project now that needs to be both highly concurrent as well as reasonably fast and I'm considering giving Numba a try since it seems like it's both faster and easier to work with. Does anyone with experience with both have a comment as to the downsides of Numba vs Cython?
- jacobush 9y ago
  Is it easy to compile standalone executables with Cython? Or do you always have to drag around the full(-ish) Python environment? For me, shedskin was the sweet spot on paper, unfortunately it doesn't work very well - I can never get its output to compile properly. Also it's Python 2.x only.
  - dagw 9y ago
    https://github.com/cython/cython/wiki/EmbeddingCython https://github.com/cython/cython/wiki/EmbeddingCython
- syllogism 9y ago
  If you're trying to rescue existing code with type annotations, Numba can be a good solution. I think Cython has a better build story for shipping libraries though. I think users need Numba if you use it -- that's not true with Cython. If you're writing fresh code, I think Cython is much better. Numba doesn't really give you a language to work in. With Cython you can basically write C or C++ --- you can declare structs, manage pointers, even have C++ classes with templates if you want. This hash table does similar work to Google's dense_hash_map, with similar performance: https://github.com/explosion/preshed/tree/master/preshed https://github.com/explosion/preshed/tree/master/preshed . I think it's hard to write something like this with Numba.
- dagw 9y ago
  since it seems like it's both faster and easier to work with. I'm not sure I agree with the statement that it's strictly easier to work with. One big problem with Numba is that it is very unpredictable performance wise. Sometimes it does an amazing job speeding up your function and sometimes it just doesn't, and I find it very difficult to guess a priori how Numba will behave. It's also not always obvious how to rewrite code that Numba doesn't like to code that Numba likes. Cython on the other hand is very easy to reason about. The way it compiles your code to C is obvious and consistent and assuming you have basic familiarity with C it's very easy to reason about how a certain piece of code will perform after being compiled to Cython. Basically when Numba is at its best behavior it is much easier to work with than Cython, but when it isn't it's much harder to deal with. Cython gives consistent and reliable performance every time.
- shalabhc 9y ago
  Haven't used Numba extensively, but Cython is pretty much 100% Python compatible, while Numba only supports a small subset of types and operations. Many numerical and scientific problems may lie completely within that subset though. For general purpose stuff, Numba isn't there yet.
dbrgn 9y ago
Alternatively, in the meantime there are also really great Python bindings for Rust: https://pyo3.github.io/pyo3/guide/ https://pyo3.github.io/pyo3/guide/ The developer has full access to the Python environment while staying away from memory safety issues and the typical FFI hassle. There's also a Python package for integration with setuptools.
DonbunEf7 9y ago
Alternatively, if you prefer sanity and readability, use cffi; it's pretty fast, especially on PyPy, and frees users from the CPython C API.

by rplnt 9y ago

dec0dedab0de 9y ago

cython (previously pyrex) Ive never used either outside of a toy implementation, but Cython is based on Pyrex. They are different projects.

3pt14159 9y ago

I've been using cython for years, it is indeed great. I don't recommend anyone write multi-platform raw C any more unless you have a really good reason to. You can even make C-extensions for things like Ruby! But I have a new project now that needs to be both highly concurrent as well as reasonably fast and I'm considering giving Numba a try since it seems like it's both faster and easier to work with. Does anyone with experience with both have a comment as to the downsides of Numba vs Cython?

jacobush 9y ago
Is it easy to compile standalone executables with Cython? Or do you always have to drag around the full(-ish) Python environment? For me, shedskin was the sweet spot on paper, unfortunately it doesn't work very well - I can never get its output to compile properly. Also it's Python 2.x only.
- dagw 9y ago
  https://github.com/cython/cython/wiki/EmbeddingCython https://github.com/cython/cython/wiki/EmbeddingCython
syllogism 9y ago
If you're trying to rescue existing code with type annotations, Numba can be a good solution. I think Cython has a better build story for shipping libraries though. I think users need Numba if you use it -- that's not true with Cython. If you're writing fresh code, I think Cython is much better. Numba doesn't really give you a language to work in. With Cython you can basically write C or C++ --- you can declare structs, manage pointers, even have C++ classes with templates if you want. This hash table does similar work to Google's dense_hash_map, with similar performance: https://github.com/explosion/preshed/tree/master/preshed https://github.com/explosion/preshed/tree/master/preshed . I think it's hard to write something like this with Numba.
dagw 9y ago
since it seems like it's both faster and easier to work with. I'm not sure I agree with the statement that it's strictly easier to work with. One big problem with Numba is that it is very unpredictable performance wise. Sometimes it does an amazing job speeding up your function and sometimes it just doesn't, and I find it very difficult to guess a priori how Numba will behave. It's also not always obvious how to rewrite code that Numba doesn't like to code that Numba likes. Cython on the other hand is very easy to reason about. The way it compiles your code to C is obvious and consistent and assuming you have basic familiarity with C it's very easy to reason about how a certain piece of code will perform after being compiled to Cython. Basically when Numba is at its best behavior it is much easier to work with than Cython, but when it isn't it's much harder to deal with. Cython gives consistent and reliable performance every time.
shalabhc 9y ago
Haven't used Numba extensively, but Cython is pretty much 100% Python compatible, while Numba only supports a small subset of types and operations. Many numerical and scientific problems may lie completely within that subset though. For general purpose stuff, Numba isn't there yet.

dbrgn 9y ago

Alternatively, in the meantime there are also really great Python bindings for Rust: https://pyo3.github.io/pyo3/guide/ https://pyo3.github.io/pyo3/guide/ The developer has full access to the Python environment while staying away from memory safety issues and the typical FFI hassle. There's also a Python package for integration with setuptools.

DonbunEf7 9y ago

Alternatively, if you prefer sanity and readability, use cffi; it's pretty fast, especially on PyPy, and frees users from the CPython C API.