2 ms·

My understanding is that Cython does not run unmodified standard Python code.

by chrisseaton 5y ago

jonatron 5y ago
It can run most unmodified Python code, but it needs types for the big speedups.
- chrisseaton 5y ago
  For example here's a Python project modified to work well on Cython - I think that's pretty substantially modified. https://github.com/jameskmurphy/nes/tree/main/nes/cycore https://github.com/jameskmurphy/nes/tree/main/nes/cycore It's more like a DSL for an FFI.
  - jonatron 5y ago
    And here's a project that's mostly Python, and optionally uses Cython https://github.com/falconry/falcon https://github.com/falconry/falcon
- DasIch 5y ago
  That's misleading as normal type annotations don't give you any performance benefit at all and you may need to do a lot more than just add type annotations. I think it's far more accurate to say that Cython is a programming language of its own that is a hybrid of Python and C++, that happens to produce CPython extension modules when compiled. The performance benefits are really achieved by incrementally changing your Python code to something that looks a lot more like C(++). This is also reflected in the Cython documentation which literally mentions the "Cython language". Cython is great as an alternative to writing C extension modules for performance reasons or to creating bindings to libraries written in C or C++. It's not so great just to make Python applications faster as it's not fully compatible[1]. [1]: https://cython.readthedocs.io/en/latest/src/userguide/limitations.html https://cython.readthedocs.io/en/latest/src/userguide/limita...
  - BBC-vs-neolibs 5y ago
    "This page used to list bugs in Cython that made the semantics of compiled code differ from that in Python. Most of the missing features have been fixed in Cython 0.15. A future version of Cython is planned to provide full Python language compatibility."
  - shoo 5y ago
    > The performance benefits are really achieved by incrementally changing your Python code to something that looks a lot more like C(++) I completely agree. Cython can become quite attractive if your alternative is "write a python extension library by hand in C / C++". I first started using Cython after doing exactly that, writing my extension library in C, then realising that Cython might save a lot of work in generating the bindings and packaging/distribution -- it did, and it ran at exactly the same speed as my pure C library with hand crafted python bindings. After that I've been pretty excited about Cython. If you've got a python program that needs to do a core of compute-heavy work, if you were to optimise this by writing a C / C++ library for python to use, the work would be: (i) think hard about how the library design will enable performance, (ii) implement that high performance library in C / C++ , (iii) figure out the interface so that python can call into the library, and (iv) figure out how to package and distribute the library so it can be used by python programs. Cython doesn't really help with parts (i) designing for performance or (ii) writing that high performance code. But it helps a lot with parts (iii) and (iv), generating Python bindings and producing wheel archives that can be managed by existing python package management tooling.
nerdponx 5y ago
It usually does work on individual small functions, in my experience. I've seen 2x improvement on string processing code (cleaning text for input to a ML model) by doing nothing other than sticking `%%cython` at the top of a notebook cell. I don't know if this technique scales to other applications; probably not. But Cython syntactically is a superset of Python.
deleted 5y ago
[deleted]
yesenadam 5y ago
I mainly programmed in Cython for a few years, and never came across standard Python code it wouldn't run. It's just usually only a few times faster than Python using unmodified Python, as opposed to 50x-200x faster or more with type annotations, GIL turned off for bottleneck functions, @ decorators etc. I loved it, being able to float between Python and C in a program, writing in Python whatever was more convenient in Python, in C whatever was more convenient in C, or anywhere in between. "While pure Python scripts can be compiled with Cython, it usually results only in a speed gain of about 20%-50%." https://cython.readthedocs.io/en/latest/src/tutorial/pure.html https://cython.readthedocs.io/en/latest/src/tutorial/pure.ht...

My understanding is that Cython does not run unmodified standard Python code.

jonatron 5y ago

It can run most unmodified Python code, but it needs types for the big speedups.

chrisseaton 5y ago
For example here's a Python project modified to work well on Cython - I think that's pretty substantially modified. https://github.com/jameskmurphy/nes/tree/main/nes/cycore https://github.com/jameskmurphy/nes/tree/main/nes/cycore It's more like a DSL for an FFI.
- jonatron 5y ago
  And here's a project that's mostly Python, and optionally uses Cython https://github.com/falconry/falcon https://github.com/falconry/falcon
DasIch 5y ago
That's misleading as normal type annotations don't give you any performance benefit at all and you may need to do a lot more than just add type annotations. I think it's far more accurate to say that Cython is a programming language of its own that is a hybrid of Python and C++, that happens to produce CPython extension modules when compiled. The performance benefits are really achieved by incrementally changing your Python code to something that looks a lot more like C(++). This is also reflected in the Cython documentation which literally mentions the "Cython language". Cython is great as an alternative to writing C extension modules for performance reasons or to creating bindings to libraries written in C or C++. It's not so great just to make Python applications faster as it's not fully compatible[1]. [1]: https://cython.readthedocs.io/en/latest/src/userguide/limitations.html https://cython.readthedocs.io/en/latest/src/userguide/limita...
- BBC-vs-neolibs 5y ago
  "This page used to list bugs in Cython that made the semantics of compiled code differ from that in Python. Most of the missing features have been fixed in Cython 0.15. A future version of Cython is planned to provide full Python language compatibility."
- shoo 5y ago
  > The performance benefits are really achieved by incrementally changing your Python code to something that looks a lot more like C(++) I completely agree. Cython can become quite attractive if your alternative is "write a python extension library by hand in C / C++". I first started using Cython after doing exactly that, writing my extension library in C, then realising that Cython might save a lot of work in generating the bindings and packaging/distribution -- it did, and it ran at exactly the same speed as my pure C library with hand crafted python bindings. After that I've been pretty excited about Cython. If you've got a python program that needs to do a core of compute-heavy work, if you were to optimise this by writing a C / C++ library for python to use, the work would be: (i) think hard about how the library design will enable performance, (ii) implement that high performance library in C / C++ , (iii) figure out the interface so that python can call into the library, and (iv) figure out how to package and distribute the library so it can be used by python programs. Cython doesn't really help with parts (i) designing for performance or (ii) writing that high performance code. But it helps a lot with parts (iii) and (iv), generating Python bindings and producing wheel archives that can be managed by existing python package management tooling.

nerdponx 5y ago

It usually does work on individual small functions, in my experience. I've seen 2x improvement on string processing code (cleaning text for input to a ML model) by doing nothing other than sticking `%%cython` at the top of a notebook cell. I don't know if this technique scales to other applications; probably not. But Cython syntactically is a superset of Python.

deleted 5y ago

[deleted]

yesenadam 5y ago

I mainly programmed in Cython for a few years, and never came across standard Python code it wouldn't run. It's just usually only a few times faster than Python using unmodified Python, as opposed to 50x-200x faster or more with type annotations, GIL turned off for bottleneck functions, @ decorators etc. I loved it, being able to float between Python and C in a program, writing in Python whatever was more convenient in Python, in C whatever was more convenient in C, or anywhere in between. "While pure Python scripts can be compiled with Cython, it usually results only in a speed gain of about 20%-50%." https://cython.readthedocs.io/en/latest/src/tutorial/pure.html https://cython.readthedocs.io/en/latest/src/tutorial/pure.ht...