12 ms·

Cython 3.0 Released

221 points by ngoldbaum 3y ago

westurner 3y ago
From https://cython.readthedocs.io/en/latest/src/changes.html#compatibility-with-python https://cython.readthedocs.io/en/latest/src/changes.html#com... : > Since Cython 3.0.0 started development, CPython 3.8-3.11 were released. All these are supported in Cython, including experimental support for the in-development CPython 3.12
navels 3y ago
And chaos ensued. Unable to install aws-cli via pip: https://github.com/aws/aws-cli/issues/8036 https://github.com/aws/aws-cli/issues/8036
- x3n0ph3n3 3y ago
  Mostly because of PyYAML: https://github.com/yaml/pyyaml/issues/724 https://github.com/yaml/pyyaml/issues/724 I was pretty annoyed to discover that it had a separate build-time dependency on cython that you can't control from your requirements.txt.
  - jborean93 3y ago
    That's unfortunately a pip limitation. You can either disable build isolation so that the build requirements are using your current environment or you can use the PIP_CONSTRAINTS env var which will be passed through to the isolated build invocation allowing you to constrain the build requirements. In this case PyYAML did release a 6.0.1 version which contains an upper bound constraint on Cython.
    - raverbashing 3y ago
      Wonder if Cython didn't think of checking with their biggest "customers" for compatibility
      - roblabla 3y ago
        PyYAML knew about the breakage since january 2022[0], and nothing really happened. After a year and a half with lots of alphas and betas, I don't think there is much cython could do, short of fixing PyYAML themselves. [0]: https://github.com/yaml/pyyaml/issues/601 https://github.com/yaml/pyyaml/issues/601
    - x3n0ph3n3 3y ago
      That's what I ended up doing just for the pyyaml installation.
  - stabbles 3y ago
    In Spack [1] we can express all these constraints for the dependency solver, and we also try to always re-cythonize sources [2]. The latter is because bundled cythonized files are sometimes forward incompatible with Python, so it's better to just regenerate those with an up to date cython. [1] https://github.com/spack/spack/ https://github.com/spack/spack/ [2] https://github.com/spack/spack/pull/35995 https://github.com/spack/spack/pull/35995
    - x3n0ph3n3 3y ago
      Oh great, yet _another_ Python package manager.
      - zincfingers 3y ago
        Spack is a lot more than just python package manager :)
      - cowsandmilk 3y ago
        It’s not a python package manager. It is a generalized package manager written in python.
    - gyrovagueGeist 3y ago
      Neat! I had no idea that Spack can be used for cythonize dependencies. I use it to manage my local compiler/CUDA/Trilinos stack but never considered it for non C++ things.
    - jborean93 3y ago
      That's what PyYAML does as well. It uses PEP 518 [1] to specify the dependencies for the build which PyYAML has included Cython [2]. It's just that for previous releases there was no upper bound here so pip and other tools just selected the latest version which was incompatible. In the past PyYAML included the cythonised .c files in the sdist but as of 5.4.0 they went the PEP 518 route and ensures the client will cythonise them if installing from the sdist. [1] - https://peps.python.org/pep-0518/ https://peps.python.org/pep-0518/ [2] - https://github.com/yaml/pyyaml/blob/release/6.0/pyproject.toml#L2 https://github.com/yaml/pyyaml/blob/release/6.0/pyproject.to...
  - formerly_proven 3y ago
    I like how the log in the issue also has another warning for breaking changes in it The license_file parameter is deprecated, use license_files instead. By 2023-Oct-30, you need to update your project and remove deprecated calls or your builds will no longer be supported. See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details. The level of churn and breakage in these basal python packaging packages is honestly insane.
- gchamonlive 3y ago
  I am reluctant to refer you to aws-cli V2 because it is way more involved to install. No idea why aws moved away from their original distribution method... However, the V2 is going to be the way to go moving forwards :/
  - garblegarble 3y ago
    >No idea why aws moved away from their original distribution method Didn't they move away exactly because of issues like this? As I understand it, aws-cli v2 is still written in Python but ships everything it needs self-contained so it can avoid Python dependency hell[1] 1: https://www.youtube.com/watch?v=U5y7JI_mHk8#t=3m40 https://www.youtube.com/watch?v=U5y7JI_mHk8#t=3m40
    - gchamonlive 3y ago
      Indeed!
    - PartiallyTyped 3y ago
      Funnily enough, I didn’t have issues with Python’s dependency hell before working for AWS, not because we have more dependencies, but because… reasons.
      - RandomThrow321 3y ago
        Is it special files that begin with a capital letter, version sets, version filter instances, and massive symlink farms?
        PartiallyTyped 3y ago
        If you already know that much then you don’t need a response ;)
  - takeda 3y ago
    cat -n shell.nix { pkgs ? import <nixpkgs> {} }: pkgs.mkShell { name = "dev-env"; buildInputs = with pkgs; [ awscli2 ]; } And then you can have it available when entering shell with nix-shell.
- BiteCode_dev 3y ago
  And this is why you pin your dependencies + never install the latest version of anything.
  - stabbles 3y ago
    The lesson here is different: package developers must put upperbounds on major versions of dependencies that follow semver.
    - pdhborges 3y ago
      Should this be the take away? It looks to me that people are building pipelines without taking into account that they change roles every time they have to build a package, from a package consumer to a package packager. Even if the package developers set an upper bound on the build dependency it is the packagers responsibility to provide a deterministic build environment.
mturk 3y ago
A project I work on is currently working through a bunch of performance issues that arose because we had inlined functions in `pxd` files that we did not declare `noexcept nogil`. So, heads up if you see similar regressions! We saw a ridiculous slowdown (7s to 100m, for one suite) because they were inside some awfully tight loops. But the fix was pretty straightforward once we worked it out.
- gyrovagueGeist 3y ago
  Yep, my yesterday was also filled with figuring out this issue and changing these annotations. Heads up that the default exception forwarding also breaks function pointer types if you share them with the wrapped C code.
Waterluvian 3y ago
The release of this pretty much hosed my morning at work. Suddenly CI wouldn’t build because this broke things in pyyaml and awscli. And then I got to re-learn how very bad Poetry and python dependency management is. I’m trying to update pyyaml to 6.0.1 and Poetry systematically downloaded every single major, minor, patch version of another dependency in trying to find which would fit the version constraints. Took half an hour.
- gaganyaan 3y ago
  If you're using poetry, why not just add a `cython = "<3.0"` to your pyproject.toml file? You can lock down subdependency versions like that, and presumably the versions of everything that worked yesterday still work today.
  - Waterluvian 3y ago
    It’s not part of my dependencies and it wasn’t on my radar. What I learned was that pyyaml had a new version that fixed the error being thrown by pyyaml.
  - aeyes 3y ago
    If a top level dependency has build time dependencies then those do not respect the version locks of your pyproject.toml unless you disable build isolation.
- nextlevelwizard 3y ago
  Why are you blindly upgrading major versions of your tools? Sounds like fundamental design issue more than anything.
  - globular-toast 3y ago
    You'd hope people would learn this time, but they'll probably just keep blaming their tools...
  - gyrovagueGeist 3y ago
gigatexal 3y ago
This broke a lot of stuff for us at work today. Building the wheels or something I can’t remember. The fix was forcing pip to use not the 3.x and use something older I think.
- gyrovagueGeist 3y ago
  Just so anyone who reads this has the version: cython<=0.29.36 is the most recent non 3.0.0 version.
- esafak 3y ago
  As always: pin your dependencies.
  - unpythonic 3y ago
    Sadly, it's not always that simple. If one of your well-pinned packages has a build dependency which is not correctly pinned, then you're subject to the problem.
    - esafak 3y ago
      You raise a great point. I believe cached builds, say with Nix, are the way to go. What do you do?
      - esafak 3y ago
        Speak of the devil: https://news.ycombinator.com/item?id=36805730 https://news.ycombinator.com/item?id=36805730
- physicsguy 3y ago
  Cython did everything right here since it used a major version number to signify the change. Other projets not pinning their dependencies is not their fault!
  - lazka 3y ago
zaps 3y ago
Comments confirmed—Python packaging is broken
ggm 3y ago
I get that if I adorn my python3 with types the right way, cython can uplift C betterer, but if I don't adorn my python3 and only use core imports (no pip) do I get any benefit? I have some humongous (for me) dict lookups I am doing on strings, counting in ints, and if I could 2x or 3x faster without having to recode I'd love it. The Cython web is directed at people willing to recode, I am ultimately willing but I wish there was some low hanging fruit evidence: is this even plausibly going to get faster or is dict() walk cost just "it is what it is" ? Oddly, it looks like across pickle dump/load I get some improvement. Does pickle restoring consume less memory than raw-made?
- gyrovagueGeist 3y ago
  You will ~likely get some improvement by directly putting your python code in a .pyx file and compiling it (without using any cdef functions). In most cases it will be as if you were calling the Python C API directly in a compiled C file. In my experience, the biggest difference will be a reduction in the function call overhead so it is better for tight loops. I wouldn't expect a full 2x in every case. But it can also be very significant in cases where Cython deduces the type of local variables.
  - ngoldbaum 3y ago
    Cython 3 also uses python type annotaions to infer C types now too, so if you add python types you might see surprising cython speedups.
    - ggm 3y ago
      Coming back to this thread, I am running one now. It's too early to say if there's a speedup. But I want to note it is ridiculously under-documented how to make what I would call a functional a.out from this toolset. You basically are assumed to run python3 >>> import 'mything' and then run mything.main(args) instead of getting a functional commandline tool which "just runs" -the stepover is small, true, but I would have thought the PRIMARY use case here is commandline executable, not "embed .so in my interpreter" There are a tonne of stack overflow "how do I" and all of them are really whacked: everyone trips over the same problems of :main undefined and a swag of other problems trying to call (g)cc themselves.
bippingchip 3y ago
This is a somewhat tangential question to the new release, but there might be folks here that can answer this question. Having used swig to create Python bindings for C++ code over 10 years ago, what’s the recommended way to do this in 2023? There’s still swig, there’s Cython, pybind11 and a few others. I do know swig and how complicated it can get with footguns abound when things grow more complex. Is Cython the way to go? How does it hold up to the alternatives? Google search gives many articles on the topic, but many typical SEO optimized low-value, and those that do show a bit of depth, focus on the basic mechanics, not really on things hold up for larger projects…
- plonk 3y ago
  The problem with SWIG bindings I’ve used is that they don’t have any type hints. They also don’t offer context managers to handle resources, so it’s a pain to use safely in Python. From the user POV, the best bindings I’ve seen were wrappers with a Python API that calls C++ using Cython.
- gyrovagueGeist 3y ago
  Cython is the most general tool. It can be used to do anything from making bindings from C/Cpp to Python, Python to C/Cpp, to writing compiled “Python-like” code in an intermediate layer that can be used for managing your wrappers or just writing performant code. If you just want the ability to provide a Python interface to a C/Cpp library PyBind11 will get you there in fewer LoC than Cython. Nanobind is an even lighter weight option. I’ve heard Swig is a pain to use.
  - bippingchip 3y ago
    Thank you! Swig indeed can be a pain but having used it before I have become somewhat blind to it. But eg smart pointers are not easy to deal with well, I’ve found out recently… I’ll have a look at pybind11. I’ve worked on Cython codebases too, which indeed allows to really nicely compile Python code and interact with c code. It does get weird when using eg pyqt and native qt…
- dagw 3y ago
itamarst 3y ago
Cython is really great at a small scale, but it has issues scaling up to large codebases (of Cython), where I think it's an anti-pattern. None of these are the fault of the Cython creators, it's really an extremely useful tool, it's just inherent in the design space, the things that make it so cool (transparently mix C and Python!) come with trade-offs. 1. Memory unsafety. It's still C or C++ in the end, the more you shift your code in that direction the easier it is to screw up. 2. Two compiler passes: first Cython->C, then C->machine code. This means some errors only get caught in second pass, when it's much harder to match back to the original code. Extra bad when using C++. Perhaps Cython 3 made this better, but it's a very hard problem to solve. 3. Lack of tooling. IDE support, linting, autoformatting... it's all much less extensive than alternatives. 4. Python only. Polars is written in Rust, so you can use it in Rust and Python, and there's work on JavaScript and R bindings. Large Cython code bases are Python only, which makes them less useful. Long version: https://pythonspeed.com/articles/cython-limitations/ https://pythonspeed.com/articles/cython-limitations/
- gyrovagueGeist 3y ago
  The tooling support isn’t ideal, but there are jump to definition extensions for Cython in vscode.
earthnail 3y ago
Amazing to read that most comments here are about broken build dependencies. I completely get the frustration, but somewhat sad to see that there's not that much talk about the actual improvements. There seem to be a lot of cool things in this. For example, semantics for division, power operator, print, classes, types and subscripting are now identical with Python 3. https://cython.readthedocs.io/en/latest/src/changes.html#improved-fidelity-to-python-semantics https://cython.readthedocs.io/en/latest/src/changes.html#imp... Improved interaction with numpy by moving that part of the code into numpy itself: https://cython.readthedocs.io/en/latest/src/changes.html#interaction-with-numpy https://cython.readthedocs.io/en/latest/src/changes.html#int... Improved support for const and volatile in C, and improved support for C++ expressions like std::move https://cython.readthedocs.io/en/latest/src/changes.html#id6 https://cython.readthedocs.io/en/latest/src/changes.html#id6 I love how the changelog also shows the bugfixes and improvements that each of these changes enabled. Hats off to the team behind this release.
ilayn 3y ago
This is probably one of the most anticipated release in the one recent times for all python ecosystem. A sincere thanks to handful (literally) of people who made this possible and despite all kinds of funding challenges. Nevermind the foo broke bar comments here.

221 points by ngoldbaum 3y ago

From https://cython.readthedocs.io/en/latest/src/changes.html#compatibility-with-python https://cython.readthedocs.io/en/latest/src/changes.html#com... : > Since Cython 3.0.0 started development, CPython 3.8-3.11 were released. All these are supported in Cython, including experimental support for the in-development CPython 3.12

navels 3y ago

And chaos ensued. Unable to install aws-cli via pip: https://github.com/aws/aws-cli/issues/8036 https://github.com/aws/aws-cli/issues/8036

x3n0ph3n3 3y ago
Mostly because of PyYAML: https://github.com/yaml/pyyaml/issues/724 https://github.com/yaml/pyyaml/issues/724 I was pretty annoyed to discover that it had a separate build-time dependency on cython that you can't control from your requirements.txt.
- jborean93 3y ago
  That's unfortunately a pip limitation. You can either disable build isolation so that the build requirements are using your current environment or you can use the PIP_CONSTRAINTS env var which will be passed through to the isolated build invocation allowing you to constrain the build requirements. In this case PyYAML did release a 6.0.1 version which contains an upper bound constraint on Cython.
  - raverbashing 3y ago
    Wonder if Cython didn't think of checking with their biggest "customers" for compatibility
    - roblabla 3y ago
      PyYAML knew about the breakage since january 2022[0], and nothing really happened. After a year and a half with lots of alphas and betas, I don't think there is much cython could do, short of fixing PyYAML themselves. [0]: https://github.com/yaml/pyyaml/issues/601 https://github.com/yaml/pyyaml/issues/601
  - x3n0ph3n3 3y ago
    That's what I ended up doing just for the pyyaml installation.
- stabbles 3y ago
  In Spack [1] we can express all these constraints for the dependency solver, and we also try to always re-cythonize sources [2]. The latter is because bundled cythonized files are sometimes forward incompatible with Python, so it's better to just regenerate those with an up to date cython. [1] https://github.com/spack/spack/ https://github.com/spack/spack/ [2] https://github.com/spack/spack/pull/35995 https://github.com/spack/spack/pull/35995
  - x3n0ph3n3 3y ago
    Oh great, yet _another_ Python package manager.
    - zincfingers 3y ago
      Spack is a lot more than just python package manager :)
    - cowsandmilk 3y ago
      It’s not a python package manager. It is a generalized package manager written in python.
  - gyrovagueGeist 3y ago
    Neat! I had no idea that Spack can be used for cythonize dependencies. I use it to manage my local compiler/CUDA/Trilinos stack but never considered it for non C++ things.
  - jborean93 3y ago
    That's what PyYAML does as well. It uses PEP 518 [1] to specify the dependencies for the build which PyYAML has included Cython [2]. It's just that for previous releases there was no upper bound here so pip and other tools just selected the latest version which was incompatible. In the past PyYAML included the cythonised .c files in the sdist but as of 5.4.0 they went the PEP 518 route and ensures the client will cythonise them if installing from the sdist. [1] - https://peps.python.org/pep-0518/ https://peps.python.org/pep-0518/ [2] - https://github.com/yaml/pyyaml/blob/release/6.0/pyproject.toml#L2 https://github.com/yaml/pyyaml/blob/release/6.0/pyproject.to...
- formerly_proven 3y ago
  I like how the log in the issue also has another warning for breaking changes in it The license_file parameter is deprecated, use license_files instead. By 2023-Oct-30, you need to update your project and remove deprecated calls or your builds will no longer be supported. See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details. The level of churn and breakage in these basal python packaging packages is honestly insane.
gchamonlive 3y ago
I am reluctant to refer you to aws-cli V2 because it is way more involved to install. No idea why aws moved away from their original distribution method... However, the V2 is going to be the way to go moving forwards :/
- garblegarble 3y ago
  >No idea why aws moved away from their original distribution method Didn't they move away exactly because of issues like this? As I understand it, aws-cli v2 is still written in Python but ships everything it needs self-contained so it can avoid Python dependency hell[1] 1: https://www.youtube.com/watch?v=U5y7JI_mHk8#t=3m40 https://www.youtube.com/watch?v=U5y7JI_mHk8#t=3m40
  - gchamonlive 3y ago
    Indeed!
  - PartiallyTyped 3y ago
    Funnily enough, I didn’t have issues with Python’s dependency hell before working for AWS, not because we have more dependencies, but because… reasons.
    - RandomThrow321 3y ago
      Is it special files that begin with a capital letter, version sets, version filter instances, and massive symlink farms?
      - PartiallyTyped 3y ago
        If you already know that much then you don’t need a response ;)
- takeda 3y ago
  cat -n shell.nix { pkgs ? import <nixpkgs> {} }: pkgs.mkShell { name = "dev-env"; buildInputs = with pkgs; [ awscli2 ]; } And then you can have it available when entering shell with nix-shell.
BiteCode_dev 3y ago
And this is why you pin your dependencies + never install the latest version of anything.
- stabbles 3y ago
  The lesson here is different: package developers must put upperbounds on major versions of dependencies that follow semver.
  - pdhborges 3y ago
    Should this be the take away? It looks to me that people are building pipelines without taking into account that they change roles every time they have to build a package, from a package consumer to a package packager. Even if the package developers set an upper bound on the build dependency it is the packagers responsibility to provide a deterministic build environment.

mturk 3y ago

A project I work on is currently working through a bunch of performance issues that arose because we had inlined functions in `pxd` files that we did not declare `noexcept nogil`. So, heads up if you see similar regressions! We saw a ridiculous slowdown (7s to 100m, for one suite) because they were inside some awfully tight loops. But the fix was pretty straightforward once we worked it out.

gyrovagueGeist 3y ago
Yep, my yesterday was also filled with figuring out this issue and changing these annotations. Heads up that the default exception forwarding also breaks function pointer types if you share them with the wrapped C code.

Waterluvian 3y ago

The release of this pretty much hosed my morning at work. Suddenly CI wouldn’t build because this broke things in pyyaml and awscli. And then I got to re-learn how very bad Poetry and python dependency management is. I’m trying to update pyyaml to 6.0.1 and Poetry systematically downloaded every single major, minor, patch version of another dependency in trying to find which would fit the version constraints. Took half an hour.

gaganyaan 3y ago
If you're using poetry, why not just add a `cython = "<3.0"` to your pyproject.toml file? You can lock down subdependency versions like that, and presumably the versions of everything that worked yesterday still work today.
- Waterluvian 3y ago
  It’s not part of my dependencies and it wasn’t on my radar. What I learned was that pyyaml had a new version that fixed the error being thrown by pyyaml.
- aeyes 3y ago
  If a top level dependency has build time dependencies then those do not respect the version locks of your pyproject.toml unless you disable build isolation.
nextlevelwizard 3y ago
Why are you blindly upgrading major versions of your tools? Sounds like fundamental design issue more than anything.
- globular-toast 3y ago
  You'd hope people would learn this time, but they'll probably just keep blaming their tools...
- gyrovagueGeist 3y ago

gigatexal 3y ago

This broke a lot of stuff for us at work today. Building the wheels or something I can’t remember. The fix was forcing pip to use not the 3.x and use something older I think.

gyrovagueGeist 3y ago
Just so anyone who reads this has the version: cython<=0.29.36 is the most recent non 3.0.0 version.
esafak 3y ago
As always: pin your dependencies.
- unpythonic 3y ago
  Sadly, it's not always that simple. If one of your well-pinned packages has a build dependency which is not correctly pinned, then you're subject to the problem.
  - esafak 3y ago
    You raise a great point. I believe cached builds, say with Nix, are the way to go. What do you do?
    - esafak 3y ago
      Speak of the devil: https://news.ycombinator.com/item?id=36805730 https://news.ycombinator.com/item?id=36805730
physicsguy 3y ago
Cython did everything right here since it used a major version number to signify the change. Other projets not pinning their dependencies is not their fault!
- lazka 3y ago

zaps 3y ago

Comments confirmed—Python packaging is broken

ggm 3y ago

I get that if I adorn my python3 with types the right way, cython can uplift C betterer, but if I don't adorn my python3 and only use core imports (no pip) do I get any benefit? I have some humongous (for me) dict lookups I am doing on strings, counting in ints, and if I could 2x or 3x faster without having to recode I'd love it. The Cython web is directed at people willing to recode, I am ultimately willing but I wish there was some low hanging fruit evidence: is this even plausibly going to get faster or is dict() walk cost just "it is what it is" ? Oddly, it looks like across pickle dump/load I get some improvement. Does pickle restoring consume less memory than raw-made?

gyrovagueGeist 3y ago
You will ~likely get some improvement by directly putting your python code in a .pyx file and compiling it (without using any cdef functions). In most cases it will be as if you were calling the Python C API directly in a compiled C file. In my experience, the biggest difference will be a reduction in the function call overhead so it is better for tight loops. I wouldn't expect a full 2x in every case. But it can also be very significant in cases where Cython deduces the type of local variables.
- ngoldbaum 3y ago
  Cython 3 also uses python type annotaions to infer C types now too, so if you add python types you might see surprising cython speedups.
  - ggm 3y ago
    Coming back to this thread, I am running one now. It's too early to say if there's a speedup. But I want to note it is ridiculously under-documented how to make what I would call a functional a.out from this toolset. You basically are assumed to run python3 >>> import 'mything' and then run mything.main(args) instead of getting a functional commandline tool which "just runs" -the stepover is small, true, but I would have thought the PRIMARY use case here is commandline executable, not "embed .so in my interpreter" There are a tonne of stack overflow "how do I" and all of them are really whacked: everyone trips over the same problems of :main undefined and a swag of other problems trying to call (g)cc themselves.

bippingchip 3y ago

This is a somewhat tangential question to the new release, but there might be folks here that can answer this question. Having used swig to create Python bindings for C++ code over 10 years ago, what’s the recommended way to do this in 2023? There’s still swig, there’s Cython, pybind11 and a few others. I do know swig and how complicated it can get with footguns abound when things grow more complex. Is Cython the way to go? How does it hold up to the alternatives? Google search gives many articles on the topic, but many typical SEO optimized low-value, and those that do show a bit of depth, focus on the basic mechanics, not really on things hold up for larger projects…

plonk 3y ago
The problem with SWIG bindings I’ve used is that they don’t have any type hints. They also don’t offer context managers to handle resources, so it’s a pain to use safely in Python. From the user POV, the best bindings I’ve seen were wrappers with a Python API that calls C++ using Cython.
gyrovagueGeist 3y ago
Cython is the most general tool. It can be used to do anything from making bindings from C/Cpp to Python, Python to C/Cpp, to writing compiled “Python-like” code in an intermediate layer that can be used for managing your wrappers or just writing performant code. If you just want the ability to provide a Python interface to a C/Cpp library PyBind11 will get you there in fewer LoC than Cython. Nanobind is an even lighter weight option. I’ve heard Swig is a pain to use.
- bippingchip 3y ago
  Thank you! Swig indeed can be a pain but having used it before I have become somewhat blind to it. But eg smart pointers are not easy to deal with well, I’ve found out recently… I’ll have a look at pybind11. I’ve worked on Cython codebases too, which indeed allows to really nicely compile Python code and interact with c code. It does get weird when using eg pyqt and native qt…
dagw 3y ago

itamarst 3y ago

Cython is really great at a small scale, but it has issues scaling up to large codebases (of Cython), where I think it's an anti-pattern. None of these are the fault of the Cython creators, it's really an extremely useful tool, it's just inherent in the design space, the things that make it so cool (transparently mix C and Python!) come with trade-offs. 1. Memory unsafety. It's still C or C++ in the end, the more you shift your code in that direction the easier it is to screw up. 2. Two compiler passes: first Cython->C, then C->machine code. This means some errors only get caught in second pass, when it's much harder to match back to the original code. Extra bad when using C++. Perhaps Cython 3 made this better, but it's a very hard problem to solve. 3. Lack of tooling. IDE support, linting, autoformatting... it's all much less extensive than alternatives. 4. Python only. Polars is written in Rust, so you can use it in Rust and Python, and there's work on JavaScript and R bindings. Large Cython code bases are Python only, which makes them less useful. Long version: https://pythonspeed.com/articles/cython-limitations/ https://pythonspeed.com/articles/cython-limitations/

gyrovagueGeist 3y ago
The tooling support isn’t ideal, but there are jump to definition extensions for Cython in vscode.

earthnail 3y ago

Amazing to read that most comments here are about broken build dependencies. I completely get the frustration, but somewhat sad to see that there's not that much talk about the actual improvements. There seem to be a lot of cool things in this. For example, semantics for division, power operator, print, classes, types and subscripting are now identical with Python 3. https://cython.readthedocs.io/en/latest/src/changes.html#improved-fidelity-to-python-semantics https://cython.readthedocs.io/en/latest/src/changes.html#imp... Improved interaction with numpy by moving that part of the code into numpy itself: https://cython.readthedocs.io/en/latest/src/changes.html#interaction-with-numpy https://cython.readthedocs.io/en/latest/src/changes.html#int... Improved support for const and volatile in C, and improved support for C++ expressions like std::move https://cython.readthedocs.io/en/latest/src/changes.html#id6 https://cython.readthedocs.io/en/latest/src/changes.html#id6 I love how the changelog also shows the bugfixes and improvements that each of these changes enabled. Hats off to the team behind this release.

ilayn 3y ago

This is probably one of the most anticipated release in the one recent times for all python ecosystem. A sincere thanks to handful (literally) of people who made this possible and despite all kinds of funding challenges. Nevermind the foo broke bar comments here.