3 ms·

https://hugovk.github.io/free-threaded-wheels/ https://hugovk.github.io/free-threaded-wheels/ is looking pretty healthy - 130 of the 360 most downloaded C exten

by simonw 1y ago

https://hugovk.github.io/free-threaded-wheels/ https://hugovk.github.io/free-threaded-wheels/ is looking pretty healthy - 130 of the 360 most downloaded C extension PyPI packages are now free-threaded Python compatible, up from 92 on 15th August https://web.archive.org/web/20250815071755/https://hugovk.github.io/free-threaded-wheels/ https://web.archive.org/web/20250815071755/https://hugovk.gi... I was curious as to how that site works - it has a build script at https://github.com/hugovk/free-threaded-wheels/blob/cdae0b45ab406e2609d01b84c379466fa64e0ce7/utils.py#L57 https://github.com/hugovk/free-threaded-wheels/blob/cdae0b45... which checks the PyPI available file downloads for a package and looks for a bdist_wheel that matches this: abi_tag = download["filename"].removesuffix(".whl").split("-")[-2] if abi_tag.endswith("t") and abi_tag.startswith("cp31"): has_free_threaded_wheel = True

rogerbinns 1y ago
Note that free threaded compatible doesn't necessarily mean the package supports free threading (concurrent execution), just that it can be loaded into a free threaded interpreter. This is the case with my own package which is on the hugovk list (apsw) which will cause the GIL to be re-enabled if you load it into a free threaded Python. The reason I provide a binary wheel is so that you don't have to keep separate GIL full and free threaded interpreters around. They have a different ABI so you can't use extensions compiled against one with the other. Free threading is at the beginning of its journey. There is a *lot* of work to on all C code that works with Python objects, and the current documentation and tools are immature. It is especially the case that anyone doing Python concurrent object mutation can cause corruption and crashes if they try, and that more auditing and locking need to be done in the C code. Even modules in the standard library have only been partially updated. You can see a lot details and discussion in the comments at https://news.ycombinator.com/item?id=45633311 https://news.ycombinator.com/item?id=45633311
- LtWorf 1y ago
  Why do you provide it at all then if it's not working as intended yet?
  - rogerbinns 1y ago
    As I stated: > so that you don't have to keep separate GIL full and free threaded interpreters around It means the user doesn't have to keep two Pythons around, install packages in both of them, etc. It is also possible with the free threaded Python to keep the GIL disabled even if a package such as mine says it needs the GIL. And my package will indeed work just fine, until you supply it with mutable data and concurrently modify it in another thread.
    - cyanydeez 1y ago
      Do you atleast warn users? This sounds like madness.
      - rogerbinns 1y ago
        Yes. The interpreter warns by default, and requires steps to disable the warning. My release notes say that the GIL will be enabled when the package is loaded. Is it madness that other packages claim they support running without the GIL, yet it is possible to cause corruption and crashes just by writing concurrent Python code? That is the case with the standard library. Compiler thread sanitizers don't work with free threaded Python. Diligent code inspection by humans is the only way to update C code so far. Free threading is at the beginning of the project. It works. You get slow downs in single threaded performance due to extra locking, and speedups in concurrent performance due to threading. But it is possible to cause corruption and crashes via Python code. Don't expose it to untrusted data and code. But do investigate it and see what works well and what doesn't. See what code patterns are now possible. Help improve the tools and documentation. It had to start somewhere, and the current state is somewhere.
        LtWorf 1y ago
        I think what you are doing is hiding problems. I think crashes and bugs are preferable to find the issues at this point. People who want the safe option will run the regular python.

by simonw 1y ago

rogerbinns 1y ago

Note that free threaded compatible doesn't necessarily mean the package supports free threading (concurrent execution), just that it can be loaded into a free threaded interpreter. This is the case with my own package which is on the hugovk list (apsw) which will cause the GIL to be re-enabled if you load it into a free threaded Python. The reason I provide a binary wheel is so that you don't have to keep separate GIL full and free threaded interpreters around. They have a different ABI so you can't use extensions compiled against one with the other. Free threading is at the beginning of its journey. There is a *lot* of work to on all C code that works with Python objects, and the current documentation and tools are immature. It is especially the case that anyone doing Python concurrent object mutation can cause corruption and crashes if they try, and that more auditing and locking need to be done in the C code. Even modules in the standard library have only been partially updated. You can see a lot details and discussion in the comments at https://news.ycombinator.com/item?id=45633311 https://news.ycombinator.com/item?id=45633311

LtWorf 1y ago
Why do you provide it at all then if it's not working as intended yet?
- rogerbinns 1y ago
  As I stated: > so that you don't have to keep separate GIL full and free threaded interpreters around It means the user doesn't have to keep two Pythons around, install packages in both of them, etc. It is also possible with the free threaded Python to keep the GIL disabled even if a package such as mine says it needs the GIL. And my package will indeed work just fine, until you supply it with mutable data and concurrently modify it in another thread.
  - cyanydeez 1y ago
    Do you atleast warn users? This sounds like madness.
    - rogerbinns 1y ago
      Yes. The interpreter warns by default, and requires steps to disable the warning. My release notes say that the GIL will be enabled when the package is loaded. Is it madness that other packages claim they support running without the GIL, yet it is possible to cause corruption and crashes just by writing concurrent Python code? That is the case with the standard library. Compiler thread sanitizers don't work with free threaded Python. Diligent code inspection by humans is the only way to update C code so far. Free threading is at the beginning of the project. It works. You get slow downs in single threaded performance due to extra locking, and speedups in concurrent performance due to threading. But it is possible to cause corruption and crashes via Python code. Don't expose it to untrusted data and code. But do investigate it and see what works well and what doesn't. See what code patterns are now possible. Help improve the tools and documentation. It had to start somewhere, and the current state is somewhere.
      - LtWorf 1y ago
        I think what you are doing is hiding problems. I think crashes and bugs are preferable to find the issues at this point. People who want the safe option will run the regular python.