4 ms·
Disabling the GIL can unlock true multi-core parallelism for multi-threaded programs, but this requires code to be restructured for safe concurrency, which isn'
by kjqgqkejbfefn 3y ago
Disabling the GIL can unlock true multi-core parallelism for multi-threaded programs, but this requires code to be restructured for safe concurrency, which isn't that difficult it seems:
> When we found out about the “nogil” fork of Python it took a single person less than half a working day to adjust the codebase to use this fork and the results were astonishing. Now we can focus on data acquisition system development rather than fine-tuning data exchange algorithms.
https://peps.python.org/pep-0703/ https://peps.python.org/pep-0703/
- kjqgqkejbfefn 3y ago>We frequently battle issues with the Python GIL at DeepMind. In many of our applications, we would like to run on the order of 50-100 threads per process. However, we often see that even with fewer than 10 threads the GIL becomes the bottleneck. To work around this problem, we sometimes use subprocesses, but in many cases the inter-process communication becomes too big of an overhead. To deal with the GIL, we usually end up translating large parts of our Python codebase into C++. This is undesirable because it makes the code less accessible to researchers.
- actionfromafar 3y agoMaybe they should look in to translating parts of their code base to Shedskin Python. It compiles (a subset of) Python to C++.
- logicchains 3y agoHow's it different from Cython, which compiles a subset of Python to C or C++?
- actionfromafar 3y agoShedskin has stricter typing, and about 10-100 times performance vs Cython.