3 ms·

The other thing I don't get is that the whole sub interpreters thing seems to totally break extension modules as well: https://github.com/PyO3/pyo3/issues/2274

The other thing I don't get is that the whole sub interpreters thing seems to totally break extension modules as well: https://github.com/PyO3/pyo3/issues/2274 https://github.com/PyO3/pyo3/issues/2274. In theory parts of sub-interpreters have been around for a while and it just happens that every extension module out there is incompatible with it because no one used it. But if it's going to become the recommended way to do parallelism going forward then they'll have to become compatible with it. The serialization thing is also a huge issue. Half of the time I want to use multiprocessing I end up finding that the serialization of data is the bottleneck and have to somehow re-architect my code to minimize it. I would much prefer a world in which asyncio is 2x faster and can benefit from real parallelism across threads. Libraries like anyio already make it super easy to work with async + threads. It would make Python a viable option for workloads where it currently just isn't.

rogerbinns 3y ago
(Disclosure: Author of a Python C extension that wraps SQLite that has 2 decades of development behind it.) Have a look at the current documentation for writing extensions. This approach is essentially unchanged since Python 2.0. https://docs.python.org/3/extending/newtypes_tutorial.html https://docs.python.org/3/extending/newtypes_tutorial.html In particular note how everything is declared static - ie only one instance of that data item will exist. If there are multiple interpreters then there needs to be one instance per sub-interpreter. That means no more static and initialisation has to be changed to attach these to the module object which is then attached to a specific interpreter instance. It also means every location you needed to access a previously static item (which often happens) has to change from a direct reference through new APIs chasing back from objects to get their owning module and then get the reference. That is the code churn the PyO3 issue is having to address. One bonus however is that you can then cleanly unload modules. This may still not be sufficient. For example I wrap SQLite and it has some global items like a logging callback. If my module was loaded into two different sub interpreters and both registered the logging callback, only one would win. These kind of gnarly issues are hard to discover and diagnose. Removing the GIL also won't magically help. I already release it at every possible opportunity. If it did go away, I would have to reintroduce a lock anyway to prevent concurrency in certain places. And there would have to be more locking around various Python data structures. For example if I am processing items in a list, I'd need a lock to prevent the list changing while processing. Currently the GIL handles that and ensures fewer bugs. I've also experienced the serialization overhead with multiprocessing. I made a client's code so much faster that any form of Python concurrency was slower because of all the overhead. I had to rearchitect the code to work on batches of data items instead of the far more natural one at a time. That finally allowed a performance improvement with multiprocessing.

rogerbinns 3y ago

(Disclosure: Author of a Python C extension that wraps SQLite that has 2 decades of development behind it.) Have a look at the current documentation for writing extensions. This approach is essentially unchanged since Python 2.0. https://docs.python.org/3/extending/newtypes_tutorial.html https://docs.python.org/3/extending/newtypes_tutorial.html In particular note how everything is declared static - ie only one instance of that data item will exist. If there are multiple interpreters then there needs to be one instance per sub-interpreter. That means no more static and initialisation has to be changed to attach these to the module object which is then attached to a specific interpreter instance. It also means every location you needed to access a previously static item (which often happens) has to change from a direct reference through new APIs chasing back from objects to get their owning module and then get the reference. That is the code churn the PyO3 issue is having to address. One bonus however is that you can then cleanly unload modules. This may still not be sufficient. For example I wrap SQLite and it has some global items like a logging callback. If my module was loaded into two different sub interpreters and both registered the logging callback, only one would win. These kind of gnarly issues are hard to discover and diagnose. Removing the GIL also won't magically help. I already release it at every possible opportunity. If it did go away, I would have to reintroduce a lock anyway to prevent concurrency in certain places. And there would have to be more locking around various Python data structures. For example if I am processing items in a list, I'd need a lock to prevent the list changing while processing. Currently the GIL handles that and ensures fewer bugs. I've also experienced the serialization overhead with multiprocessing. I made a client's code so much faster that any form of Python concurrency was slower because of all the overhead. I had to rearchitect the code to work on batches of data items instead of the far more natural one at a time. That finally allowed a performance improvement with multiprocessing.