4 ms·
> You can parallelize with threads! This is a great example of something that's more than the sum of its parts. A GIL-constrained language and a single-threade
by pansa2 5y ago
> You can parallelize with threads!
This is a great example of something that's more than the sum of its parts. A GIL-constrained language and a single-threaded language working together to run code in parallel - something that neither language can achieve on its own.
Although I believe there is a plan to add support for this pattern (multiple threads with one interpreter state each) to pure Python in the form of "subinterpreters".
- jjice 5y agoInteresting, how does that compare to the multiprocessing module python has now?
- O5vYtytb 5y agoMultiprocessing requires multiple python processes which are expensive with regards to startup time and resource use. Multiple interpreters give you parallelism without so much overhead.
- wruza 5y agoWait, isn’t python startup slow because of the interpreter startup and not process creation itself? A thread also has to be created (saves dll mapping time but that’s it) and python state also has to be initialized in a complete isolation on that thread (means setting up the entire scene again and loading stdlib from sources). Am I wrong?
- O5vYtytb 5y agoIt's a number of factors, including the loading of the modules necessary to start python and the standard library. Check out https://github.com/faster-cpython/ideas/issues?q=is%3Aissue+is%3Aopen+faster+startup https://github.com/faster-cpython/ideas/issues?q=is%3Aissue+... for some ideas regarding improvement there.
- maxnoe 5y agoThe point is each process has to start its own interpreter.
- wruza 5y agoThreads don’t magically start with a python interpreter that is ready to use, it has to be initialized for every thread out there separately as well, unless some clever immutability tricks are involved to skip big parts of that initialization.
- pansa2 5y agoAFAIK subinterpreters are similar to multiprocessing in that the interpreters don't share memory - but because they are within the same process, communication between them can be more efficient.