4 ms·

> Code that needs performance and would benefit from multithreading, usually written by professional software engineers, likely isn't written in Python in the f

> Code that needs performance and would benefit from multithreading, usually written by professional software engineers, likely isn't written in Python in the first place. There are a lot of simple cases where multi-threading can easily triple or quadruple the performance.

Certhas 2y ago
But multiprocessing can't? I used to write a ton of MPI based parallel python. It's pretty straightforward. But one could easily imagine trying to improve the multiprocessing ergonomics rather than introducing threading. Obviously the people who made the choice to push forward with this are aware of these options, too. Still mildly puzzling to me why threads for Python are needed/reasonable.
- pdhborges 2y ago
  It doesn't to be puzzling just read the motivation section of https://peps.python.org/pep-0703/ https://peps.python.org/pep-0703/
  - Certhas 2y ago
    I know that document. It doesn't really answer this point though. It motivates the need for parallelism in slow python by noting that this is important once other performance critical code is in extensions. But the mai point against multiprocessing seems to be that spawning new processes is slow ... That single "alternatives" paragraph doesn't answer at all why mp isn't viable for python level parallelism. I am no longer invested in python heavily, I am sure there are discussions or documents somewhere that go into this more. Might be that it's simply that everyone is used to threads so you should support it for sheer familiarity. All I am saying is it's not obvious to a casual observer.
- bdd8f1df777b 2y ago
  A common pain point for me is parallel data loading and processing for PyTorch or TensorFlow. Multiprocessing has a lot of bugs and pain points to deal with when doing ML. Examples: https://github.com/search?q=repo%3Apytorch%2Fpytorch+multiprocessing&type=issues https://github.com/search?q=repo%3Apytorch%2Fpytorch+multipr.... Most of these issues do not exist in a multithreading world, because resource sharing is trivial in that case. Since Python leads over any other languages in the ML community, and ML is a hot topic right now, it makes sense for Python developers to secure the lead by making the life of ML developers easier, which is by introducing GIL-less multi-threading.

Certhas 2y ago

But multiprocessing can't? I used to write a ton of MPI based parallel python. It's pretty straightforward. But one could easily imagine trying to improve the multiprocessing ergonomics rather than introducing threading. Obviously the people who made the choice to push forward with this are aware of these options, too. Still mildly puzzling to me why threads for Python are needed/reasonable.

pdhborges 2y ago
It doesn't to be puzzling just read the motivation section of https://peps.python.org/pep-0703/ https://peps.python.org/pep-0703/
- Certhas 2y ago
  I know that document. It doesn't really answer this point though. It motivates the need for parallelism in slow python by noting that this is important once other performance critical code is in extensions. But the mai point against multiprocessing seems to be that spawning new processes is slow ... That single "alternatives" paragraph doesn't answer at all why mp isn't viable for python level parallelism. I am no longer invested in python heavily, I am sure there are discussions or documents somewhere that go into this more. Might be that it's simply that everyone is used to threads so you should support it for sheer familiarity. All I am saying is it's not obvious to a casual observer.
bdd8f1df777b 2y ago
A common pain point for me is parallel data loading and processing for PyTorch or TensorFlow. Multiprocessing has a lot of bugs and pain points to deal with when doing ML. Examples: https://github.com/search?q=repo%3Apytorch%2Fpytorch+multiprocessing&type=issues https://github.com/search?q=repo%3Apytorch%2Fpytorch+multipr.... Most of these issues do not exist in a multithreading world, because resource sharing is trivial in that case. Since Python leads over any other languages in the ML community, and ML is a hot topic right now, it makes sense for Python developers to secure the lead by making the life of ML developers easier, which is by introducing GIL-less multi-threading.