4 ms·

> The fact of the matter is that Python (already today) allows you to achieve parallelism across both IO-bound and CPU-bound workloads. I also don't need to us

> The fact of the matter is that Python (already today) allows you to achieve parallelism across both IO-bound and CPU-bound workloads. I also don't need to use goroutines. I could simply spin up my golang application as a couple of processes, and use pipes and other IPC to coordinate them. "There is another way to do X" doesn't imply that other way is better.

csmpltn 3y ago
> "doesn't imply that other way is better" I never said that multiprocessing is "better" than multithreading. Both are mechanisms to achieve parallelism with different pros/cons, and there are legit cases where threads are a necessity that can't be satisfied with processes (ref my previous comment with examples). This conversation would've been much easier to have given specific constraints and examples (which nobody seems to give). Within the context of a general purpose VM which was never built to support multithreading (CPython), and which carries the baggage of 30+ years' worth of 3rd-party packages and libraries which were never built with multithreading in-mind - I think we can both agree that the costs and risks of changing literally everything may outweigh the benefits. It's not a difficult stance to accept. My main argument, if you distill it down to the abstract, is "use the right tool for the job" - and stop pretending that a hammer and a knife are the same thing, when they're not. If you need hardcore, ultra efficient and parallelized workflows - Python is just the wrong tool for the job period. This isn't up for debate - it's a given fact. This isn't just about the GIL - it's about the type system, the frameworks, the bloat, etc. There's nothing wrong with Python being this way - Python fills a niche of its own incredibly well, something which others tools suck at, and Python is loved by millions for it. Python is used today by certain demographics to perform certain jobs - and it excels really well at those jobs for those demographics. This whole discussion now (GIL vs. no GIL) is about bending and twisting Python to fit the use cases of few companies like DeepMind or Facebook - who I'm certain represent a miniscule usage compared to the millions of students, schools and universities, research institutes, web shops, hobbyists, tinkerers, etc. Those people want to get shit done quickly - and mutexes, sempahores, events, threads, synchronization primitives, atomics, etc - will do nothing but make their lives a misery, and drive them away. Also, I keep asking you for concrete examples where the Python community absolutely needs multithreading (where multiprocessing fails) and you're not really responding which makes me feel like we're not conversing here...
- commonlisp94 3y ago
  I completely agree with this position. If you need an optimized paralyzed code to chew through a tough workload, write that part in C, and spawn it from python. Python is a coordinating scripting language.
- soulbadguy 3y ago
  > I never said that multiprocessing is "better" than multithreading. Maybe not, but you seem to be implying that multiprocessing is a sufficient for the most case (or the average case ?) however ill define that average case is. > This conversation would've been much easier to have given specific constraints and examples (which nobody seems to give). Not quite, there is no need grand example here, processes vs threads is just a function of how chatty the interaction between the independent agent are. We should have both mechanism available and let the user choose. > My main argument, if you distill it down to the abstract, is "use the right tool for the job" - and stop pretending that a hammer and a knife are the same thing, when they're not. You don't get to decide in the abstract what the right tool is for other people and context we don't know anything about. The people pushing for proper threading are telling you that for them python + better threading is the right tool. > If you need hardcore, ultra efficient and parallelized workflows - Python is just the wrong tool for the job period. This isn't up for debate - it's a given fact. Maybe; But that's not the point here. Proper threading in python doesn't make python more efficient, but it makes the * scaling * of python performance more efficient. Those are two differents things. > Python fills a niche of its own incredibly well, something which others tools suck at, and Python is loved by millions for it. > This whole discussion now (GIL vs. no GIL) is about bending and twisting Python to fit the use cases of few companies like DeepMind or Facebook > who I'm certain represent a miniscule usage compared to the millions of students, schools and universities, research institutes, web shops, hobbyists, tinkerers, etc. Those people want to get shit done quickly - and mutexes, sempahores, events, threads, synchronization primitives, atomics, etc - will do nothing but make their lives a misery, and drive them away. This seems to me like the core of the problem, somehow you say that there is no valid need for threading, but at the same time the people needing threading are not important enough for python to care about... You have the magical ability to be plug into the python hivemind and decide which use case are valid, what are the true value of the python community and what python should and should not consider important... And after all that mental gymnastic you expect us to jump through hoops to convince you otherwise... That's a lot of non technical assumption for a technical problem. > Those people want to get shit done quickly - and mutexes, sempahores, events, threads, synchronization primitives, atomics, etc - will do nothing but make their lives a misery, and drive them away. Not really. If anything, they will benefits from better libraries and more efficient C-extensions. > absolutely needs multithreading Nothing is absolutely needed. Even something are core as a type system is not absolutely needed. The fact that you ask for this level of qualification for a feature that pretty much every other language has is the problem. The problem is not that deep, implementing multi-threading is not that complicated. I do agree that we need to be careful to provide a better API than raw-thread to end users (which no-gil is COMPLETLY agnostic about).

csmpltn 3y ago

> "doesn't imply that other way is better" I never said that multiprocessing is "better" than multithreading. Both are mechanisms to achieve parallelism with different pros/cons, and there are legit cases where threads are a necessity that can't be satisfied with processes (ref my previous comment with examples). This conversation would've been much easier to have given specific constraints and examples (which nobody seems to give). Within the context of a general purpose VM which was never built to support multithreading (CPython), and which carries the baggage of 30+ years' worth of 3rd-party packages and libraries which were never built with multithreading in-mind - I think we can both agree that the costs and risks of changing literally everything may outweigh the benefits. It's not a difficult stance to accept. My main argument, if you distill it down to the abstract, is "use the right tool for the job" - and stop pretending that a hammer and a knife are the same thing, when they're not. If you need hardcore, ultra efficient and parallelized workflows - Python is just the wrong tool for the job period. This isn't up for debate - it's a given fact. This isn't just about the GIL - it's about the type system, the frameworks, the bloat, etc. There's nothing wrong with Python being this way - Python fills a niche of its own incredibly well, something which others tools suck at, and Python is loved by millions for it. Python is used today by certain demographics to perform certain jobs - and it excels really well at those jobs for those demographics. This whole discussion now (GIL vs. no GIL) is about bending and twisting Python to fit the use cases of few companies like DeepMind or Facebook - who I'm certain represent a miniscule usage compared to the millions of students, schools and universities, research institutes, web shops, hobbyists, tinkerers, etc. Those people want to get shit done quickly - and mutexes, sempahores, events, threads, synchronization primitives, atomics, etc - will do nothing but make their lives a misery, and drive them away. Also, I keep asking you for concrete examples where the Python community absolutely needs multithreading (where multiprocessing fails) and you're not really responding which makes me feel like we're not conversing here...

commonlisp94 3y ago
I completely agree with this position. If you need an optimized paralyzed code to chew through a tough workload, write that part in C, and spawn it from python. Python is a coordinating scripting language.
soulbadguy 3y ago
> I never said that multiprocessing is "better" than multithreading. Maybe not, but you seem to be implying that multiprocessing is a sufficient for the most case (or the average case ?) however ill define that average case is. > This conversation would've been much easier to have given specific constraints and examples (which nobody seems to give). Not quite, there is no need grand example here, processes vs threads is just a function of how chatty the interaction between the independent agent are. We should have both mechanism available and let the user choose. > My main argument, if you distill it down to the abstract, is "use the right tool for the job" - and stop pretending that a hammer and a knife are the same thing, when they're not. You don't get to decide in the abstract what the right tool is for other people and context we don't know anything about. The people pushing for proper threading are telling you that for them python + better threading is the right tool. > If you need hardcore, ultra efficient and parallelized workflows - Python is just the wrong tool for the job period. This isn't up for debate - it's a given fact. Maybe; But that's not the point here. Proper threading in python doesn't make python more efficient, but it makes the * scaling * of python performance more efficient. Those are two differents things. > Python fills a niche of its own incredibly well, something which others tools suck at, and Python is loved by millions for it. > This whole discussion now (GIL vs. no GIL) is about bending and twisting Python to fit the use cases of few companies like DeepMind or Facebook > who I'm certain represent a miniscule usage compared to the millions of students, schools and universities, research institutes, web shops, hobbyists, tinkerers, etc. Those people want to get shit done quickly - and mutexes, sempahores, events, threads, synchronization primitives, atomics, etc - will do nothing but make their lives a misery, and drive them away. This seems to me like the core of the problem, somehow you say that there is no valid need for threading, but at the same time the people needing threading are not important enough for python to care about... You have the magical ability to be plug into the python hivemind and decide which use case are valid, what are the true value of the python community and what python should and should not consider important... And after all that mental gymnastic you expect us to jump through hoops to convince you otherwise... That's a lot of non technical assumption for a technical problem. > Those people want to get shit done quickly - and mutexes, sempahores, events, threads, synchronization primitives, atomics, etc - will do nothing but make their lives a misery, and drive them away. Not really. If anything, they will benefits from better libraries and more efficient C-extensions. > absolutely needs multithreading Nothing is absolutely needed. Even something are core as a type system is not absolutely needed. The fact that you ask for this level of qualification for a feature that pretty much every other language has is the problem. The problem is not that deep, implementing multi-threading is not that complicated. I do agree that we need to be careful to provide a better API than raw-thread to end users (which no-gil is COMPLETLY agnostic about).