3 ms·

Is it really too late to not do this ? The only reason to get rid of the GIL is to help threading, but that's not a thing we should be doing. Threads need to ju

Is it really too late to not do this ? The only reason to get rid of the GIL is to help threading, but that's not a thing we should be doing. Threads need to just die, and be replaced by something less idiotic. Seriously, having the CPU run fragments of your program at random, so that all the previously ordered pieces are now contending with each other and even themselves ? How can anyone not see that this is the stupidest idea in the world ?

Jabrov 3y ago
As opposed to?
- n2d4 3y ago
  In Python, asyncio and multiprocessing packages can get nearly the same or better performance for IO- and CPU-intensive tasks respectively as no-GIL multithreading (and are more performant than GIL multithreading), with only a tiny fraction of the pitfalls. For any use case where the last few percent matter, consider not using Python (which will be much much more significant). Regardless, we did somehow end up here, and there's plenty of multi-threaded Python code that would benefit from no-GIL, so I support the proposal just from a practical perspective. But when designing a new codebase, you'll almost almost almost always want to avoid Python threads, even with no-GIL.
  - usrbinbash 3y ago
    > asyncio ...is useless for CPU bound tasks. The event loop uses only one core. > multiprocessing ...relies on IPC and running actual system processes, both of which have alot more overhead than switching thread context and using shared memory. > For any use case where the last few percent matter, consider not using Python (which will be much much more significant). Here is an interesting question: If asyncio and multiprocessing already give us "nearly the same or better performance", then why is "use another language" such a common advice to escape parallelism-problems in Python? Because, curiously enough, the languages that are usually recommended for this (C, Go, Rust, C++, Java) all implement thread-based parallelism.
    - n2d4 3y ago
      If you're interested, I wrote a more elaborate comment on asyncio vs. multiprocessing vs. multithreading here: https://news.ycombinator.com/item?id=36915581 https://news.ycombinator.com/item?id=36915581 (fyi, Python's multiprocessing has support for interprocess shared memory, with overhead) Those languages you mentioned are not only recommended to escape parallelism problems in Python. They are recommended because they are much, much faster, period. Four of those you listed compile down to machine code, the last one has a highly optimized JIT. Just two are garbage-collected, and none do reference counting. All of them are strongly typed. These differences sum up to orders of magnitudes. If well-written Python code were equally fast as well-written C code but the only way to do parallelism was using processes, then I promise you you wouldn't get (most of) those voices telling you to escape Python. Plenty of well-written C programs choose a multiprocessing approach for parallelism over multithreading. Long story short, if you worry about the performance overhead between multithreading and multiprocessing, make sure you worry about plenty of more significant factors that differentiate Python from faster languages first.
      - usrbinbash 3y ago
        > (fyi, Python's multiprocessing has support for interprocess shared memory, with overhead) I know. I have a werkzeug/gunicorn application that currently runs 60 worker processes on a 64 core server. And I would love nothing better than to rip out every. single. last. one. of the IPC facilities, and replace them with the same easy and convenient mutex and CSP systems, that I can use in my Golang applications. > They are recommended because they are much, much faster, period. I know, but "rewrite it in Rust/Go/C++" isn't always an option, be it because of legacy status, constraints in available dev-hours, compatibility problems, library support or simply ease of use.
- catnibbler 3y ago
  Keep the GIL and avoid the problems its removal will cause. Allow parts of your program to run in a separate namespace with explicit passing of objects. No sharing means no contention, so no overhead.
  - miraculixx 3y ago
    Indeed!
miraculixx 3y ago
I was called inconsiderate for essentially asking this very question. I still think it's the right question to ask: Why should Python even have a free threading model?
AlphaSite 3y ago
Python already has threads, they just have huge downsides in their current form, so this ship has long since sailed.
nottorp 3y ago
It's a workaround for not having 1200 Ghz CPUs but instead 128 cores at 3 Ghz...
usrbinbash 3y ago
> Threads need to just die, and be replaced by something less idiotic. Please tell us: What other solution do you propose for running CPU bound workloads in parallel? There are exactly 2: Multiprocessing and using another language.

Jabrov 3y ago

As opposed to?

n2d4 3y ago
In Python, asyncio and multiprocessing packages can get nearly the same or better performance for IO- and CPU-intensive tasks respectively as no-GIL multithreading (and are more performant than GIL multithreading), with only a tiny fraction of the pitfalls. For any use case where the last few percent matter, consider not using Python (which will be much much more significant). Regardless, we did somehow end up here, and there's plenty of multi-threaded Python code that would benefit from no-GIL, so I support the proposal just from a practical perspective. But when designing a new codebase, you'll almost almost almost always want to avoid Python threads, even with no-GIL.
- usrbinbash 3y ago
  > asyncio ...is useless for CPU bound tasks. The event loop uses only one core. > multiprocessing ...relies on IPC and running actual system processes, both of which have alot more overhead than switching thread context and using shared memory. > For any use case where the last few percent matter, consider not using Python (which will be much much more significant). Here is an interesting question: If asyncio and multiprocessing already give us "nearly the same or better performance", then why is "use another language" such a common advice to escape parallelism-problems in Python? Because, curiously enough, the languages that are usually recommended for this (C, Go, Rust, C++, Java) all implement thread-based parallelism.
  - n2d4 3y ago
    If you're interested, I wrote a more elaborate comment on asyncio vs. multiprocessing vs. multithreading here: https://news.ycombinator.com/item?id=36915581 https://news.ycombinator.com/item?id=36915581 (fyi, Python's multiprocessing has support for interprocess shared memory, with overhead) Those languages you mentioned are not only recommended to escape parallelism problems in Python. They are recommended because they are much, much faster, period. Four of those you listed compile down to machine code, the last one has a highly optimized JIT. Just two are garbage-collected, and none do reference counting. All of them are strongly typed. These differences sum up to orders of magnitudes. If well-written Python code were equally fast as well-written C code but the only way to do parallelism was using processes, then I promise you you wouldn't get (most of) those voices telling you to escape Python. Plenty of well-written C programs choose a multiprocessing approach for parallelism over multithreading. Long story short, if you worry about the performance overhead between multithreading and multiprocessing, make sure you worry about plenty of more significant factors that differentiate Python from faster languages first.
    - usrbinbash 3y ago
      > (fyi, Python's multiprocessing has support for interprocess shared memory, with overhead) I know. I have a werkzeug/gunicorn application that currently runs 60 worker processes on a 64 core server. And I would love nothing better than to rip out every. single. last. one. of the IPC facilities, and replace them with the same easy and convenient mutex and CSP systems, that I can use in my Golang applications. > They are recommended because they are much, much faster, period. I know, but "rewrite it in Rust/Go/C++" isn't always an option, be it because of legacy status, constraints in available dev-hours, compatibility problems, library support or simply ease of use.
catnibbler 3y ago
Keep the GIL and avoid the problems its removal will cause. Allow parts of your program to run in a separate namespace with explicit passing of objects. No sharing means no contention, so no overhead.
- miraculixx 3y ago
  Indeed!

miraculixx 3y ago

I was called inconsiderate for essentially asking this very question. I still think it's the right question to ask: Why should Python even have a free threading model?

AlphaSite 3y ago

Python already has threads, they just have huge downsides in their current form, so this ship has long since sailed.

nottorp 3y ago

It's a workaround for not having 1200 Ghz CPUs but instead 128 cores at 3 Ghz...

usrbinbash 3y ago

> Threads need to just die, and be replaced by something less idiotic. Please tell us: What other solution do you propose for running CPU bound workloads in parallel? There are exactly 2: Multiprocessing and using another language.