3 ms·
It's only about performance. asyncio is still inherently single-threaded, and hence also single core. multiprocessing is multi-core and hence better for perform
by n2d4 3y ago
It's only about performance. asyncio is still inherently single-threaded, and hence also single core. multiprocessing is multi-core and hence better for performance, but each process is relatively heavy and there's additional overhead to shared memory. GIL multi-threading is both single-core and difficult to use correctly.
No-GIL multi-threading is multi-core, though difficult to use. I don't know the Python implementation but shared memory should be faster than using multiprocessing.
That said, when designing a system from scratch, I completely agree with you that for almost almost almost all Python use cases, threads should never be touched and asyncio/multiprocessing is the way to go instead. Most Python programs that need fast multi-threading instead should not have been written in Python. Still, we're here now and people did write CPU-intensive code in Python for one reason or another, so no-GIL is practical.
In these threads, I also always see a lot of people who simply aren't aware of asyncio/multiprocessing. I assume these are also a significant share of people asking for no-GIL, though probably not the ones pushing the change in the committee.
- chippiewill 3y ago> asyncio is still inherently single-threaded, and hence also single core. IIRC some of the proposals around removing the GIL in the past have actually suggested that the asyncio paradigm could become multithreaded for parallelism.
- slt2021 3y agoI would argue that if you have large concurrency and shared complex state - you better off use kafka and redis/memcached as a shared state - and design proper fan-out. This design scales much better for systems that will eventually overgrow one big machine. the No-GIL pytohn will be of no use, when you need to deploy your app across 100s machines. I understand people want to take advantage of all cores etc, but at large scale - you will eventually need to split computation across machines and will resort to in-memory cache/queue anyways - so better just architect your system since day0
- n2d4 3y agoStores like that, while scaling well, are orders of magnitudes slower than CPU memory. The kind of application I was thinking of is more compute-intensive, eg. image processing or fancy algorithms.
- whalesalad 3y agosometimes you need all of that data in-process. when you move that state into redis, you still need to perform i/o to access it. when speed matters, this is troublesome.
- pxeger1 3y agoBut many systems will never need to run at that kind of scale, but could still benefit from better threading performance, so it's good to have an "intermediate" choice, if that's what you consider it.
- AlphaSite 3y agoThere are a lot of performance sensitive codebases where something like this would destroy performance, it works well for shared nothing parallelism, but the moment you have shared state it kinda falls over.
- lozenge 3y agoPostgreSQL shows how far you can get with a single big box and using multiple cores and shared memory. It's incredibly powerful and the vast majority of applications never have data big enough to warrant "100s of machines".
- merb 3y agoand even they are looking into changing its shared memory and process model to work and make it multi-threaded: https://www.postgresql.org/message-id/flat/31cc6df9-53fe-3cd9-af5b-ac0d801163f4%40iki.fi https://www.postgresql.org/message-id/flat/31cc6df9-53fe-3cd... probably a way more problematic effort than no gil python tough and it might fail
- mrkeen 3y ago
- KeplerBoy 3y agoEven with multithreading Python will still be slow. It's not like singlethreaded Python can keep up with singlethreaded C or even JS. This will just allow people to waste even more compute resources to get somewhat quick results from Python instead of doing the right thing. From an ecological viewpoint this PEP will be disastrous and keep entire power plants online.
- tjpnz 3y agoThe text editors people use to write said code will keep even more power plants online.
- miraculixx 3y agoEspecially PyCharm with its endless indexing. Have the time its key features are not available bc it has decided to go on another indexing spree. Alas that's for another thread ;)
- AlphaSite 3y agoPython is also getting a JIT, they’re pursuing performance on all axis.
- hanselot 3y agoSometimes the right thing is writing code in a language that reaches market before runway runs out.
- dgb23 3y agoEcosystem churn and exposing more issues to python contradicts this. The whole point of using languages like python, js etc. is to not worry about whole classes of problems and intricacies. There are modern languages with great tooling like Go, Rust or Clojure that have excellent support for safe concurrency and parallel execution etc. And there are reliable, well established languages like Java/C# that have been doing this forever.
- 3y ago
- geekraver 3y agoA free-threaded Python will be harder to make faster for single-threaded cases. So this could be a win for those who want to write multi-threaded code at the expense of everyone else.