5 ms·

> Parallelize at the fork level IPC is a PITA, and orchestrating processes is even worse. > or at the isolated numeric library level Not everything I want to

> Parallelize at the fork level IPC is a PITA, and orchestrating processes is even worse. > or at the isolated numeric library level Not everything I want to parallelise in python runs in numpy. Simple example: WebService Backends. I have a 64 core server running a Werkzeug/Gunicorn application. The Service is mostly doing CPU bound tasks (data aggregation and analysis), so asyncio is pointless. What happens is, it runs 60 worker processes. Which puts hefty limitations on any crosstalk and data sharing, because these either require IPC, or using redis/sql. Which are nowhere near as performant as actually shared memory would be.

seanthemon 3y ago
Exactly my thinking, using more cores is exactly your use case and will hopefully make Python superb at large scale data processing with cross-communication. I think, rightfully, the concern is people who will try to use this incorrectly causing major bloat to CPython.
miraculixx 3y ago
You would be well served with arena based (shared) object allocation and a gil per thread model. No need to draw everyone else into this.
- usrbinbash 3y ago
  > No need to draw everyone else into this Please explain who exactly is "drawn into this"? If you want to use async, this change doesn't affect your code. If you want to use multiprocessing, this change doesn't affect your code. Even if you already use threading, and do it correctly, this change doesn't actually affect your code. So who is "drawn into this"? And please don't say library developers. a) Having to update libraries to have them remain relevant, is normal procedure, in all languages. b) a lot of the people who want this to change are library devs.
  - miraculixx 3y ago
    There will be 5+ years of parallel gil/nogil builds. Every library developer will have to deal with this at some point.
    - usrbinbash 3y ago
      You do realize that this grace period is intended precisely to make it easier for libdevs to adapt, yes? 3rd item in the list in the linked article, quote: "We also need to bring along the rest of the Python community as we gain those insights and make sure the changes we want to make, and the changes we want them to make, are palatable." End quote. Yes, library developers have to keep up with developments in the underlying language as well as changes in usage patterns by the community. That is true for all programming languages. And as I have pointed out numerous times before, this change is on the wishlist of many libdevs in the python community.
      - naclet 3y ago
        No, for C/C++ you don't have to do anything unless the compiler writers add yet another new warning to -Wall -W, which you have to disable because it is spurious. Java is more backwards compatible, so is Common Lisp. Someone here said that the thread on the Python "discussion" forum was shut down. That does not sound like everyone except for the proponents is supposed to be heard (or is even aware of the discussion).
        usrbinbash 3y ago
        > No, for C/C++ you don't have to do anything Even for such stable languages, a library maintainer has to, at the very least, patch security problems as they are discovered. > That does not sound like everyone except for the proponents is supposed to be heard (or is even aware of the discussion). There is an official poll among the Python core devs, linked in the article, which shows overwhelming support for the change. Since they are the ones who have to work this out, that's the only discussion about this that is relevant.

seanthemon 3y ago

Exactly my thinking, using more cores is exactly your use case and will hopefully make Python superb at large scale data processing with cross-communication. I think, rightfully, the concern is people who will try to use this incorrectly causing major bloat to CPython.

miraculixx 3y ago

You would be well served with arena based (shared) object allocation and a gil per thread model. No need to draw everyone else into this.

usrbinbash 3y ago
> No need to draw everyone else into this Please explain who exactly is "drawn into this"? If you want to use async, this change doesn't affect your code. If you want to use multiprocessing, this change doesn't affect your code. Even if you already use threading, and do it correctly, this change doesn't actually affect your code. So who is "drawn into this"? And please don't say library developers. a) Having to update libraries to have them remain relevant, is normal procedure, in all languages. b) a lot of the people who want this to change are library devs.
- miraculixx 3y ago
  There will be 5+ years of parallel gil/nogil builds. Every library developer will have to deal with this at some point.
  - usrbinbash 3y ago
    You do realize that this grace period is intended precisely to make it easier for libdevs to adapt, yes? 3rd item in the list in the linked article, quote: "We also need to bring along the rest of the Python community as we gain those insights and make sure the changes we want to make, and the changes we want them to make, are palatable." End quote. Yes, library developers have to keep up with developments in the underlying language as well as changes in usage patterns by the community. That is true for all programming languages. And as I have pointed out numerous times before, this change is on the wishlist of many libdevs in the python community.
    - naclet 3y ago
      No, for C/C++ you don't have to do anything unless the compiler writers add yet another new warning to -Wall -W, which you have to disable because it is spurious. Java is more backwards compatible, so is Common Lisp. Someone here said that the thread on the Python "discussion" forum was shut down. That does not sound like everyone except for the proponents is supposed to be heard (or is even aware of the discussion).
      - usrbinbash 3y ago
        > No, for C/C++ you don't have to do anything Even for such stable languages, a library maintainer has to, at the very least, patch security problems as they are discovered. > That does not sound like everyone except for the proponents is supposed to be heard (or is even aware of the discussion). There is an official poll among the Python core devs, linked in the article, which shows overwhelming support for the change. Since they are the ones who have to work this out, that's the only discussion about this that is relevant.