15 ms·

It seems to me that in python3 using: [ asyncio, event loop, ProcessPoolExecutor, or ThreadPoolExecutor ] satisfies this problem. If I am misunderstood please c

It seems to me that in python3 using: [ asyncio, event loop, ProcessPoolExecutor, or ThreadPoolExecutor ] satisfies this problem. If I am misunderstood please correct me! https://pymotw.com/3/asyncio/executors.html https://pymotw.com/3/asyncio/executors.html

pdonis 7y ago
The only one of these that solves the GIL problem (that only one thread in a given Python process can be running Python bytecode at a time) is ProcessPoolExecutor, or more generally forking separate Python processes for each computation you want to do that uses Python bytecode. The others all still are limited by the GIL. What subinterpreter support will do is basically to allow multiple GILs inside a single Python process, each in its own thread, so multiple threads in the same process can now run Python bytecode.
- duckerude 7y ago
  Notably, subinterpreters might be able to share data without copying it. ProcessPoolExecutor can only share data by pickling it.
  - pdonis 7y ago
    > subinterpreters might be able to share data without copying it In the current proposal as described in the PEP, they wouldn't. Subinterpreters will only be able to communicate through channels, and arbitrary objects can't even be sent through channels; they would have to be pickled or marshaled, similar to sharing data between processes.
    - duckerude 7y ago
      If I'm reading the PEP correctly, it's possible to send memoryview buffers through channels without copying the underlying data. The few other types supported by channels (None, bytes, str, int) are all immutable, so I would guess that those are also shared without copying.
      - pdonis 7y ago
        Yes, these particular types of data can be shared (as you say, sharing immutable objects obviously doesn't require copying since they're immutable); but I took "shared data" to mean being able to share arbitrary objects, since that's the sense of "sharing" that is relevant when we're talking about various forms of concurrency. If all I can share between concurrent computations is those limited data types, then I have to pickle/marshal everything else, which is effectively the same thing I have to do with multiple processes.
        duckerude 7y ago
        I see. I was paraphrasing from the article originally, and didn't mean "share data" in a very general way. Sharing memoryviews lets you share e.g. numpy arrays, so it should be powerful enough for some applications.
ggm 7y ago
From my own experience, ThreadPoolExecutor is a tarpit for the naieve user (me) -I wound up in multiprocessing_in_dill because serialisation hits you out of the blue as an issue. Things can work with pool executor count 1 and fail at 10 and you have no idea why. A map(func, [list]) abstraction should work. It just sometimes doesn't. I have no doubt better abstraction and core knowledge could get over this, and they are probably traps for the unwary in MP as a whole, not the specific model (been a CS person for 40 years btw, so this is perpetual newb state)

pdonis 7y ago

The only one of these that solves the GIL problem (that only one thread in a given Python process can be running Python bytecode at a time) is ProcessPoolExecutor, or more generally forking separate Python processes for each computation you want to do that uses Python bytecode. The others all still are limited by the GIL. What subinterpreter support will do is basically to allow multiple GILs inside a single Python process, each in its own thread, so multiple threads in the same process can now run Python bytecode.

duckerude 7y ago
Notably, subinterpreters might be able to share data without copying it. ProcessPoolExecutor can only share data by pickling it.
- pdonis 7y ago
  > subinterpreters might be able to share data without copying it In the current proposal as described in the PEP, they wouldn't. Subinterpreters will only be able to communicate through channels, and arbitrary objects can't even be sent through channels; they would have to be pickled or marshaled, similar to sharing data between processes.
  - duckerude 7y ago
    If I'm reading the PEP correctly, it's possible to send memoryview buffers through channels without copying the underlying data. The few other types supported by channels (None, bytes, str, int) are all immutable, so I would guess that those are also shared without copying.
    - pdonis 7y ago
      Yes, these particular types of data can be shared (as you say, sharing immutable objects obviously doesn't require copying since they're immutable); but I took "shared data" to mean being able to share arbitrary objects, since that's the sense of "sharing" that is relevant when we're talking about various forms of concurrency. If all I can share between concurrent computations is those limited data types, then I have to pickle/marshal everything else, which is effectively the same thing I have to do with multiple processes.
      - duckerude 7y ago
        I see. I was paraphrasing from the article originally, and didn't mean "share data" in a very general way. Sharing memoryviews lets you share e.g. numpy arrays, so it should be powerful enough for some applications.

ggm 7y ago

From my own experience, ThreadPoolExecutor is a tarpit for the naieve user (me) -I wound up in multiprocessing_in_dill because serialisation hits you out of the blue as an issue. Things can work with pool executor count 1 and fail at 10 and you have no idea why. A map(func, [list]) abstraction should work. It just sometimes doesn't. I have no doubt better abstraction and core knowledge could get over this, and they are probably traps for the unwary in MP as a whole, not the specific model (been a CS person for 40 years btw, so this is perpetual newb state)