5 ms·

It’s more important to prioritize single-threaded performance because it’s much harder to improve by throwing money at the problem. With multithreaded performa

It’s more important to prioritize single-threaded performance because it’s much harder to improve by throwing money at the problem. With multithreaded performance, you can just add another core to (more than) offset whatever overheads there are from using process-based parallelism. I think that this entire GIL vs No-GIL dichotomy is misguided. The biggest problem people have with multiprocessing is that you can’t share memory. So add virtual processes with an explicit mechanism for memory sharing. Then you can keep all of your single-threaded optimizations like refcounting without barriers because the objects for one thread will stay in that thread.

Galanwe 3y ago
Agree 100%. If you need concurrency at the moment, you have already switched to using multiprocessing, so having a no-GIL multithreading is useless. The only issue with Python/multiprocessing, is that sometimes you don't want queues, but shared mutable state. And as you said, placing Python objects in shared memory at the moment is convoluted, restrictive, and suboptimal. Fixing _that_ should be the objective. What Python need is better support for placing native instances in shared memory.
- Znafon 3y ago
  > If you need concurrency at the moment, you have already switched to using multiprocessing, so having a no-GIL multithreading is useless. > The only issue with Python/multiprocessing, is that sometimes you don't want queues, but shared mutable state. And as you said, placing Python objects in shared memory at the moment is convoluted, restrictive, and suboptimal. The PEP goes into the motivation behind this work, and using multiple process does not magically solves all the issues: > Multiprocessing, with communication via shared memory or UNIX sockets, adds much complexity and in effect rules out interacting with CUDA from different workers, severely restricting the design space. > I reimplemented parts of HMMER, a standard method for multiple-sequence alignment. I chose this method because it stresses both single-thread performance (scoring) and multi-threaded performance (searching a database of sequences). The GIL became the bottleneck when using only eight threads. This is a method where the current popular implementations rely on 64 or even 128 threads per process. I tried moving to subprocesses but was blocked by the prohibitive IPC costs. > NumPy does release the GIL in its inner loops (which do the heavy lifting), but that is not nearly enough. NumPy doesn’t offer a solution to utilize all CPU cores of a single machine well, and instead leaves that to Dask and other multiprocessing solutions. Those aren’t very efficient and are also more clumsy to use. That clumsiness comes mainly in the extra abstractions and layers the users need to concern themselves with when using, e.g., dask.array which wraps numpy.ndarray. It also shows up in oversubscription issues that the user must explicitly be aware of and manage via either environment variables or a third package, threadpoolctl. The main reason is that NumPy calls into BLAS for linear algebra - and those calls it has no control over, they do use all cores by default via either pthreads or OpenMP. and it discusses the alternatives at https://peps.python.org/pep-0703/#alternatives https://peps.python.org/pep-0703/#alternatives.
  - zarzavat 3y ago
    You don’t need OS processes for multiprocessing. You can use threads in the same OS process. See: Erlang.
    - Znafon 3y ago
      Would the work on sub-interpreters be interested for that then (https://lwn.net/SubscriberLink/941090/8bcb029dbf548f26/ https://lwn.net/SubscriberLink/941090/8bcb029dbf548f26/) ?
      - zarzavat 3y ago
        Yes, this is exactly what I’m taking about. You don’t need to share objects. Have explicitly shared buffers instead. Python is a dynamic language so you can easily build proxy objects that are views into a shared buffer, and this allows you to keep all your single threaded performance because no objects are shared. For example: buf = sharedbytes.alloc(1 * GiB) with buf.lock(lockid): buf[10:20] = 42 message_other_process(buf.id) # other process def recv_buffer(bufid): buf = sharedbytes.get(bufid) with buf.lock(lockid): print(buf[10:20]) Most people don’t even need that and would be satisfied with just virtual processes and copying message passing between them.
- usrbinbash 3y ago
  > so having a no-GIL multithreading is useless. Wrong. Processes pose hurdles via the limits of IPC and control between processes that Threaded applications don't have to bother with. There are ample examples for this in the PEP. > What Python need is better support for placing native instances in shared memory. If that were the case, then threading wouldn't exist in the first place. Shared memory is still IPC. It still requires context switching into the Kernel. It still poses problems that threads don't have.
  - Galanwe 3y ago
    You are talking nonsense... > It still requires context switching into the Kernel. There is no less context switching between two threads than between two processes. > It still poses problems that threads don't have. Hu, no, it's actually the same problem. If you see threads as different processes with heap being mmap'd at the same location, then you're basically 99.9% right, up to some minor process accounting metadata differences.
    - usrbinbash 3y ago
      > There is no less context switching between two threads than between two processes. I am not talking about switching context between the threads of execution. I am talking about a context switch to kernel code simply to access the shared memory. All interactions with SM require syscalls. So no, I am not "talking nonsense", and yes, communication between multiple processes requires more context switches than communication between threads.
      - Galanwe 3y ago
        > I am talking about a context switch to kernel code simply to access the shared memory. All interactions with SM require syscalls. Please enlighten me with the syscall you use to access a shared memory... because there are none. As far as the kernel is concerned, "memory" is just a set of pages mapped to a process at a specific address. These pages can be anonymous or named (meaning other processes can map them through that name). There is no syscall, no context switch, involved to read or write memory, named or anonymous. Hell, that's even the whole purpose of memory mapped IO. As mentioned earlier, there is also no real difference between a thread and a process as far as the kernel is concerned. A thread is just a special case of process which maps its heap to its parent. Even the word "thread" doesn't really exist for the Linux kernel. We just call these "lightweight processes", because these are just processes with a shared heap.
jillesvangurp 3y ago
The fallacy here is assuming this is a necessary trade off. You can have good single threaded performance without a GIL. Rust has this notion of zero cost abstractions. It does Threads just fine. Java does single threaded stuff just fine as well. Lots of other languages do. This is not science fiction. The locking gets optimized away. Or you can choose to have code without locks at all. Or fine-grained/structured concurrency. Something being thread safe or not is basically just an API contract. And of course optimistic locking is also a thing. There are no good reasons why python wouldn't be able to do similar things. But you need to get rid of the GIL first. A lot of the performance reduction in python without the Gil is basically just unaddressed technical debt. That should be fixable over time. Adding a lot of locks is a stop gap solution. That indeed makes things slower. But the proper fix is probably rethinking how that stuff works internally in a lot of places or having API contracts that document the thread safety or lack thereof. And also having faster python runtimes and compilers actually enables re-implementing a lot of things that currently depend on native libraries in python. A lot of native code interactions are precisely where you need locks. Unless you change how that works. The point of removing the GIL is getting systematic about finding and fixing those things. It will get better over time.
- logicchains 3y ago
  >The fallacy here is assuming this is a necessary trade off. You can have good single threaded performance without a GIL. It depends what you're doing. Classic example: I have a huge dataframe, 10s-100s of GB. I want to process it in multiple threads, with each thread handling a different part of it. If I just use multiprocessing, it has to copy the memory to the new processes and then copy the results back, which is super slow. Sure there are hacky work-arounds and alternative approaches, but in a language without the GIL I don't need to fiddle around with those, I can just do the equivalent of Pool.map() and it doesn't need to copy anything.
  - Galanwe 3y ago
    1) GIL or no GIL does not make a single visible difference in terms of your Python code, how it works, what it can or cannot do. It's only a matter of whether the interpreter internally use locks or not. 2) There is no more copy in multiprocessing that in multithreading, to the very minor exception of the reference counting structures used internally by the interpreter which will get copied on write. 3) The problem you describe is not related to multiprocessing or multithreading, it's just related to a misunderstanding of the Pool.map() API. What you provide to Pool.map() is sent over a queue (and thus pickled) to worker threads - or processes. You don't _have_ to use this queue, so long that your function has a way to access the variable you want to use. That is, the following code will have your subprocesses share, without copying, your 100GB dataframe: DF = pd.read_parquet('100GB.pq') def worker_process(id): # Do something with DF return 42 with multiprocessing.Pool(10): pool.map(work_process, [1, 2, 3, 4, 5])

Galanwe 3y ago

Agree 100%. If you need concurrency at the moment, you have already switched to using multiprocessing, so having a no-GIL multithreading is useless. The only issue with Python/multiprocessing, is that sometimes you don't want queues, but shared mutable state. And as you said, placing Python objects in shared memory at the moment is convoluted, restrictive, and suboptimal. Fixing _that_ should be the objective. What Python need is better support for placing native instances in shared memory.

Znafon 3y ago
> If you need concurrency at the moment, you have already switched to using multiprocessing, so having a no-GIL multithreading is useless. > The only issue with Python/multiprocessing, is that sometimes you don't want queues, but shared mutable state. And as you said, placing Python objects in shared memory at the moment is convoluted, restrictive, and suboptimal. The PEP goes into the motivation behind this work, and using multiple process does not magically solves all the issues: > Multiprocessing, with communication via shared memory or UNIX sockets, adds much complexity and in effect rules out interacting with CUDA from different workers, severely restricting the design space. > I reimplemented parts of HMMER, a standard method for multiple-sequence alignment. I chose this method because it stresses both single-thread performance (scoring) and multi-threaded performance (searching a database of sequences). The GIL became the bottleneck when using only eight threads. This is a method where the current popular implementations rely on 64 or even 128 threads per process. I tried moving to subprocesses but was blocked by the prohibitive IPC costs. > NumPy does release the GIL in its inner loops (which do the heavy lifting), but that is not nearly enough. NumPy doesn’t offer a solution to utilize all CPU cores of a single machine well, and instead leaves that to Dask and other multiprocessing solutions. Those aren’t very efficient and are also more clumsy to use. That clumsiness comes mainly in the extra abstractions and layers the users need to concern themselves with when using, e.g., dask.array which wraps numpy.ndarray. It also shows up in oversubscription issues that the user must explicitly be aware of and manage via either environment variables or a third package, threadpoolctl. The main reason is that NumPy calls into BLAS for linear algebra - and those calls it has no control over, they do use all cores by default via either pthreads or OpenMP. and it discusses the alternatives at https://peps.python.org/pep-0703/#alternatives https://peps.python.org/pep-0703/#alternatives.
- zarzavat 3y ago
  You don’t need OS processes for multiprocessing. You can use threads in the same OS process. See: Erlang.
  - Znafon 3y ago
    Would the work on sub-interpreters be interested for that then (https://lwn.net/SubscriberLink/941090/8bcb029dbf548f26/ https://lwn.net/SubscriberLink/941090/8bcb029dbf548f26/) ?
    - zarzavat 3y ago
      Yes, this is exactly what I’m taking about. You don’t need to share objects. Have explicitly shared buffers instead. Python is a dynamic language so you can easily build proxy objects that are views into a shared buffer, and this allows you to keep all your single threaded performance because no objects are shared. For example: buf = sharedbytes.alloc(1 * GiB) with buf.lock(lockid): buf[10:20] = 42 message_other_process(buf.id) # other process def recv_buffer(bufid): buf = sharedbytes.get(bufid) with buf.lock(lockid): print(buf[10:20]) Most people don’t even need that and would be satisfied with just virtual processes and copying message passing between them.
usrbinbash 3y ago
> so having a no-GIL multithreading is useless. Wrong. Processes pose hurdles via the limits of IPC and control between processes that Threaded applications don't have to bother with. There are ample examples for this in the PEP. > What Python need is better support for placing native instances in shared memory. If that were the case, then threading wouldn't exist in the first place. Shared memory is still IPC. It still requires context switching into the Kernel. It still poses problems that threads don't have.
- Galanwe 3y ago
  You are talking nonsense... > It still requires context switching into the Kernel. There is no less context switching between two threads than between two processes. > It still poses problems that threads don't have. Hu, no, it's actually the same problem. If you see threads as different processes with heap being mmap'd at the same location, then you're basically 99.9% right, up to some minor process accounting metadata differences.
  - usrbinbash 3y ago
    > There is no less context switching between two threads than between two processes. I am not talking about switching context between the threads of execution. I am talking about a context switch to kernel code simply to access the shared memory. All interactions with SM require syscalls. So no, I am not "talking nonsense", and yes, communication between multiple processes requires more context switches than communication between threads.
    - Galanwe 3y ago
      > I am talking about a context switch to kernel code simply to access the shared memory. All interactions with SM require syscalls. Please enlighten me with the syscall you use to access a shared memory... because there are none. As far as the kernel is concerned, "memory" is just a set of pages mapped to a process at a specific address. These pages can be anonymous or named (meaning other processes can map them through that name). There is no syscall, no context switch, involved to read or write memory, named or anonymous. Hell, that's even the whole purpose of memory mapped IO. As mentioned earlier, there is also no real difference between a thread and a process as far as the kernel is concerned. A thread is just a special case of process which maps its heap to its parent. Even the word "thread" doesn't really exist for the Linux kernel. We just call these "lightweight processes", because these are just processes with a shared heap.

jillesvangurp 3y ago

The fallacy here is assuming this is a necessary trade off. You can have good single threaded performance without a GIL. Rust has this notion of zero cost abstractions. It does Threads just fine. Java does single threaded stuff just fine as well. Lots of other languages do. This is not science fiction. The locking gets optimized away. Or you can choose to have code without locks at all. Or fine-grained/structured concurrency. Something being thread safe or not is basically just an API contract. And of course optimistic locking is also a thing. There are no good reasons why python wouldn't be able to do similar things. But you need to get rid of the GIL first. A lot of the performance reduction in python without the Gil is basically just unaddressed technical debt. That should be fixable over time. Adding a lot of locks is a stop gap solution. That indeed makes things slower. But the proper fix is probably rethinking how that stuff works internally in a lot of places or having API contracts that document the thread safety or lack thereof. And also having faster python runtimes and compilers actually enables re-implementing a lot of things that currently depend on native libraries in python. A lot of native code interactions are precisely where you need locks. Unless you change how that works. The point of removing the GIL is getting systematic about finding and fixing those things. It will get better over time.

logicchains 3y ago
>The fallacy here is assuming this is a necessary trade off. You can have good single threaded performance without a GIL. It depends what you're doing. Classic example: I have a huge dataframe, 10s-100s of GB. I want to process it in multiple threads, with each thread handling a different part of it. If I just use multiprocessing, it has to copy the memory to the new processes and then copy the results back, which is super slow. Sure there are hacky work-arounds and alternative approaches, but in a language without the GIL I don't need to fiddle around with those, I can just do the equivalent of Pool.map() and it doesn't need to copy anything.
- Galanwe 3y ago
  1) GIL or no GIL does not make a single visible difference in terms of your Python code, how it works, what it can or cannot do. It's only a matter of whether the interpreter internally use locks or not. 2) There is no more copy in multiprocessing that in multithreading, to the very minor exception of the reference counting structures used internally by the interpreter which will get copied on write. 3) The problem you describe is not related to multiprocessing or multithreading, it's just related to a misunderstanding of the Pool.map() API. What you provide to Pool.map() is sent over a queue (and thus pickled) to worker threads - or processes. You don't _have_ to use this queue, so long that your function has a way to access the variable you want to use. That is, the following code will have your subprocesses share, without copying, your 100GB dataframe: DF = pd.read_parquet('100GB.pq') def worker_process(id): # Do something with DF return 42 with multiprocessing.Pool(10): pool.map(work_process, [1, 2, 3, 4, 5])