5 ms·

> What is the real world benefit we will get in return? If you have many CPU cores and an embarrassingly parallel algorithm, multi-threaded Python can now appr

by pansa2 2y ago

> What is the real world benefit we will get in return? If you have many CPU cores and an embarrassingly parallel algorithm, multi-threaded Python can now approach the performance of a single-threaded compiled language.

Certhas 2y ago
The question really is if one couldn't make multiprocess better instead of multithreaded. I did a ton of MPI work with python ten years ago already. What's more I am now seeing in Julia that multithreading doesn't scale to larger core counts (like 128) due to the garbage collector. I had to revert to multithreaded again.
- Sakos 2y ago
  I assume you meant you had to revert to multiprocess?
  - Certhas 2y ago
    Yes exactly. Thanks.
0x000xca0xfe 2y ago
You could already easily parallelize with the multiprocessing module. The real difference is the lower communication overhead between threads vs. processes thanks to a shared address space.
- mharig 2y ago
  [dead]
- bmitc 2y ago
  Easily is an overstatement. Multiprocessing is fraught with quirks.
  - 0x000xca0xfe 2y ago
    Well I once had an analytics/statistics tool that regularly chewed through a couple GBs of CSV files. After enough features had been added it took almost 5 minutes per run which got really annoying. It took me less than an hour to add multiprocessing to analyze each file in its own process and merge the results together at the end. The runtime dropped to a couple seconds on my 24 thread machine. It really was much easier than expected. Rewriting it in C++ would have probably taken a week.
    - bmitc 2y ago
      In F#, it would just be let results = files |> Array.Parralel.map processFile Literally that easy. Earlier this week, I used a ProcessPoolExecutor to run some things in their own process. I needed a bare minimum of synchronization, so I needed a queue. Well, multiprocessing has its own queue. But that queue is not joinable. So I chose the multiprocessing JoinableQueue. Well, it turns out that that queue can't be used across processes. For that, you need to get a queue from the launching process' manager. That Queue is the regular Python queue. It is a gigantic mess. And yes, asyncio also has its own queue class. So in Python, you literally have a half a dozen or so queue classes that are all incompatible, have different interfaces, and have different limitations that are rarely documented. That's just one highlight of the mess between threading, asyncio, and multiprocessing.
      - 0x000xca0xfe 2y ago
        Well I'm not here to debate the API cleanliness, I just wanted to point out to OP that Python can utilize multicore processors without threads ;) Here is the part of multiprocessing I used: with Pool() as p: results = p.map(calc_func, file_paths) So, pretty easy too IMO.
  - sitkack 2y ago
    Fraught with quirks sounds quite ominous. Quuuiiirkkksss. I agree though.
bmitc 2y ago
That's not really correct. Python is by far the slowest mainstream language. It is embarrassingly slow. Further more, several mainstream compiled languages are already multicore compatible and have been for decades. So comparing against a single-threaded language or program doesn't make sense. All this really means is that Python catches up on decades old language design. However, it simply adds yet another design input. Python's threading, multiprocessing, and asyncio paradigms were all developed to get around the limitations of Python's performance issues and the lack of support for multicore. So my question is, how does this change affect the decision tree for selecting which paradigm(s) to use?
- masklinn 2y ago
  > Python's threading, multiprocessing, and asyncio paradigms were all developed to get around the limitations of Python's performance issues and the lack of support for multicore. Threading is literally just Python's multithreading support, using standard OS threads, and async exists for the same reason it exists in a bunch of languages without even a GIL: OS threads have overhead, multiplexing IO-bound work over OS threads is useful. Only multiprocessing can be construed as having been developed to get around the GIL.
  - bmitc 2y ago
    No, asyncio's implementation exists because threading in Python has huge overhead for switching between threads and because threads don't use more than one core. So asyncio was introduced as a single threaded solution specifically for only network-based IO. In any other language, async is implemented on top of the threading model, both because the threading model is more efficient than Python's and because it actually supports multiple cores. Multiprocessing isn't needed in other languages because, again, their threading models support multiple cores. So the three, relatively incompatible paradigms of asyncio, threading, and multiprocessing specifically in Python are indeed separate attempts to account for Python's poor design. Other languages do not have this embedded complexity.
    - cwalv 2y ago

by pansa2 2y ago

Certhas 2y ago

The question really is if one couldn't make multiprocess better instead of multithreaded. I did a ton of MPI work with python ten years ago already. What's more I am now seeing in Julia that multithreading doesn't scale to larger core counts (like 128) due to the garbage collector. I had to revert to multithreaded again.

Sakos 2y ago
I assume you meant you had to revert to multiprocess?
- Certhas 2y ago
  Yes exactly. Thanks.

0x000xca0xfe 2y ago

You could already easily parallelize with the multiprocessing module. The real difference is the lower communication overhead between threads vs. processes thanks to a shared address space.

mharig 2y ago
[dead]
bmitc 2y ago
Easily is an overstatement. Multiprocessing is fraught with quirks.
- 0x000xca0xfe 2y ago
  Well I once had an analytics/statistics tool that regularly chewed through a couple GBs of CSV files. After enough features had been added it took almost 5 minutes per run which got really annoying. It took me less than an hour to add multiprocessing to analyze each file in its own process and merge the results together at the end. The runtime dropped to a couple seconds on my 24 thread machine. It really was much easier than expected. Rewriting it in C++ would have probably taken a week.
  - bmitc 2y ago
    In F#, it would just be let results = files |> Array.Parralel.map processFile Literally that easy. Earlier this week, I used a ProcessPoolExecutor to run some things in their own process. I needed a bare minimum of synchronization, so I needed a queue. Well, multiprocessing has its own queue. But that queue is not joinable. So I chose the multiprocessing JoinableQueue. Well, it turns out that that queue can't be used across processes. For that, you need to get a queue from the launching process' manager. That Queue is the regular Python queue. It is a gigantic mess. And yes, asyncio also has its own queue class. So in Python, you literally have a half a dozen or so queue classes that are all incompatible, have different interfaces, and have different limitations that are rarely documented. That's just one highlight of the mess between threading, asyncio, and multiprocessing.
    - 0x000xca0xfe 2y ago
      Well I'm not here to debate the API cleanliness, I just wanted to point out to OP that Python can utilize multicore processors without threads ;) Here is the part of multiprocessing I used: with Pool() as p: results = p.map(calc_func, file_paths) So, pretty easy too IMO.
- sitkack 2y ago
  Fraught with quirks sounds quite ominous. Quuuiiirkkksss. I agree though.

bmitc 2y ago

That's not really correct. Python is by far the slowest mainstream language. It is embarrassingly slow. Further more, several mainstream compiled languages are already multicore compatible and have been for decades. So comparing against a single-threaded language or program doesn't make sense. All this really means is that Python catches up on decades old language design. However, it simply adds yet another design input. Python's threading, multiprocessing, and asyncio paradigms were all developed to get around the limitations of Python's performance issues and the lack of support for multicore. So my question is, how does this change affect the decision tree for selecting which paradigm(s) to use?

masklinn 2y ago
> Python's threading, multiprocessing, and asyncio paradigms were all developed to get around the limitations of Python's performance issues and the lack of support for multicore. Threading is literally just Python's multithreading support, using standard OS threads, and async exists for the same reason it exists in a bunch of languages without even a GIL: OS threads have overhead, multiplexing IO-bound work over OS threads is useful. Only multiprocessing can be construed as having been developed to get around the GIL.
- bmitc 2y ago
  No, asyncio's implementation exists because threading in Python has huge overhead for switching between threads and because threads don't use more than one core. So asyncio was introduced as a single threaded solution specifically for only network-based IO. In any other language, async is implemented on top of the threading model, both because the threading model is more efficient than Python's and because it actually supports multiple cores. Multiprocessing isn't needed in other languages because, again, their threading models support multiple cores. So the three, relatively incompatible paradigms of asyncio, threading, and multiprocessing specifically in Python are indeed separate attempts to account for Python's poor design. Other languages do not have this embedded complexity.
  - cwalv 2y ago