4 ms·

Given Python has the multiprocessing module, I always get confused when people talk about Python lack of support for multicore. What are the shortcomings of the

by Qem 5y ago

Given Python has the multiprocessing module, I always get confused when people talk about Python lack of support for multicore. What are the shortcomings of the multiprocessing module, that cause people to disregard it?

amelius 5y ago
Python's multiprocessing module forces you to serialize all your data, or use ctypes. This means it's either going to be slow, or it's like writing code in a hybrid of Pythonic and C-style code. Which of course goes against the idea of using the simplest tool for the job.
- fulafel 5y ago
  You can use shared memory without ctypes: https://docs.python.org/3/library/multiprocessing.shared_memory.html https://docs.python.org/3/library/multiprocessing.shared_mem... But you still do have to jump through some hoops then. But like the example in the docs show there are shareable primitives, you can share numpy arrays, etc.
stonemetal12 5y ago
The multiprocessing module is a hack. It has so much overhead that it is not worth the effort in all but the most embarrassingly parallel situations.
nicolaslem 5y ago
One thing that is often overlooked regarding Python multiprocessing is that depending on the version of Python and the OS, the processes are not launched the same way. For example MacOS cannot use fork while it was (is?) the default on Linux.
formerly_proven 5y ago
Multiprocessing is pretty heavyweight, everything needs to be serialized, you can't share Python objects directly or with good performance across processes, there is (fairly new) shared memory support, but it's just a bag of bytes. It's not particularly good on the cross-platform angle, while it does work on Windows, the startup time for each process is like starting a fresh interpreter, because that's exactly what it is doing. Multiprocessing can work if you have all your logic in Python already, and there is no notable shared state between the processes, and you don't need to coordinate much at all. So pretty much just the very simplest problems in concurrency. Everything else might technically be possible, but will likely be very slow and annoying.
jerf 5y ago
As a sibling mentions, the multiprocessing support is basically support for spawning new processes. It's better than nothing, but it's not very good for the performance. The real problem with Python multicore is that the main problem is solves is the one ctur is talking about, namely, "Oh crap, I used Python and it doesn't run fast enough for my needs... maybe I can run more processes?" Using a language that is already in the slowest class of languages and realistically 40-50x slower than other languages means that just to recover the performance you'd get from switching to a compiled language, you need ~50 perfectly parallel processes solving an embarrassingly parallel problem. If you can't do that... and that's a fairly common case, things that are a full, true 50x embarrassingly parallel aren't actually that common on real hardware because you'll get some sort of contention... then you can't even work your way up to the performance that you could have gotten by starting with Go or C# or some other reasonable language. And that's ignoring the serialization overhead between the processes. If that starts costing you noticeable amounts the number of processes you need to recover goes up really fast, due to the way the math works. Note the only thing about Python causing this effect is its performance; it is equally true of anything else that is the noticeably slower performance league. So, also, note that if you are using Python in one of the ways where it doesn't have this performance problem, such as NumPy with almost all your compute in native code, this analysis doesn't apply. In the 1990s when Python was born, programming in Python was massively easier than programming in the static languages of the day. The landscape has shifted... Python is now only incrementally better than modern static languages in some dimensions, and I tend to agree with ctur, it just plain isn't better once you pass a certain size and the stereotypical problems with dynamically-typed code start hitting you harder and harder. There are now a lot of good statically-typed languages where you can start a new project, writing something just incrementally harder to write than Python, with the type inference, built-in associative types, tons more libraries, etc. all the advances of the past 25 years, and get static-typed performance. The gulf between "easy Python" and "pull your hair out static language" is not quite closed. Not quite. But it's a lot closer than it used to be; less "Grand Canyon" as it was in the 1990s and more "can I jump that creek? I mean, it's pretty close... I think I can jump it...". The upshot is, if you're reaching for multiprocessing for performance reasons, not convenience reasons, you've very nearly already lost. There's a narrow window where it might still make a bit of sense, but you're getting perilously close to "You need to switch languages" just by reaching for it at all.
fiedzia 5y ago
Sharing data between processes is _very_ expensive, so whole classes of programs are not suited for this approach. For example I had a program that preloads large immutable dataset. and a bunch of threads use it. I couldn't do it efficiently in Python (at least not CPython).
- rich_sasha 5y ago
  Outside of Windows, I believe, you can do it so that the dataset is read-only shared (or copy-on-write) via fork, and then it’s as quick as local access. Not on Windows though, as it lacks fork.
  - bawr_hszm 5y ago
    Loading data before you fork works pretty well, but the overall effectiveness heavily depends on the type of data involved. If it's something like numpy arrays or similar large, indivisible objects, you're golden. If you want to preload and share something like a huge nested Python dictionary or other large collections of small objects, you immediately collide with the reference counters. Basically, since the reference count is kept right before the object data, as soon as the child process touches it - even just to look at it! - you immediately trigger a copy on write on the nearest 4k of memory, which tends to add up fast if you're not careful. Even if you never touch 99% of them, the garbage collector is happy to do it for you. At my previous job it was bad enough that I ended up writing a small patch to be able to set some refcounts to 0xFF...FF and treat them specially, never changing their value. (Yes, this also meant that they never got destroyed properly, and the extra checks made our codebase around 4% slower, but it was an acceptable tradeoff. No, the patch was no longer small by the time it hit production.)
mpdehaan2 5y ago
I am probably the most commercially successful user of the multiprocessing module :) It's basically fine anywhere you need a function call that you can dispatch out to like 50 or 500 workers on a queue and then do something after that returns, but any shared memory or IPC between the workers is up to you. Python is also fine for webserving because most web servers pre-fork workers or whatever, so this doesn't come into play there either. It's harder if you want to do something different where you want threaded workflows with synchronized/protected like constructs that folks might be familiar with from say Java. Firing up multiprocessing (forking) has some costs to bringing up the interpreters so it's not something you want to start up a lot and then close down a lot, better if you can start things and leave them running. Once it's up it is pretty fast. I guess mainly it changes the style of your program too much - it's basically just glue around forks.
- zohch 5y ago
  > any shared memory or IPC between the workers is up to you. There is this: - https://docs.python.org/3/library/multiprocessing.shared_memory.html https://docs.python.org/3/library/multiprocessing.shared_mem... - https://docs.python.org/3/library/queue.html https://docs.python.org/3/library/queue.html - https://docs.python.org/3.8/library/multiprocessing.html#multiprocessing.Pipe https://docs.python.org/3.8/library/multiprocessing.html#mul... - https://docs.python.org/3.8/library/multiprocessing.html#multiprocessing.connection.Connection https://docs.python.org/3.8/library/multiprocessing.html#mul... > It's harder if you want to do something different where you want threaded workflows with synchronized/protected like constructs that folks might be familiar with from say Java. There is this: - https://docs.python.org/3/library/threading.html https://docs.python.org/3/library/threading.html - which you can use as a context manager (i.e. `with lock`)
  - deleted 5y ago
    [deleted]
wassenaar10 5y ago
The performance overhead of multiprocessing is pretty high, to the point that you really have to take care to ensure that whatever problem you're solving with multiprocessing is large and parallel enough to justify it, otherwise your multiprocessing solution may very well be slower than your single-process one. That's not to say it's useless, it's a nice tool to have in the standard library, but if you're coming from one of the many mainstream programming languages with lightweight threading libraries which make it much easier to take advantage of parallelism it's easy to get frustrated at both python's standard threading and multiprocessing libraries for their respective shortcomings, at least in their CPython implementations.
rich_sasha 5y ago
On Windows multiprocessing is very poor due to lack of forking. This means for example no memory at all is shared; modules, data, external shared libraries etc are all individually loaded by each subprocess. There are also some Windows-specific oddities that make using it a further pain (can’t remember the details). There is a project to effectively internalise multiprocessing by running separate interpreters inside one process. It’s meant to be cheaper than having separate modules too.

by Qem 5y ago

amelius 5y ago

Python's multiprocessing module forces you to serialize all your data, or use ctypes. This means it's either going to be slow, or it's like writing code in a hybrid of Pythonic and C-style code. Which of course goes against the idea of using the simplest tool for the job.

fulafel 5y ago
You can use shared memory without ctypes: https://docs.python.org/3/library/multiprocessing.shared_memory.html https://docs.python.org/3/library/multiprocessing.shared_mem... But you still do have to jump through some hoops then. But like the example in the docs show there are shareable primitives, you can share numpy arrays, etc.

stonemetal12 5y ago

The multiprocessing module is a hack. It has so much overhead that it is not worth the effort in all but the most embarrassingly parallel situations.

nicolaslem 5y ago

One thing that is often overlooked regarding Python multiprocessing is that depending on the version of Python and the OS, the processes are not launched the same way. For example MacOS cannot use fork while it was (is?) the default on Linux.

formerly_proven 5y ago

Multiprocessing is pretty heavyweight, everything needs to be serialized, you can't share Python objects directly or with good performance across processes, there is (fairly new) shared memory support, but it's just a bag of bytes. It's not particularly good on the cross-platform angle, while it does work on Windows, the startup time for each process is like starting a fresh interpreter, because that's exactly what it is doing. Multiprocessing can work if you have all your logic in Python already, and there is no notable shared state between the processes, and you don't need to coordinate much at all. So pretty much just the very simplest problems in concurrency. Everything else might technically be possible, but will likely be very slow and annoying.

jerf 5y ago

As a sibling mentions, the multiprocessing support is basically support for spawning new processes. It's better than nothing, but it's not very good for the performance. The real problem with Python multicore is that the main problem is solves is the one ctur is talking about, namely, "Oh crap, I used Python and it doesn't run fast enough for my needs... maybe I can run more processes?" Using a language that is already in the slowest class of languages and realistically 40-50x slower than other languages means that just to recover the performance you'd get from switching to a compiled language, you need ~50 perfectly parallel processes solving an embarrassingly parallel problem. If you can't do that... and that's a fairly common case, things that are a full, true 50x embarrassingly parallel aren't actually that common on real hardware because you'll get some sort of contention... then you can't even work your way up to the performance that you could have gotten by starting with Go or C# or some other reasonable language. And that's ignoring the serialization overhead between the processes. If that starts costing you noticeable amounts the number of processes you need to recover goes up really fast, due to the way the math works. Note the only thing about Python causing this effect is its performance; it is equally true of anything else that is the noticeably slower performance league. So, also, note that if you are using Python in one of the ways where it doesn't have this performance problem, such as NumPy with almost all your compute in native code, this analysis doesn't apply. In the 1990s when Python was born, programming in Python was massively easier than programming in the static languages of the day. The landscape has shifted... Python is now only incrementally better than modern static languages in some dimensions, and I tend to agree with ctur, it just plain isn't better once you pass a certain size and the stereotypical problems with dynamically-typed code start hitting you harder and harder. There are now a lot of good statically-typed languages where you can start a new project, writing something just incrementally harder to write than Python, with the type inference, built-in associative types, tons more libraries, etc. all the advances of the past 25 years, and get static-typed performance. The gulf between "easy Python" and "pull your hair out static language" is not quite closed. Not quite. But it's a lot closer than it used to be; less "Grand Canyon" as it was in the 1990s and more "can I jump that creek? I mean, it's pretty close... I think I can jump it...". The upshot is, if you're reaching for multiprocessing for performance reasons, not convenience reasons, you've very nearly already lost. There's a narrow window where it might still make a bit of sense, but you're getting perilously close to "You need to switch languages" just by reaching for it at all.

fiedzia 5y ago

Sharing data between processes is _very_ expensive, so whole classes of programs are not suited for this approach. For example I had a program that preloads large immutable dataset. and a bunch of threads use it. I couldn't do it efficiently in Python (at least not CPython).

rich_sasha 5y ago
Outside of Windows, I believe, you can do it so that the dataset is read-only shared (or copy-on-write) via fork, and then it’s as quick as local access. Not on Windows though, as it lacks fork.
- bawr_hszm 5y ago
  Loading data before you fork works pretty well, but the overall effectiveness heavily depends on the type of data involved. If it's something like numpy arrays or similar large, indivisible objects, you're golden. If you want to preload and share something like a huge nested Python dictionary or other large collections of small objects, you immediately collide with the reference counters. Basically, since the reference count is kept right before the object data, as soon as the child process touches it - even just to look at it! - you immediately trigger a copy on write on the nearest 4k of memory, which tends to add up fast if you're not careful. Even if you never touch 99% of them, the garbage collector is happy to do it for you. At my previous job it was bad enough that I ended up writing a small patch to be able to set some refcounts to 0xFF...FF and treat them specially, never changing their value. (Yes, this also meant that they never got destroyed properly, and the extra checks made our codebase around 4% slower, but it was an acceptable tradeoff. No, the patch was no longer small by the time it hit production.)

mpdehaan2 5y ago

I am probably the most commercially successful user of the multiprocessing module :) It's basically fine anywhere you need a function call that you can dispatch out to like 50 or 500 workers on a queue and then do something after that returns, but any shared memory or IPC between the workers is up to you. Python is also fine for webserving because most web servers pre-fork workers or whatever, so this doesn't come into play there either. It's harder if you want to do something different where you want threaded workflows with synchronized/protected like constructs that folks might be familiar with from say Java. Firing up multiprocessing (forking) has some costs to bringing up the interpreters so it's not something you want to start up a lot and then close down a lot, better if you can start things and leave them running. Once it's up it is pretty fast. I guess mainly it changes the style of your program too much - it's basically just glue around forks.

zohch 5y ago
> any shared memory or IPC between the workers is up to you. There is this: - https://docs.python.org/3/library/multiprocessing.shared_memory.html https://docs.python.org/3/library/multiprocessing.shared_mem... - https://docs.python.org/3/library/queue.html https://docs.python.org/3/library/queue.html - https://docs.python.org/3.8/library/multiprocessing.html#multiprocessing.Pipe https://docs.python.org/3.8/library/multiprocessing.html#mul... - https://docs.python.org/3.8/library/multiprocessing.html#multiprocessing.connection.Connection https://docs.python.org/3.8/library/multiprocessing.html#mul... > It's harder if you want to do something different where you want threaded workflows with synchronized/protected like constructs that folks might be familiar with from say Java. There is this: - https://docs.python.org/3/library/threading.html https://docs.python.org/3/library/threading.html - which you can use as a context manager (i.e. `with lock`)
- deleted 5y ago
  [deleted]

wassenaar10 5y ago

The performance overhead of multiprocessing is pretty high, to the point that you really have to take care to ensure that whatever problem you're solving with multiprocessing is large and parallel enough to justify it, otherwise your multiprocessing solution may very well be slower than your single-process one. That's not to say it's useless, it's a nice tool to have in the standard library, but if you're coming from one of the many mainstream programming languages with lightweight threading libraries which make it much easier to take advantage of parallelism it's easy to get frustrated at both python's standard threading and multiprocessing libraries for their respective shortcomings, at least in their CPython implementations.

rich_sasha 5y ago

On Windows multiprocessing is very poor due to lack of forking. This means for example no memory at all is shared; modules, data, external shared libraries etc are all individually loaded by each subprocess. There are also some Windows-specific oddities that make using it a further pain (can’t remember the details). There is a project to effectively internalise multiprocessing by running separate interpreters inside one process. It’s meant to be cheaper than having separate modules too.