5 ms·

This covers just the most basic things that can be looked up in the manual (IDK if it's ChatGPT output or not, doesn't matter). It doesn't answer * why you ne

This covers just the most basic things that can be looked up in the manual (IDK if it's ChatGPT output or not, doesn't matter). It doesn't answer * why you need multiprocessing (rather than threads) * what are costs and benefits of them * how to process a sequence of tasks maintaining results in order * ...or without maintaining the order * how to handle exceptions in the processes * how to gracefully exit by Ctrl+C or other exceptions in the main process * how to minimize data transmission between processes, because it's costy etc. None of this is covered.

benterix 3y ago
Yeah, that's struck me as well. "Let's get down to some work" without explaining why it should (or should not!) be done. And in this case it's crucial.
- nerdponx 3y ago
  It's low quality content marketing for Red Hat. Equivalent to all those Medium articles written by boot camp grads, except it's a Red Hat ad instead of a "hire me" ad.
- maegul 3y ago
  I mean par for the course for tech blogs/articles right.
  - benterix 3y ago
    Agreed, but the bar for HN submissions is slightly higher.
velosol 3y ago
I don't think I can do all of those justice right this moment but let's get a few here so people can link to an HN thread instead of the red hat article ;) MP is good when the GIL would hamper your thread concurrency, ie your problem is likely CPU bound rather than IO/network bound (where Cython is good about releasing the GIL). Cost is that there's process overhead of each new instance of Python running the worker code and pickling any required data to/from the worker across a process boundary (rather than between threads). Benefit is mostly the last answer: can saturate all available CPUs. Few options but generally something like `concurrent.future`'s `.map` will keep tasks with order while a `.submit` and then checking with `.as_completed` will be tasks out of order (but if you return an ID of what you were working on you could reorder after and that may be worthwhile if the workloads are highly variable). Exceptions: Capture all in your worker and make available to the main via event or queue and check that signal periodically in your main and take action as needed. For the other (Ctrl+C in your main) have your workers periodically check a signal from main as often as needed for the responsiveness desired and have the worker cleanup/quit on Interrupt signals. Data transmission feels too problem-dependent to give a single answer to but if you're processing say, files, don't read and pass the files bytes to a worker, pass the file's location and let the worker read the file and return/write results.
- culebron21 3y ago
  Thanks. That's more or less how I solved these problems too. And the questions are what I had to painstakingly answer myself at work. BTW, I haven't touched MP in Py for couple of years now, but remember there's some misalignment in concurrent futures between pool.map and pool.submit. And for exceptions in background processes, I had a shared bool var called "emergency brake", and an error queue.
  - velosol 3y ago
    I like "emergency brake" as a variable name! Glad to help, it was similar for me with a lot of effort to synthesize best practices and really appreciate all the concurrent options (asyncio, threads, MP) for where they each are valuable.
mmmmpancakes 3y ago
I mean, hopefully why you might need multiprocessing in python is clear? If you have a python task that is highly parallelizable on a single machine with multiple cores, then multiprocessing is probably the right tool to quickly see if you can dramatically speed up your code with parallelism with basically no code overhead or investment in distributed solutions (there are edge cases where it is not, but it takes very little time to test if you are an edge case). I encounter this situation in my data science workflow routinely. It is an easy way to impress product / managers and say "hey, I made this batch algorithm 50x faster, so now it runs in 10 minutes instead of 500."
- bshipp 3y ago
  As a simple example, Threading in Python works good for i/o-bound operations like scraping a website, whereas Multiprocessing works best for CPU-bound operations like result hashing/transforming the data you just scraped.
- nomel 3y ago
  > I mean, hopefully why you might need multiprocessing in python is clear? My experience, after first getting into python, was: I needed to do something concurrently on one set of data. Python threading doesn't provide concurrent execution, so my program slowed down when I used threads. So, I tried multiprocessing. My program slowed down even more, because any communication between processes uses pickle. I was trying to process one dataset in parallel, and pass big chunks back, for a final processing. So, I saved it to disk, loaded the dataset into each process, multiplying my memory usage by 16x. I then threw it all out and wrote the performant bits in C++, using swig to automagically make the python interface for it. So, knowing why (concurrency) isn't necessarily enough.
  - mmmmpancakes 3y ago
    I mean, of course? Concurrency != parallelism, so that makes sense. Where multiprocessing shines is when you have an algorithm that can be fully parallelized and represented in a baby map-reduce framework, where the data being sent to each process isn't too big. The idiom I often reuse is literally the first example in the python docs: from multiprocessing import Pool def f(x): return x\*x if __name__ == '__main__': with Pool(5) as p: print(p.map(f, [1, 2, 3])) # to give more of a map reduce flavor: print(sum(p.map(f, [1, 2, 3]))) This can be modified to fit a pretty big range of tasks in my use cases and chop hours off my workflows. It's so much faster in DS workflows to just have extra cores and use them than to spin up a cluster for distributed compute.
TheLocehiliosan 3y ago
I often point people here to get some good background when doing concurrency in Python. https://realpython.com/python-concurrency/ https://realpython.com/python-concurrency/
culebron21 3y ago
I've just left the question open, but the best learning material for me on this was this lecture. Other points down the list, I had to figure out just by trial and error. Raymond Hettinger, Keynote on Concurrency, PyBay 2017 https://www.youtube.com/watch?v=9zinZmE3Ogk https://www.youtube.com/watch?v=9zinZmE3Ogk

benterix 3y ago

Yeah, that's struck me as well. "Let's get down to some work" without explaining why it should (or should not!) be done. And in this case it's crucial.

nerdponx 3y ago
It's low quality content marketing for Red Hat. Equivalent to all those Medium articles written by boot camp grads, except it's a Red Hat ad instead of a "hire me" ad.
maegul 3y ago
I mean par for the course for tech blogs/articles right.
- benterix 3y ago
  Agreed, but the bar for HN submissions is slightly higher.

velosol 3y ago

I don't think I can do all of those justice right this moment but let's get a few here so people can link to an HN thread instead of the red hat article ;) MP is good when the GIL would hamper your thread concurrency, ie your problem is likely CPU bound rather than IO/network bound (where Cython is good about releasing the GIL). Cost is that there's process overhead of each new instance of Python running the worker code and pickling any required data to/from the worker across a process boundary (rather than between threads). Benefit is mostly the last answer: can saturate all available CPUs. Few options but generally something like `concurrent.future`'s `.map` will keep tasks with order while a `.submit` and then checking with `.as_completed` will be tasks out of order (but if you return an ID of what you were working on you could reorder after and that may be worthwhile if the workloads are highly variable). Exceptions: Capture all in your worker and make available to the main via event or queue and check that signal periodically in your main and take action as needed. For the other (Ctrl+C in your main) have your workers periodically check a signal from main as often as needed for the responsiveness desired and have the worker cleanup/quit on Interrupt signals. Data transmission feels too problem-dependent to give a single answer to but if you're processing say, files, don't read and pass the files bytes to a worker, pass the file's location and let the worker read the file and return/write results.

culebron21 3y ago
Thanks. That's more or less how I solved these problems too. And the questions are what I had to painstakingly answer myself at work. BTW, I haven't touched MP in Py for couple of years now, but remember there's some misalignment in concurrent futures between pool.map and pool.submit. And for exceptions in background processes, I had a shared bool var called "emergency brake", and an error queue.
- velosol 3y ago
  I like "emergency brake" as a variable name! Glad to help, it was similar for me with a lot of effort to synthesize best practices and really appreciate all the concurrent options (asyncio, threads, MP) for where they each are valuable.

mmmmpancakes 3y ago

I mean, hopefully why you might need multiprocessing in python is clear? If you have a python task that is highly parallelizable on a single machine with multiple cores, then multiprocessing is probably the right tool to quickly see if you can dramatically speed up your code with parallelism with basically no code overhead or investment in distributed solutions (there are edge cases where it is not, but it takes very little time to test if you are an edge case). I encounter this situation in my data science workflow routinely. It is an easy way to impress product / managers and say "hey, I made this batch algorithm 50x faster, so now it runs in 10 minutes instead of 500."

bshipp 3y ago
As a simple example, Threading in Python works good for i/o-bound operations like scraping a website, whereas Multiprocessing works best for CPU-bound operations like result hashing/transforming the data you just scraped.
nomel 3y ago
> I mean, hopefully why you might need multiprocessing in python is clear? My experience, after first getting into python, was: I needed to do something concurrently on one set of data. Python threading doesn't provide concurrent execution, so my program slowed down when I used threads. So, I tried multiprocessing. My program slowed down even more, because any communication between processes uses pickle. I was trying to process one dataset in parallel, and pass big chunks back, for a final processing. So, I saved it to disk, loaded the dataset into each process, multiplying my memory usage by 16x. I then threw it all out and wrote the performant bits in C++, using swig to automagically make the python interface for it. So, knowing why (concurrency) isn't necessarily enough.
- mmmmpancakes 3y ago
  I mean, of course? Concurrency != parallelism, so that makes sense. Where multiprocessing shines is when you have an algorithm that can be fully parallelized and represented in a baby map-reduce framework, where the data being sent to each process isn't too big. The idiom I often reuse is literally the first example in the python docs: from multiprocessing import Pool def f(x): return x\*x if __name__ == '__main__': with Pool(5) as p: print(p.map(f, [1, 2, 3])) # to give more of a map reduce flavor: print(sum(p.map(f, [1, 2, 3]))) This can be modified to fit a pretty big range of tasks in my use cases and chop hours off my workflows. It's so much faster in DS workflows to just have extra cores and use them than to spin up a cluster for distributed compute.

TheLocehiliosan 3y ago

I often point people here to get some good background when doing concurrency in Python. https://realpython.com/python-concurrency/ https://realpython.com/python-concurrency/

culebron21 3y ago

I've just left the question open, but the best learning material for me on this was this lecture. Other points down the list, I had to figure out just by trial and error. Raymond Hettinger, Keynote on Concurrency, PyBay 2017 https://www.youtube.com/watch?v=9zinZmE3Ogk https://www.youtube.com/watch?v=9zinZmE3Ogk