14 ms·

Parallel Programming with Python

218 points by uaaa 8y ago

natvert 8y ago
Sweet, a guide! I always end up rolling my own thread pool / manager. I wish something like the parallel gem for Ruby existed in pyland...
- guiriduro 8y ago
  If your tasks are fairly coarse-grained (take >50ms each), Celery [1] has existed for a several years; takes a bit of setting up but works well, its very flexible. If your needs are simple, don't forget that your common or garden webserver can parallelize workloads too (distribute web requests to workers on multiple cores), it depends mostly on your client code for fan-out, and redis has worked well for synchronization for me. Nowadays you can also use serverless to parallelize coarse-grained workloads in the cloud. [1] http://www.celeryproject.org/ http://www.celeryproject.org/
elcombato 8y ago
> (note that you must be using Python 2 for this workshop and not using Python 3. Complete this workshop using Python 2, then read about the small changes if you are interested in using Python 3) Why using legacy Python for this?
- ggm 8y ago
  why not re-write the workshop for python3 and require python2 users to wear the pain downgrade brings?
  - brennebeck 8y ago
    Because python2 is a deprecated language that will EOL?
    - ggm 8y ago
      ok. If python2 is deprecated, why write a tutorial in python2 and say "python3 people can work it out"
- kjeetgill 8y ago
  I'm not sure it's fair to call it legacy just yet. Most linux distributions (minus Arch I think) still use 2.7 as the default. I get EOL/deprecation is here but lets not jump the gun to legacy just yet. I just see more 2 than 3 @ Day Job.
jillesvangurp 8y ago
Did they ever fix the global interpreter lock? Sort of a show stopper with doing stuff concurrently in python. I've done a bit of batch processing using the multi process module; which uses processes instead of threads. This works but it is a bit of a kludge if you are used to languages that support concurrency properly.
- armitron 8y ago
  They did not, which is why this "course" illustrates taking advantage of multiple cores via multiprocessing without mentioning the GIL at all. Which is a little misleading if you think about it. Also, by having the introductory chapter be about "functional programming" (which incidentally Python does not do well), he completely bypasses the serious issue of shared state. Which goes to show that parallelism in Python is more like a gimmick than a real-world solution since it doesn't let you do in-process shared-memory processing via threads in parallel which is so important for many applications. In my case, the vast majority of the time I do not want to farm workers out to different operating system processes and deal with serialization and communication, but this is the only way for Python code to take advantage of multiple cores [1]. [1] Another way is to write a module in C and have Python code call into it on a new thread and release the GIL while doing so, but of course this is even worse pain-wise than doing it with multiprocessing and you end up writing/compiling C.
  - devxpy 8y ago
    > deal with serialization and communication I thought a lot about this problem, for over 2 years, and came up with zproc https://github.com/pycampers/zproc https://github.com/pycampers/zproc Basically, > It lets you do message passing parallelism without the effort of tedious wiring. You'll be doing message passing without ever dealing with sockets! Also, Shared memory parallelism is hard to get right irregardless of which language you use. I would recommend strongly against it, unless you're writing some really really really niche thing where message passing is a bottleneck (it isn't most of the time)
    - TickleSteve 8y ago
mpweiher 8y ago
"...take advantage of the processing power of multicore processors" Step 1: stop using Python. "You can have a second core when you know how to use one" Now don't get me wrong, Python is a perfectly fine language for lots of things, but not for taking optimal advantage of the CPU. https://benchmarksgame-team.pages.debian.net/benchmarksgame/faster/python3-gcc.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Relative performance compared to C is somewhere between an order of magnitude or two slower. Considering how much harder and more error-prone multi-core is, maybe first try a fast sequential solution.
- auggierose 8y ago
  Yeah. Recently switched some Blender Python algorithms I wrote to Swift/Metal, and the speedup was somewhere between 1000 and 1000000 depending on the algorithm.
  - stevesimmons 8y ago
    Speedups of that magnitude suggest the original Python approach was particularly inefficient...
    - targafarian 8y ago
      Yeah, properly written Python is at extreme worst O(1000) times slower than speeding it up code with a Numpy/Numba/c/Fortran/etc. implementation. Brute-force loopy code in Python I've seen is 100x slower than the compiled alternatives. So I agree, these extreme numbers are the sign of writing the worst possible Python implementation of a thing and saying Python sucks.
    - auggierose 8y ago
      Not going to dispute that. If I spend time optimising code, I might do it as well in an environment like Swift/Metal instead of Python.
  - kilon 8y ago
    Who would have guessed that compiled, static, non-dynamic, hardware accelerated code would be a ton more performant than runtime, highly dynamic, garbage collected and very powerful code that is not hardware accelerated.
wenning 8y ago
i think use python3 multiprocess and async is better for product.
another-cuppa 8y ago
I think a lot of this complexity can be avoided by just writing single threaded python and using GNU parallel for running it on multiple cores. You can even trivially distribute the work across a cluster that way.
- quiq 8y ago
  This is the approach I've taken, albeit at the "top level" of the program. Since I know I don't have to deal with Windows I much prefer simply piping to parallel instead of xargs, or calling make -j8, or similarly letting some shell wrapper handle it over dealing with the overhead inside of python, especially multiprocessing. However, where I think having this stuff available inside of python is useful is that it's cross platform and consumable from "higher levels" of python. A library can do some mucky stuff internally to speed computation but still present a simple sync interface, all without external dependencies.
ram_rar 8y ago
I love python. But its seriously, incapable for doing non trivial concurrent tasks. Multiprocessing module doesnt count. I hope the python core-devs take some inspiration from golang for developing the right abstractions for concurrency.
- sidlls 8y ago
  While I agree that python isn't ideal beyond a certain scope, I think you're overstating how bad it is. My team and I have built a number of non-trivial machine learning products with pipelines that use both the ThreadPool and ProcessPool components successfully. The headaches we have are related more to the fact that Python is dynamic than its concurrency story.
  - weberc2 8y ago
    OP probably is overstating a bit, but it is hard to efficiently parallelize computation in Python. For example, if you have a large Python object graph that you need to compute over, you can't easily parallize the computation without paying some significant serialization cost. You can probably alleviate that by carefully choosing algorithms that minimize the amount of serialization per worker process, but at the end of the day, all of this is still quite a lot harder than using shared memory and goroutines. And not to mention Go is 1-2 orders of magnitude faster than Python in single-threaded execution... Python is great for lots of things, but efficient parallel programming in Python is _hard_, even if there are a handful of cases where it's not so hard.
- deleted 8y ago
  [deleted]
- azag0 8y ago
  Concurrent or parallel? For concurrency, python has asyncio, which many people consider a success. For parallel execution, there's the GIL, but in practice it rarely matters, because once you want to do parallel execution, you have most likely a computationally intensive task to do, at which point you call down to C or something, and then GIL doesn't matter.
quietbritishjim 8y ago
In response to the multiple comments here complaining that multithreading is impossible in Python without using multiple processes, because of the GIL (global interpreter lock): This is just not true, because C extension modules (i.e. libraries written to be used from Python but whose implementations are written in C) can release the global interpreter lock while inside a function call. Examples of these include numpy, scipy, pandas and tensorflow, and there are many others. Most Python processes that are doing CPU-intensive computation spend relatively little time actually executing Python, and are really just coordinating the C libraries (e.g. "mutiply these two matrices together"). The GIL is also released during IO operations like writing to a file or waiting for a subprocess to finish or send data down its pipe. So in most practical situations where you have a performance-critical application written in Python (or more precisely, the top layer is written in Python), multithreading works fine. If you are doing CPU intensive work in pure Python and you find things are unacceptably slow, then the simplest way to boost performance (and probably simplify your code) is to rewrite chunks of your code in terms of these C extension modules. If you can't do this for some reason then you will have to throw in the Python towel and re-write some or all of your code in a natively compiled language (if it's just a small fraction of your code then Cython is a good option). But this is the best course of action regardless of the threads situation, because pure Python code runs orders of magnitude slower than native code.
- Rotareti 8y ago
  Does anyone know how well Python and Rust team up compared to Python and C in practice?
  - ikornaselur 8y ago
    I've yet to play with beyond just experimenting a little bit, but it seems it works very well. I've mainly been looking at these resources: https://github.com/rochacbruno/rust-python-example https://github.com/rochacbruno/rust-python-example https://github.com/PyO3/pyo3 https://github.com/PyO3/pyo3 Though I have not done rust <-> python in real practice
  - jwandborg 8y ago
ilovetux 8y ago
I find it strange that nobody ever seems to mention python's concurrent.futures module [0] which is new in Python 3.2. I think asyncio got a lot of attention when it came out in Python 3.4 and concurrent.futures took a back seat. This article also doesn't mention the module in it's Python 2 and 3 differences link. asyncio is a good library for asyncronous I/O but concurrent.futures gives us some pretty nifty tooling which makes concurrent programming (with ThreadPoolExecutor) and parallel programming (with ProcessPoolExecutor) pretty easy to get right. The Future class is a pretty elegant solution for continuing execution while a background task is being executed. [0] https://docs.python.org/3/library/concurrent.futures.html https://docs.python.org/3/library/concurrent.futures.html
- ZeroCool2u 8y ago
  ThreadPoolExecutor and ProcessPoolExecutor were exactly what I was waiting for someone to mention. I was doing some Python as a systems architect at my previous position and now as a full time data scientist, my life has pretty much been consumed by Python. Unsurprisingly, a lot of my initial work is retrieving and cleaning very large volumes of data, the later usually being I/O bound and the former being CPU bound and frankly myself and a lot of my team immediately default to using both ThreadPoolExecutor and ProcessPoolExecutor respectively, because of how simple and performant they are. Perhaps asyncio is more familiar terminology to people coming from Web Dev, so that's why they're gravitating towards it, but there are few times when I find myself needing that particular tooling outside of Web Dev anyways.
walterstucco 8y ago
> Parallel Programming with Python? What about no? Don't get me wrong, i don't like Python as a language, but it's a fine tool and many useful programs have been written with it But parallel programming? No, thanks.
goerz 8y ago
The GIL has considerable benefits: I don’t have to worry about whether Python functions are thread-safe. Thread-based parallelism is hard to get right, and given the number of workarounds, Python’s GIL is a total non-issue.
- jashmatthews 8y ago
  > The GIL has considerable benefits: I don’t have to worry about whether Python functions are thread-safe. Hold on, the GIL doesn't make Python automatically thread-safe! You can still have classic data races as the VM can pause and resume two threads writing to the same variable.
  - goerz 8y ago
    Can you elaborate on that? Is there a blog post somewhere that illustrates the problem you're talking about? I was under the assumption that Python interpreters run single-threaded.
- devxpy 8y ago
  Small correction: It makes the _implementation_ thread-safe. It also simplifies a lot of CPython code, making it a lot easier to maintain.
kilon 8y ago
One more epic discussion on Python, where we have the unique opportunity to learn that using C libraries from Python is "cheating". I could not agree more It's definitely cheating to use C code with the exception of most Python libraries that already are to a large extent nothing more than thin wrappers over existing C libraries or the tiny fact that the most popular by far implementation of Python , CPython, is almost 50% implemented in the C language, including the standard library.The author even dared include "C" in the name of the implementation. Those cheaters, becoming bolder and bolder every day. Damn them !!!
- deleted 8y ago
  [deleted]
magwa101 8y ago
Concurrency in python always ends up the reason to drop it and reimplement in Go. Also, the code ends up littered with type checks....
andbberger 8y ago
IMO ray[1] is the greatest thing to happen in python parallelism since the invention of sliced bread. Also includes best currently available hyperparameter tuning framework! [1] https://github.com/ray-project/ray https://github.com/ray-project/ray
mwyau 8y ago
mpi4py should be included. It's a wrapper for the MPI library, which is the de facto standard for scientific computing: https://mpi4py.readthedocs.io/en/stable/ https://mpi4py.readthedocs.io/en/stable/
gnufx 8y ago
Multi-core parallelism isn't so interesting for serious computation. You want to be able to use large distributed HPC systems, but Python doesn't seem to have the equivalent of https://pbdr.org https://pbdr.org for R.

218 points by uaaa 8y ago

Sweet, a guide! I always end up rolling my own thread pool / manager. I wish something like the parallel gem for Ruby existed in pyland...

guiriduro 8y ago
If your tasks are fairly coarse-grained (take >50ms each), Celery [1] has existed for a several years; takes a bit of setting up but works well, its very flexible. If your needs are simple, don't forget that your common or garden webserver can parallelize workloads too (distribute web requests to workers on multiple cores), it depends mostly on your client code for fan-out, and redis has worked well for synchronization for me. Nowadays you can also use serverless to parallelize coarse-grained workloads in the cloud. [1] http://www.celeryproject.org/ http://www.celeryproject.org/

elcombato 8y ago

> (note that you must be using Python 2 for this workshop and not using Python 3. Complete this workshop using Python 2, then read about the small changes if you are interested in using Python 3) Why using legacy Python for this?

ggm 8y ago
why not re-write the workshop for python3 and require python2 users to wear the pain downgrade brings?
- brennebeck 8y ago
  Because python2 is a deprecated language that will EOL?
  - ggm 8y ago
    ok. If python2 is deprecated, why write a tutorial in python2 and say "python3 people can work it out"
kjeetgill 8y ago
I'm not sure it's fair to call it legacy just yet. Most linux distributions (minus Arch I think) still use 2.7 as the default. I get EOL/deprecation is here but lets not jump the gun to legacy just yet. I just see more 2 than 3 @ Day Job.

jillesvangurp 8y ago

Did they ever fix the global interpreter lock? Sort of a show stopper with doing stuff concurrently in python. I've done a bit of batch processing using the multi process module; which uses processes instead of threads. This works but it is a bit of a kludge if you are used to languages that support concurrency properly.

armitron 8y ago
They did not, which is why this "course" illustrates taking advantage of multiple cores via multiprocessing without mentioning the GIL at all. Which is a little misleading if you think about it. Also, by having the introductory chapter be about "functional programming" (which incidentally Python does not do well), he completely bypasses the serious issue of shared state. Which goes to show that parallelism in Python is more like a gimmick than a real-world solution since it doesn't let you do in-process shared-memory processing via threads in parallel which is so important for many applications. In my case, the vast majority of the time I do not want to farm workers out to different operating system processes and deal with serialization and communication, but this is the only way for Python code to take advantage of multiple cores [1]. [1] Another way is to write a module in C and have Python code call into it on a new thread and release the GIL while doing so, but of course this is even worse pain-wise than doing it with multiprocessing and you end up writing/compiling C.
- devxpy 8y ago
  > deal with serialization and communication I thought a lot about this problem, for over 2 years, and came up with zproc https://github.com/pycampers/zproc https://github.com/pycampers/zproc Basically, > It lets you do message passing parallelism without the effort of tedious wiring. You'll be doing message passing without ever dealing with sockets! Also, Shared memory parallelism is hard to get right irregardless of which language you use. I would recommend strongly against it, unless you're writing some really really really niche thing where message passing is a bottleneck (it isn't most of the time)
  - TickleSteve 8y ago

mpweiher 8y ago

"...take advantage of the processing power of multicore processors" Step 1: stop using Python. "You can have a second core when you know how to use one" Now don't get me wrong, Python is a perfectly fine language for lots of things, but not for taking optimal advantage of the CPU. https://benchmarksgame-team.pages.debian.net/benchmarksgame/faster/python3-gcc.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Relative performance compared to C is somewhere between an order of magnitude or two slower. Considering how much harder and more error-prone multi-core is, maybe first try a fast sequential solution.

auggierose 8y ago
Yeah. Recently switched some Blender Python algorithms I wrote to Swift/Metal, and the speedup was somewhere between 1000 and 1000000 depending on the algorithm.
- stevesimmons 8y ago
  Speedups of that magnitude suggest the original Python approach was particularly inefficient...
  - targafarian 8y ago
    Yeah, properly written Python is at extreme worst O(1000) times slower than speeding it up code with a Numpy/Numba/c/Fortran/etc. implementation. Brute-force loopy code in Python I've seen is 100x slower than the compiled alternatives. So I agree, these extreme numbers are the sign of writing the worst possible Python implementation of a thing and saying Python sucks.
  - auggierose 8y ago
    Not going to dispute that. If I spend time optimising code, I might do it as well in an environment like Swift/Metal instead of Python.
- kilon 8y ago
  Who would have guessed that compiled, static, non-dynamic, hardware accelerated code would be a ton more performant than runtime, highly dynamic, garbage collected and very powerful code that is not hardware accelerated.

wenning 8y ago

i think use python3 multiprocess and async is better for product.

another-cuppa 8y ago

I think a lot of this complexity can be avoided by just writing single threaded python and using GNU parallel for running it on multiple cores. You can even trivially distribute the work across a cluster that way.

quiq 8y ago
This is the approach I've taken, albeit at the "top level" of the program. Since I know I don't have to deal with Windows I much prefer simply piping to parallel instead of xargs, or calling make -j8, or similarly letting some shell wrapper handle it over dealing with the overhead inside of python, especially multiprocessing. However, where I think having this stuff available inside of python is useful is that it's cross platform and consumable from "higher levels" of python. A library can do some mucky stuff internally to speed computation but still present a simple sync interface, all without external dependencies.

ram_rar 8y ago

I love python. But its seriously, incapable for doing non trivial concurrent tasks. Multiprocessing module doesnt count. I hope the python core-devs take some inspiration from golang for developing the right abstractions for concurrency.

sidlls 8y ago
While I agree that python isn't ideal beyond a certain scope, I think you're overstating how bad it is. My team and I have built a number of non-trivial machine learning products with pipelines that use both the ThreadPool and ProcessPool components successfully. The headaches we have are related more to the fact that Python is dynamic than its concurrency story.
- weberc2 8y ago
  OP probably is overstating a bit, but it is hard to efficiently parallelize computation in Python. For example, if you have a large Python object graph that you need to compute over, you can't easily parallize the computation without paying some significant serialization cost. You can probably alleviate that by carefully choosing algorithms that minimize the amount of serialization per worker process, but at the end of the day, all of this is still quite a lot harder than using shared memory and goroutines. And not to mention Go is 1-2 orders of magnitude faster than Python in single-threaded execution... Python is great for lots of things, but efficient parallel programming in Python is _hard_, even if there are a handful of cases where it's not so hard.
deleted 8y ago
[deleted]
azag0 8y ago
Concurrent or parallel? For concurrency, python has asyncio, which many people consider a success. For parallel execution, there's the GIL, but in practice it rarely matters, because once you want to do parallel execution, you have most likely a computationally intensive task to do, at which point you call down to C or something, and then GIL doesn't matter.

quietbritishjim 8y ago

In response to the multiple comments here complaining that multithreading is impossible in Python without using multiple processes, because of the GIL (global interpreter lock): This is just not true, because C extension modules (i.e. libraries written to be used from Python but whose implementations are written in C) can release the global interpreter lock while inside a function call. Examples of these include numpy, scipy, pandas and tensorflow, and there are many others. Most Python processes that are doing CPU-intensive computation spend relatively little time actually executing Python, and are really just coordinating the C libraries (e.g. "mutiply these two matrices together"). The GIL is also released during IO operations like writing to a file or waiting for a subprocess to finish or send data down its pipe. So in most practical situations where you have a performance-critical application written in Python (or more precisely, the top layer is written in Python), multithreading works fine. If you are doing CPU intensive work in pure Python and you find things are unacceptably slow, then the simplest way to boost performance (and probably simplify your code) is to rewrite chunks of your code in terms of these C extension modules. If you can't do this for some reason then you will have to throw in the Python towel and re-write some or all of your code in a natively compiled language (if it's just a small fraction of your code then Cython is a good option). But this is the best course of action regardless of the threads situation, because pure Python code runs orders of magnitude slower than native code.

Rotareti 8y ago
Does anyone know how well Python and Rust team up compared to Python and C in practice?
- ikornaselur 8y ago
  I've yet to play with beyond just experimenting a little bit, but it seems it works very well. I've mainly been looking at these resources: https://github.com/rochacbruno/rust-python-example https://github.com/rochacbruno/rust-python-example https://github.com/PyO3/pyo3 https://github.com/PyO3/pyo3 Though I have not done rust <-> python in real practice
- jwandborg 8y ago

ilovetux 8y ago

I find it strange that nobody ever seems to mention python's concurrent.futures module [0] which is new in Python 3.2. I think asyncio got a lot of attention when it came out in Python 3.4 and concurrent.futures took a back seat. This article also doesn't mention the module in it's Python 2 and 3 differences link. asyncio is a good library for asyncronous I/O but concurrent.futures gives us some pretty nifty tooling which makes concurrent programming (with ThreadPoolExecutor) and parallel programming (with ProcessPoolExecutor) pretty easy to get right. The Future class is a pretty elegant solution for continuing execution while a background task is being executed. [0] https://docs.python.org/3/library/concurrent.futures.html https://docs.python.org/3/library/concurrent.futures.html

ZeroCool2u 8y ago
ThreadPoolExecutor and ProcessPoolExecutor were exactly what I was waiting for someone to mention. I was doing some Python as a systems architect at my previous position and now as a full time data scientist, my life has pretty much been consumed by Python. Unsurprisingly, a lot of my initial work is retrieving and cleaning very large volumes of data, the later usually being I/O bound and the former being CPU bound and frankly myself and a lot of my team immediately default to using both ThreadPoolExecutor and ProcessPoolExecutor respectively, because of how simple and performant they are. Perhaps asyncio is more familiar terminology to people coming from Web Dev, so that's why they're gravitating towards it, but there are few times when I find myself needing that particular tooling outside of Web Dev anyways.

walterstucco 8y ago

> Parallel Programming with Python? What about no? Don't get me wrong, i don't like Python as a language, but it's a fine tool and many useful programs have been written with it But parallel programming? No, thanks.

goerz 8y ago

The GIL has considerable benefits: I don’t have to worry about whether Python functions are thread-safe. Thread-based parallelism is hard to get right, and given the number of workarounds, Python’s GIL is a total non-issue.

jashmatthews 8y ago
> The GIL has considerable benefits: I don’t have to worry about whether Python functions are thread-safe. Hold on, the GIL doesn't make Python automatically thread-safe! You can still have classic data races as the VM can pause and resume two threads writing to the same variable.
- goerz 8y ago
  Can you elaborate on that? Is there a blog post somewhere that illustrates the problem you're talking about? I was under the assumption that Python interpreters run single-threaded.
devxpy 8y ago
Small correction: It makes the _implementation_ thread-safe. It also simplifies a lot of CPython code, making it a lot easier to maintain.

kilon 8y ago

One more epic discussion on Python, where we have the unique opportunity to learn that using C libraries from Python is "cheating". I could not agree more It's definitely cheating to use C code with the exception of most Python libraries that already are to a large extent nothing more than thin wrappers over existing C libraries or the tiny fact that the most popular by far implementation of Python , CPython, is almost 50% implemented in the C language, including the standard library.The author even dared include "C" in the name of the implementation. Those cheaters, becoming bolder and bolder every day. Damn them !!!

deleted 8y ago
[deleted]

magwa101 8y ago

Concurrency in python always ends up the reason to drop it and reimplement in Go. Also, the code ends up littered with type checks....

andbberger 8y ago

IMO ray[1] is the greatest thing to happen in python parallelism since the invention of sliced bread. Also includes best currently available hyperparameter tuning framework! [1] https://github.com/ray-project/ray https://github.com/ray-project/ray

mwyau 8y ago

mpi4py should be included. It's a wrapper for the MPI library, which is the de facto standard for scientific computing: https://mpi4py.readthedocs.io/en/stable/ https://mpi4py.readthedocs.io/en/stable/

gnufx 8y ago

Multi-core parallelism isn't so interesting for serious computation. You want to be able to use large distributed HPC systems, but Python doesn't seem to have the equivalent of https://pbdr.org https://pbdr.org for R.