9 ms·

How to choose the right Python concurrency API

134 points by EntICOnc 4y ago

shankr 4y ago
I always struggle with this. Even though I understand IO-bound and CPU-bound tasks, it's not always one or the other when you build an application. For example, I have to train a machine learning model where I need to query a database or read some files which are considered to be IO bound. But then when I am training a huge model it will become a CPU bound task. How do you go about this? Would you then separate the logic in such a way that you can use both multi-processing and multi-threading/co-routine. Is this possible when using async? I have limited experience but it feels like the moment you introduce async everything has to be async in the code.
- DrScientist 4y ago
  Queues. If you want to do currency programming and you want to pass data between different tasks, then thread safe queues are quite a simple effective way to orchestrate the demand complexity in my experience. After all that's how parallel processing is managed in the Post Office.
- uniqueuid 4y ago
  To be honest, your best cheap shot for making pipelines fast is to get all your data into RAM and then run your models. Ingestion i/o has lots of surprising bottlenecks, from small file i/o to NFS to decoding e.g. of image/video frames. If you can afford it, create a standardized representation for your data and keep it in memory as much as possible. If that's not feasible, write the parsed representation into uncompressed tar files and load these on batch start.
uniqueuid 4y ago
The article gives a good summary of the quite complex landscape of concurrency in python. There's more to it, for example gil-free c-extensions, subprocesses and cross-machine (plus IPC) communication. But I'm particularly bothered by the fact that many articles and tutorials look at concurrency as if it's only about factoring primes or writing a web server with many (perhaps even idempotent) parallel requests. In reality, people will often want and need to combine multiple of these approaches, and then it gets VERY messy. I.e. try to combine a multiprocessing executor with multiple asyncio loops and boom you're in some very deep waters. One project that does this (async loops inside multiple processes) is proxy.py - very enlightening to read its code base [1]. But I really, really wish python would do more to provide simple and robust abstractions for these kinds of tasks. My dream would be a robust actor system similar to erlang, but we'll probably never get that. [1] https://github.com/abhinavsingh/proxy.py https://github.com/abhinavsingh/proxy.py
- matsemann 4y ago
  It's actually frustratingly complex. Right now I'm working on a "bridge" that receives http requests, and then needs to send that on an existing websocket connection to another system then wait for some responses, send some more on the ws etc and keep monitoring the ws. So the idea is basically to have multiple permanent websocket workers (one for each machine we need to speak with), that get tasks sent to them. Some added complexity is that each machine can only ever have one socket opened at a time. If I do it using asyncio, I end up with the issue of GIL making it so that I can't really do stuff concurrently, which sucks as the number of incoming requests and machines I need to bridge to increases. Or if I do it using multiple or sub-processes they can no longer communicate, I may risk having multiple processes trying to establish a connection, or processes dying I have to handle manually vs k8s doing it. Or I can do it with different deploys, and having a queue/db or whatever inbetween, drastically increasing complexity. So hard to combine different modes in the same app. While in java land I would just have fired up a webserver and made a thread+worker per ws connection and it would've scaled to thousands without issue.
  - mrweasel 4y ago
    Then why not use Java? I love Python, it’s my language of choice for most things, but as you rightly point out, for some things it just invites complexity.
    - rmbyrro 4y ago
      Same I was thinking. As it is today, Python just isn't a (relatively) good tool for concurrency. Use Go, Java. Clojure has some of the most interesting abstractions in this area, but the lisp syntax scares people away.
  - gspetr 4y ago
    How do you test all of it?
  - uniqueuid 4y ago
    Part of the problem - to me at least - is that in practice, concurrency with python is pretty much only heavy (slow) multiprocessing.queues. It's what multiprocessing.pool and the corresponding executors use. At the same time, my experience (along with that of many others) has been that these queues are not exactly robust - there seem to be many horrible edge cases where queues and/or processes get stuck and it's really hard to have a robust all-python app that recovers from e.g. a dead child (!). Ask ML frameworks, for example, how they handle spawning while holding GPU contexts and the like. It's a nightmare. So if we had a single robust, elegant and half-way fast abstraction for IPC, that would solve a lot of the existing issues. And it needs to be bi-directional of course.
XorNot 4y ago
I kind of wish we'd just add Go-style CSP to python and call it a day - it seems to be the best of both worlds when it comes "do two things at the same time".
- TOMDM 4y ago
  Go's concurrency is such a breath of fresh air whenever I interact with it. Python would be immediately so much easier to use for so many other cases if it implemented something similar.
- meowface 4y ago
  I'm not sure how much it replicates the CSP model, but the closest thing I've found to Go-style concurrency in Python is gevent: https://github.com/gevent/gevent https://github.com/gevent/gevent I personally still prefer to use it in all my projects.
  - Mlller 4y ago
    I do, too, for websites, although I try to avoid dependencies like a snowflake the fire. But doing nothing more than using gevent server and putting from gevent import monkey monkey.patch_all() at the top of the main module was such a game changer.
- Yaser_Amiri 4y ago
  I actually tried to build a CSP interface on top of AsyncIO, It's still experimental an I haven't developed it for sometime but you know, It works. https://github.com/Yaser-Amiri/one-ring/blob/main/docs/samples.rst https://github.com/Yaser-Amiri/one-ring/blob/main/docs/sampl... As long as GIL exists with current conditions, we will not have a Go-like CSP, but that's about scheduling, not the CSP itself.
  - XorNot 4y ago
    Well that's the thought: GIL-less Python is coming, at which point we can actually have something like this and it'll just work.
ramchip 4y ago
I didn't see gevent mentioned. Did asyncio make it obsolete?
- est 4y ago
  Gevent4life!
RhysU 4y ago
The articles' discussion on multiprocessing approaches links to [1] on Pool vs ProcessPoolExecutor. I wish that rather recommendations for non-standard libraries appeared. Several years ago, I spent an inordinate amount of time chasing down multiprocessing-usage edge cases. For example: How can I reliably learn that my subprocess died? How can I avoid a fork bomb if my work itself creates new work? Sane serialization semantics? Type hinting? Consistent behavior for spawn, fork, and forkserver? Register callbacks on Futures to run eagerly at completion? Etc. I ultimately wrote a single-file, drop-it-anywhere “jobserver“ [2] to remove many footcanons. It aims to be the well-tested core inside any more ergonomic API enforcing any additional semantics one might want. No canceling, but that could probably be added. My "jobserver" is loky-esque [3] as I believe they and I independently were fighting similar challenges but, of course, I prefer mine. Implementation is 560 lines including doc strings. Tests, in the same file, are about another 600 lines. [1] https://superfastpython.com/multiprocessing-pool-vs-processpoolexecutor/ https://superfastpython.com/multiprocessing-pool-vs-processp... [2] https://github.com/RhysU/jobserver/blob/master/jobserver.py#L7-L28 https://github.com/RhysU/jobserver/blob/master/jobserver.py#... [3] https://loky.readthedocs.io/en/stable/ https://loky.readthedocs.io/en/stable/
crabbygrabby 4y ago
Super good lay person summary. I like how this explains the underlying concepts more than the python specific stuff. Will recommend this to new comers.
jonstewart 4y ago
If Python is slow and you need concurrency, and you have a long-ish timeframe in mind, do yourself a favor and use one of C++/Java/Rust/Go/Erlang/Elixir/(other newer programming language). Especially, don’t rely on Multiprocessing. I’ve been developing and maintaining a largish batch processing Python project where performance is a key feature, and it’s been very frustrating. I should have rewritten it when I had the chance, but I’m slowly outsourcing critical pieces to C++ and Rust libraries. Multiprocessing has been a source of subtle portability errors as it has changed over different minor versions.
- wenc 4y ago
  If you need to stick with Python, Joblib is more reliable and works better than Multiprocessing in my experience. I ran into so many subtle quirks that needed workarounds in Multiprocessing. Joblib.parallel with the Loky backend just seems to be Multiprocessing done right.
- jeremycarter 4y ago
  I have a similar project and now have to split up the single application to actually get the muli core processing I demand. I regret not using Rust/Go... or even just .NET.
- ansre 4y ago
  [dead]
  - jonstewart 4y ago
    Scaling horizontally is well and good (since my unit of input is large and I get a lot of them, I just use autoscaling groups in AWS to achieve much the same), but I don’t want to pay for 10 systems slowly doing what 1 could do with fast software. Especially, while 1 system has a hard ceiling, there’s a lot of headroom on 1 system when you achieve mechanical sympathy with the cache hierarchy, memory controller, and disk.
- capableweb 4y ago
  Hear hear! If you already know Python, picking up the basics of Java/Rust/Go/Elixir is trivial. Picking up the basics of C++ would be slightly harder, but not impossible. Erlang even further away, at least syntax wise. But, you'll be doing future you a huge service for whenever you need to reach for hard performance/latency and/or concurrency/parallelism again.
jhokanson 4y ago
"Firstly, there are three main Python concurrency APIs, they are:" asyncio, threading, multiprocessing, ... oh, and concurrent.futures All kidding aside, I used the multiprocessing module lately and it was a mess. Do I want 'map', 'starmap', 'imap', etc.? All I wanted was to run a function multiple times with different inputs (and multiple inputs per function call) and to fail when any launched process failed rather than waiting for every input variation to execute and then telling me about the error(which honestly I didn't think was asking for too much).
- jayemar 4y ago
  There's a lot of functionality in the multiprocessing library and it has its own problems, but I wouldn't call it a "mess" from your description. map, starmap, and imap are all useful for different applications depending on your usage and priorities. I'll agree that it can sometimes be difficult to understand the differences and which is best for your use case, but having used in different use cases I certainly appreciate the different functions available.
O5vYtytb 4y ago
I will say that structured concurrency (coroutine based) via Trio/Anyio is, in my experience, so much better for most applications that can support it. Reasoning/testing threading code is so impossible once a project gets large enough. I'd highly recommend reading https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ https://vorpus.org/blog/notes-on-structured-concurrency-or-g... I work with trio/anyio on a daily basis for my job, and I'd always recommend people use it for their concurrency framework and then use await anyio.to_thread.run_sync() to spawn threads if needed. I'll also add that if you need multi-processor execution, look at https://pypi.org/project/tractor/ https://pypi.org/project/tractor/
- infinite8s 4y ago
  +1 for trio. I used it in a past job to proxy a socket-based API to websockets and it was easy to write and debug. Tractor is also architected similar to Erlangs OTP system, which trio with it's structured concurrency makes easy to reason about.
zach_garwood 4y ago
Every time I read about the difficulties in writing concurrent Python, it reminds me of the joke: Patient: "Doc, it hurts when I do this!" Doctor: "Well, stop doing that." I love me some Python, but it's just not good at this. When your requirements dictate that you need concurrency, you should start evaluating a different language that is designed for it, in order to build that part of your system.
fulafel 4y ago
This glosses over the distinction between concurrency and parallelism, and the traditional & still common "pool of server processes" parallelism option.
pdimitar 4y ago
Or here's a crazy idea, maybe Python is not a holy gospel and you shouldn't use it for things that it's obviously badly suited for. You have a GIL in there, what other subtle hints do you need that your scenario is not well-supported? Erlang/Elixir and Rust will serve you perfectly for concurrent and parallel programming. OCaml 5.0, once stabilized, will do so as well. This Python worshipping and trying to use it for everything just betrays amateur programmers underneath. Whatever happened to "use the right tool for the job"?
- jeremycarter 4y ago
  This is the correct answer. The love affair to do everything in python must end this year!
  - Frotag 4y ago
    But it does almost everything! And it's especially painless if you're not building for scale, and just hobby projects. But yeah, stuff like concurrency and type annotations seem to go against some of Python's fundamental designs (GIL and syntax) so are kinda painful at times.
    - pdimitar 4y ago
      > But it does almost everything! Ehhh... does it though? I've been doing shell scripts all my career and never stumbled upon a problem I couldn't solve by combining various tools. I did a lot of homegrown data science that way. With Python it seems it's mostly familiarity. People just love the idea of one universal tool, no matter how many times history proves there's no such thing.
vpribish 4y ago
content of article is ok, but why is it written like an SEO recipe website?
malkia 4y ago
Any resolution to this (python's multiprocessing on Windows and WaitForMultipleObjects failing when the number of objects (threads) to wait > 60) https://stackoverflow.com/questions/65252807/multiprocessing-pool-pool-on-windows-cpu-limit-of-63 https://stackoverflow.com/questions/65252807/multiprocessing... Just curious, as yesterday had to deal with the `black` formatter, and it had this hack into it to prevent no more than 60 being created.
girfan 4y ago
This is a great analysis; thanks for writing. I have also been working on running multiple Python interpreters in the same process by isolating them in different namespaces using `dlmopen` [1]. The objective on a high level is to receive requests for some compute intensive operations from a TCP/HTTP server and dispatch them on to different workers. In this case, a thin C++ shim receives the requests and dispatches them on to one of the Python interpreters in a namespace. This eliminates contention for the GIL amongst the interpreters and can exploit parallelism by running each interpreter on a different set of cores. The data obtained from the request does not need to be copied into the interpreter because everything is in the same address space; similarly the output produced by the Python interpreter is also just passed back without any copies to the server. [1] https://www.man7.org/linux/man-pages/man3/dlmopen.3.html https://www.man7.org/linux/man-pages/man3/dlmopen.3.html
toolslive 4y ago
Is stackless still an alternative? (It used to be quite hot 1.5 decade ago) https://github.com/stackless-dev/stackless/wiki/ https://github.com/stackless-dev/stackless/wiki/
mistrial9 4y ago
zero mention of DASK here, is it that obscure?
gshulegaard 4y ago
I am not sure when this article was written, but I really wouldn't recommend threads for even IO-bound tasks. Since threads don't escape the GIL the added contention with OS thread scheduling (context switching) is a performance overhead that coroutines (asyncio) don't have. I haven't benchmarked threads vs. coroutines for IO-bound tasks...but my gut feeling is coroutines are going to generally be better because of the lack of thread switching overhead. So for me, there really are only 2 concurrency choices: coroutines or multiprocessing. And generally, if I find myself reaching for multiprocessing I seriously evaluate if the logic shouldn't be ported to a different language.
- lelandbatey 4y ago
  I wouldn't quite say that there's only 2 concurrency choices, but I agree that there's only 2 concurrency choices in Python if you are operating under heavy load. The multi-threaded concurrency style can be a great choice if your performance requirements are low and you only want a small amount of concurrency. Then it can be very easy to emulate a Golang-esq model, by having just a few threads which are communicating by writing data via queues to one another.
- hbrn 4y ago
  But threads do release GIL on IO. There's a slight overhead of course, but if you're not running hundreds of servers and thousands of threads, the overhead can be negligible. I've seen more than once people rewriting thread-based code to coroutines, despite not having any noticeable benefits other than being more sexy. In fact I was one of those people almost a decade ago, rewriting the whole networking portion of the app only to achieve ~5% performance improvement. One should always measure the overhead instead assuming how big it is.
  - gshulegaard 4y ago
    I am not expert, and this might be dated information, but historically there is a "thrashing" introduced between the GIL and OS scheduled threads which has surprising performance implications. David Beazley does a much better illustrating the nuances than I ever could: https://www.youtube.com/watch?v=Obt-vMVdM8s https://www.youtube.com/watch?v=Obt-vMVdM8s Including a wild phenomenon where a threaded Python program's performance can degrade by increasing number of CPU cores available to a system. Coroutines avoid this by not requiring individual thread scheduling by the OS. So they effectively release the GIL like threads on IO but avoid OS level thread switching.
vector57 4y ago
I will briefly plug my library `unsync` (https://github.com/alex-sherman/unsync#quick-overview https://github.com/alex-sherman/unsync#quick-overview) which wraps all these methods (multiprocessing/threading/asyncio) into singular/simple-ish API. It's a bit overly simple, but it's helped a few times writing code the makes use of several concurrency methods and combining them together etc.

134 points by EntICOnc 4y ago

I always struggle with this. Even though I understand IO-bound and CPU-bound tasks, it's not always one or the other when you build an application. For example, I have to train a machine learning model where I need to query a database or read some files which are considered to be IO bound. But then when I am training a huge model it will become a CPU bound task. How do you go about this? Would you then separate the logic in such a way that you can use both multi-processing and multi-threading/co-routine. Is this possible when using async? I have limited experience but it feels like the moment you introduce async everything has to be async in the code.

DrScientist 4y ago
Queues. If you want to do currency programming and you want to pass data between different tasks, then thread safe queues are quite a simple effective way to orchestrate the demand complexity in my experience. After all that's how parallel processing is managed in the Post Office.
uniqueuid 4y ago
To be honest, your best cheap shot for making pipelines fast is to get all your data into RAM and then run your models. Ingestion i/o has lots of surprising bottlenecks, from small file i/o to NFS to decoding e.g. of image/video frames. If you can afford it, create a standardized representation for your data and keep it in memory as much as possible. If that's not feasible, write the parsed representation into uncompressed tar files and load these on batch start.

uniqueuid 4y ago

The article gives a good summary of the quite complex landscape of concurrency in python. There's more to it, for example gil-free c-extensions, subprocesses and cross-machine (plus IPC) communication. But I'm particularly bothered by the fact that many articles and tutorials look at concurrency as if it's only about factoring primes or writing a web server with many (perhaps even idempotent) parallel requests. In reality, people will often want and need to combine multiple of these approaches, and then it gets VERY messy. I.e. try to combine a multiprocessing executor with multiple asyncio loops and boom you're in some very deep waters. One project that does this (async loops inside multiple processes) is proxy.py - very enlightening to read its code base [1]. But I really, really wish python would do more to provide simple and robust abstractions for these kinds of tasks. My dream would be a robust actor system similar to erlang, but we'll probably never get that. [1] https://github.com/abhinavsingh/proxy.py https://github.com/abhinavsingh/proxy.py

matsemann 4y ago
It's actually frustratingly complex. Right now I'm working on a "bridge" that receives http requests, and then needs to send that on an existing websocket connection to another system then wait for some responses, send some more on the ws etc and keep monitoring the ws. So the idea is basically to have multiple permanent websocket workers (one for each machine we need to speak with), that get tasks sent to them. Some added complexity is that each machine can only ever have one socket opened at a time. If I do it using asyncio, I end up with the issue of GIL making it so that I can't really do stuff concurrently, which sucks as the number of incoming requests and machines I need to bridge to increases. Or if I do it using multiple or sub-processes they can no longer communicate, I may risk having multiple processes trying to establish a connection, or processes dying I have to handle manually vs k8s doing it. Or I can do it with different deploys, and having a queue/db or whatever inbetween, drastically increasing complexity. So hard to combine different modes in the same app. While in java land I would just have fired up a webserver and made a thread+worker per ws connection and it would've scaled to thousands without issue.
- mrweasel 4y ago
  Then why not use Java? I love Python, it’s my language of choice for most things, but as you rightly point out, for some things it just invites complexity.
  - rmbyrro 4y ago
    Same I was thinking. As it is today, Python just isn't a (relatively) good tool for concurrency. Use Go, Java. Clojure has some of the most interesting abstractions in this area, but the lisp syntax scares people away.
- gspetr 4y ago
  How do you test all of it?
- uniqueuid 4y ago
  Part of the problem - to me at least - is that in practice, concurrency with python is pretty much only heavy (slow) multiprocessing.queues. It's what multiprocessing.pool and the corresponding executors use. At the same time, my experience (along with that of many others) has been that these queues are not exactly robust - there seem to be many horrible edge cases where queues and/or processes get stuck and it's really hard to have a robust all-python app that recovers from e.g. a dead child (!). Ask ML frameworks, for example, how they handle spawning while holding GPU contexts and the like. It's a nightmare. So if we had a single robust, elegant and half-way fast abstraction for IPC, that would solve a lot of the existing issues. And it needs to be bi-directional of course.

XorNot 4y ago

I kind of wish we'd just add Go-style CSP to python and call it a day - it seems to be the best of both worlds when it comes "do two things at the same time".

TOMDM 4y ago
Go's concurrency is such a breath of fresh air whenever I interact with it. Python would be immediately so much easier to use for so many other cases if it implemented something similar.
meowface 4y ago
I'm not sure how much it replicates the CSP model, but the closest thing I've found to Go-style concurrency in Python is gevent: https://github.com/gevent/gevent https://github.com/gevent/gevent I personally still prefer to use it in all my projects.
- Mlller 4y ago
  I do, too, for websites, although I try to avoid dependencies like a snowflake the fire. But doing nothing more than using gevent server and putting from gevent import monkey monkey.patch_all() at the top of the main module was such a game changer.
Yaser_Amiri 4y ago
I actually tried to build a CSP interface on top of AsyncIO, It's still experimental an I haven't developed it for sometime but you know, It works. https://github.com/Yaser-Amiri/one-ring/blob/main/docs/samples.rst https://github.com/Yaser-Amiri/one-ring/blob/main/docs/sampl... As long as GIL exists with current conditions, we will not have a Go-like CSP, but that's about scheduling, not the CSP itself.
- XorNot 4y ago
  Well that's the thought: GIL-less Python is coming, at which point we can actually have something like this and it'll just work.

ramchip 4y ago

I didn't see gevent mentioned. Did asyncio make it obsolete?

est 4y ago
Gevent4life!

RhysU 4y ago

The articles' discussion on multiprocessing approaches links to [1] on Pool vs ProcessPoolExecutor. I wish that rather recommendations for non-standard libraries appeared. Several years ago, I spent an inordinate amount of time chasing down multiprocessing-usage edge cases. For example: How can I reliably learn that my subprocess died? How can I avoid a fork bomb if my work itself creates new work? Sane serialization semantics? Type hinting? Consistent behavior for spawn, fork, and forkserver? Register callbacks on Futures to run eagerly at completion? Etc. I ultimately wrote a single-file, drop-it-anywhere “jobserver“ [2] to remove many footcanons. It aims to be the well-tested core inside any more ergonomic API enforcing any additional semantics one might want. No canceling, but that could probably be added. My "jobserver" is loky-esque [3] as I believe they and I independently were fighting similar challenges but, of course, I prefer mine. Implementation is 560 lines including doc strings. Tests, in the same file, are about another 600 lines. [1] https://superfastpython.com/multiprocessing-pool-vs-processpoolexecutor/ https://superfastpython.com/multiprocessing-pool-vs-processp... [2] https://github.com/RhysU/jobserver/blob/master/jobserver.py#L7-L28 https://github.com/RhysU/jobserver/blob/master/jobserver.py#... [3] https://loky.readthedocs.io/en/stable/ https://loky.readthedocs.io/en/stable/

crabbygrabby 4y ago

Super good lay person summary. I like how this explains the underlying concepts more than the python specific stuff. Will recommend this to new comers.

jonstewart 4y ago

If Python is slow and you need concurrency, and you have a long-ish timeframe in mind, do yourself a favor and use one of C++/Java/Rust/Go/Erlang/Elixir/(other newer programming language). Especially, don’t rely on Multiprocessing. I’ve been developing and maintaining a largish batch processing Python project where performance is a key feature, and it’s been very frustrating. I should have rewritten it when I had the chance, but I’m slowly outsourcing critical pieces to C++ and Rust libraries. Multiprocessing has been a source of subtle portability errors as it has changed over different minor versions.

wenc 4y ago
If you need to stick with Python, Joblib is more reliable and works better than Multiprocessing in my experience. I ran into so many subtle quirks that needed workarounds in Multiprocessing. Joblib.parallel with the Loky backend just seems to be Multiprocessing done right.
jeremycarter 4y ago
I have a similar project and now have to split up the single application to actually get the muli core processing I demand. I regret not using Rust/Go... or even just .NET.
ansre 4y ago
[dead]
- jonstewart 4y ago
  Scaling horizontally is well and good (since my unit of input is large and I get a lot of them, I just use autoscaling groups in AWS to achieve much the same), but I don’t want to pay for 10 systems slowly doing what 1 could do with fast software. Especially, while 1 system has a hard ceiling, there’s a lot of headroom on 1 system when you achieve mechanical sympathy with the cache hierarchy, memory controller, and disk.
capableweb 4y ago
Hear hear! If you already know Python, picking up the basics of Java/Rust/Go/Elixir is trivial. Picking up the basics of C++ would be slightly harder, but not impossible. Erlang even further away, at least syntax wise. But, you'll be doing future you a huge service for whenever you need to reach for hard performance/latency and/or concurrency/parallelism again.

jhokanson 4y ago

"Firstly, there are three main Python concurrency APIs, they are:" asyncio, threading, multiprocessing, ... oh, and concurrent.futures All kidding aside, I used the multiprocessing module lately and it was a mess. Do I want 'map', 'starmap', 'imap', etc.? All I wanted was to run a function multiple times with different inputs (and multiple inputs per function call) and to fail when any launched process failed rather than waiting for every input variation to execute and then telling me about the error(which honestly I didn't think was asking for too much).

jayemar 4y ago
There's a lot of functionality in the multiprocessing library and it has its own problems, but I wouldn't call it a "mess" from your description. map, starmap, and imap are all useful for different applications depending on your usage and priorities. I'll agree that it can sometimes be difficult to understand the differences and which is best for your use case, but having used in different use cases I certainly appreciate the different functions available.

O5vYtytb 4y ago

I will say that structured concurrency (coroutine based) via Trio/Anyio is, in my experience, so much better for most applications that can support it. Reasoning/testing threading code is so impossible once a project gets large enough. I'd highly recommend reading https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ https://vorpus.org/blog/notes-on-structured-concurrency-or-g... I work with trio/anyio on a daily basis for my job, and I'd always recommend people use it for their concurrency framework and then use await anyio.to_thread.run_sync() to spawn threads if needed. I'll also add that if you need multi-processor execution, look at https://pypi.org/project/tractor/ https://pypi.org/project/tractor/

infinite8s 4y ago
+1 for trio. I used it in a past job to proxy a socket-based API to websockets and it was easy to write and debug. Tractor is also architected similar to Erlangs OTP system, which trio with it's structured concurrency makes easy to reason about.

zach_garwood 4y ago

Every time I read about the difficulties in writing concurrent Python, it reminds me of the joke: Patient: "Doc, it hurts when I do this!" Doctor: "Well, stop doing that." I love me some Python, but it's just not good at this. When your requirements dictate that you need concurrency, you should start evaluating a different language that is designed for it, in order to build that part of your system.

fulafel 4y ago

This glosses over the distinction between concurrency and parallelism, and the traditional & still common "pool of server processes" parallelism option.

pdimitar 4y ago

Or here's a crazy idea, maybe Python is not a holy gospel and you shouldn't use it for things that it's obviously badly suited for. You have a GIL in there, what other subtle hints do you need that your scenario is not well-supported? Erlang/Elixir and Rust will serve you perfectly for concurrent and parallel programming. OCaml 5.0, once stabilized, will do so as well. This Python worshipping and trying to use it for everything just betrays amateur programmers underneath. Whatever happened to "use the right tool for the job"?

jeremycarter 4y ago
This is the correct answer. The love affair to do everything in python must end this year!
- Frotag 4y ago
  But it does almost everything! And it's especially painless if you're not building for scale, and just hobby projects. But yeah, stuff like concurrency and type annotations seem to go against some of Python's fundamental designs (GIL and syntax) so are kinda painful at times.
  - pdimitar 4y ago
    > But it does almost everything! Ehhh... does it though? I've been doing shell scripts all my career and never stumbled upon a problem I couldn't solve by combining various tools. I did a lot of homegrown data science that way. With Python it seems it's mostly familiarity. People just love the idea of one universal tool, no matter how many times history proves there's no such thing.

vpribish 4y ago

content of article is ok, but why is it written like an SEO recipe website?

malkia 4y ago

Any resolution to this (python's multiprocessing on Windows and WaitForMultipleObjects failing when the number of objects (threads) to wait > 60) https://stackoverflow.com/questions/65252807/multiprocessing-pool-pool-on-windows-cpu-limit-of-63 https://stackoverflow.com/questions/65252807/multiprocessing... Just curious, as yesterday had to deal with the `black` formatter, and it had this hack into it to prevent no more than 60 being created.

girfan 4y ago

This is a great analysis; thanks for writing. I have also been working on running multiple Python interpreters in the same process by isolating them in different namespaces using `dlmopen` [1]. The objective on a high level is to receive requests for some compute intensive operations from a TCP/HTTP server and dispatch them on to different workers. In this case, a thin C++ shim receives the requests and dispatches them on to one of the Python interpreters in a namespace. This eliminates contention for the GIL amongst the interpreters and can exploit parallelism by running each interpreter on a different set of cores. The data obtained from the request does not need to be copied into the interpreter because everything is in the same address space; similarly the output produced by the Python interpreter is also just passed back without any copies to the server. [1] https://www.man7.org/linux/man-pages/man3/dlmopen.3.html https://www.man7.org/linux/man-pages/man3/dlmopen.3.html

toolslive 4y ago

Is stackless still an alternative? (It used to be quite hot 1.5 decade ago) https://github.com/stackless-dev/stackless/wiki/ https://github.com/stackless-dev/stackless/wiki/

mistrial9 4y ago

zero mention of DASK here, is it that obscure?

gshulegaard 4y ago

I am not sure when this article was written, but I really wouldn't recommend threads for even IO-bound tasks. Since threads don't escape the GIL the added contention with OS thread scheduling (context switching) is a performance overhead that coroutines (asyncio) don't have. I haven't benchmarked threads vs. coroutines for IO-bound tasks...but my gut feeling is coroutines are going to generally be better because of the lack of thread switching overhead. So for me, there really are only 2 concurrency choices: coroutines or multiprocessing. And generally, if I find myself reaching for multiprocessing I seriously evaluate if the logic shouldn't be ported to a different language.

lelandbatey 4y ago
I wouldn't quite say that there's only 2 concurrency choices, but I agree that there's only 2 concurrency choices in Python if you are operating under heavy load. The multi-threaded concurrency style can be a great choice if your performance requirements are low and you only want a small amount of concurrency. Then it can be very easy to emulate a Golang-esq model, by having just a few threads which are communicating by writing data via queues to one another.
hbrn 4y ago
But threads do release GIL on IO. There's a slight overhead of course, but if you're not running hundreds of servers and thousands of threads, the overhead can be negligible. I've seen more than once people rewriting thread-based code to coroutines, despite not having any noticeable benefits other than being more sexy. In fact I was one of those people almost a decade ago, rewriting the whole networking portion of the app only to achieve ~5% performance improvement. One should always measure the overhead instead assuming how big it is.
- gshulegaard 4y ago
  I am not expert, and this might be dated information, but historically there is a "thrashing" introduced between the GIL and OS scheduled threads which has surprising performance implications. David Beazley does a much better illustrating the nuances than I ever could: https://www.youtube.com/watch?v=Obt-vMVdM8s https://www.youtube.com/watch?v=Obt-vMVdM8s Including a wild phenomenon where a threaded Python program's performance can degrade by increasing number of CPU cores available to a system. Coroutines avoid this by not requiring individual thread scheduling by the OS. So they effectively release the GIL like threads on IO but avoid OS level thread switching.

vector57 4y ago

I will briefly plug my library `unsync` (https://github.com/alex-sherman/unsync#quick-overview https://github.com/alex-sherman/unsync#quick-overview) which wraps all these methods (multiprocessing/threading/asyncio) into singular/simple-ish API. It's a bit overly simple, but it's helped a few times writing code the makes use of several concurrency methods and combining them together etc.