6 ms·
Show HN: Bringing multithreading to Python's async event loop
This project explores the integration of multithreading into the asyncio event loop in Python.
While this was initially built with enhancing CPU utilization for FastAPI servers in mind, the approach can be used with more general async programs too.
If you’re interested in diving deeper into the details, I’ve written a blog post about it here: https://www.neilbotelho.com/blog/multithreaded-async.html https://www.neilbotelho.com/blog/multithreaded-async.html
- quotemstr 2y agoAt that point, why bother with asyncio? What we really want is something like Java virtual threads, something that doesn't have a code color.
- btown 2y agogevent is exactly this! http://www.gevent.org/ http://www.gevent.org/ My startup has been using it in production for years. It excels at I/O bound workflows where you have highly concurrent real-time usage of slow/unpredictable partner APIs. You just write normal (non-async) Python code and the patched system internals create yields to the event loop whenever you’d be waiting for I/O, giving you essentially unlimited concurrency (as long as all pending requests and their context fit in RAM). https://github.com/gfmio/asyncio-gevent https://github.com/gfmio/asyncio-gevent does exist to let you use asyncio code in a gevent context, but it’s far less battle-tested and we’ve avoided using it so far.
- sidmitra 2y agoI love gevent, but i never was 100% sure that nothing is secretly breaking or some weird thread safety issue. In a large SaaS app all sorts of 3rd party libs do weird background threading stuff or someone randomly starts doing threading.Local and shared global context. After hitting some weird hanging redis-py client issues, i turned gevent off and it went away. Never really got around to spend time to debug the issue(especially since it happened on prod and hard to replicate on stage/local). Does your app have a lot of dependencies that do background threads? Like Launchdarkly(feature flags), redis, spyne(rpc) and on and on.
- whalesalad 2y agoWe also heavily use gevent but this is indeed the greatest frustration. Random and difficult to diagnose issues in external libraries like sockets being closed prematurely or timing out.
- drowsspa 2y agoNot for those of us stuck in Java 8... Is it stable already?
- quotemstr 2y agoI hear that some Amish sects permit the use of technology that's older and proven not to be too worldly, like washing machines, chainsaws, and Java 11. Have you considered converting?
- drowsspa 2y agoYeah... Unfortunately golden handcuffs are binding me to the financial sector and, specifically, to Hadoop
- fnord123 2y agoHadoop >=3.3 supports Java11.
- svieira 2y agoVirtual threads only became stable in Java 21, unfortunately (https://openjdk.org/jeps/444 https://openjdk.org/jeps/444). If the issue is "I'm bound to Java 8" proposing running a research build of Java 11 in production is going to fly as well as a lead balloon on the moon.
- drowsspa 2y agoEven so, it's pretty new, isn't it? I don't quite trust the claim it's completely transparent to applications and libraries...
- nbsande 2y agoHmmm. That would indeed be better. Seems like an interesting experiment to try and implement virtual threads for python!
- noident 2y ago> It got me wondering if it was actually possible to make Python’s async event loop work with multiple threads. There is built-in support for this. Take a look at loop.run_in_executor. You can await something scheduled in a separate Thread/ProcessPoolExecutor. Granted, this is different than making the async library end-to-end multi-threaded as you seem to be trying to do, but it does seem worth mentioning in this context. You _can_ have async and multiple threads at the same time!
- nbsande 2y agorun_in_executor is pretty powerful for running sync code in async code, but my use case was more making async code utlize the cpu better. I think, just using run_in_executor would add a lot of complication and changes to how you use async await. But great point none the less!
- zo1 2y agoI've had plenty of success using run_in_executor in production. You basically just use the decorator, and it mostly just works.
- sandeep1998 2y agoWhich decorator?
- zo1 2y agoI went back to look at some of the old code as my memory was hazy. It was tornado's implementation of the run_on_executor method that we used, which is used as a decorator. https://www.tornadoweb.org/en/stable/concurrent.html#tornado.concurrent.run_on_executor https://www.tornadoweb.org/en/stable/concurrent.html#tornado...
- noident 2y ago> but my use case was more making async code utlize the cpu better But run_in_executor achieves that as well! If you use no-GIL Python and a thread pool (or GIL Python with a Process pool), you will utilize more CPU cores.
- m11a 2y agoPerhaps a silly question, but should an event loop actually be multithreaded? My understanding was that tasks in an event loop should yield after they dispatch IO tasks, which means the event loop should be CPU-bound right? If so, multithreading should not help much in theory?
- spiffytech 2y agoIf your workload is actually CPU-bound after you've deferred IO to the background, that's exactly when multithreading can help. I've seen code that spends disproportionate CPU time spent on e.g., JSON (de)serializing large objects, or converting Postgres result sets into native data structures, but sometimes it's just plain ol' business logic. And with enough traffic, any app gets too busy for one core. Single-threaded langs get around this by deploying multiple copies of the app on each server to use up the cores. But that's less efficient than a single, parallel runtime, and eliminates some architectural options.
- limaoscarjuliet 2y agoYou are correct, async-io is cooperative. This seems an attempt to enhance these async-io cooperative threads work more like goroutines in golang. Golang can start threads if it "thinks" the workload needs more CPU.
- bastawhiz 2y agoThis was exactly my question. Why do you even need an event loop? If awaits are just thread joins then what is the event loop actually doing? IO can just block, since other coroutines are on other threads and are unaffected. Which is to say, why even bother with async if you want your code to be fully threaded? Async is an abstraction designed specifically to address the case where you're dealing with blocking IO on a single thread. If you're fully threaded, the problems async addresses don't exist anymore. So why bother?
- svieira 2y agoLooking at the article he's not implementing `Task` with `Thread` - he's round-robinning processing `Task`s through simple `ThreadPool`. So instead of a single `Thread` making continuous progress on the work in the event loop he instead has a set of `Thread`s making progress _in parallel_ on work in the event loop. This is very much Java 21's approach to virtual threads (as well as in-language task runners like the kind you find in Scala libraries like ZIO, Monix, Cats, and the venerable Scalaz).
- game_the0ry 2y agoI am fairly confident I will get some down votes for this, but here goes... When I am trying to solve a technical problem, the problem is going to dictate my choice of tooling. If I am doing some fast scripting or I need to write some glue code, python is my go-to. But if I have a need for resource efficiency, multi threading, non-blocking async i/o, and/or hi performance, I would not consider python - I would probably use JVM over the best python option. Don't get me wrong, I think its a worthwhile effort to explore this effort, and I certainly do not think its a wasted effort (quite the opposite, this gets my up vote) I just don't think I would ever use it if I had use case for perf and resource efficiency.
- trashtester 2y agoThe main object is the in-between-land, where you need 10x-50x the performance of Python, not 500x the performance on some parallelizable workload. And in many teams, just having to worry about python makes it easier to keep team members productive if they're not expected to handle several different languages productively.
- game_the0ry 2y agoFair point. Tho I will push back on... > And in many teams, just having to worry about python makes it easier to keep team members productive if they're not expected to handle several different languages productively. I think this makes sense for individuals and teams, but for an org or company I think having specialist teams makes sense where teams that require perf use JVM and teams that make business-ware or devops or something not perf would use python.
- trashtester 2y agoJVM based development has a place for some teams. Others need to go all the way to C++/C/Rust etc for the performance they need. But plenty of tasks can be done beautifully in Python. That's especially true in a data processing or ML setting where most of the heavy lifting is done in libraries such as numpy, spark, pytorch etc. (Also Python is the industry standard for such teams). Still, even for such teams there are times where you want to do SOME heavier compute tasks within the core language, and offloading this to some other dev team simply doesn't scale. The solution is to use multi processing instead of multi threading. But this workaround is quite inflexible. Some dev teams may have developers that can deliver this in scala (especially if spark is involved). Other may have the ability to build C++ (or CUDA) libraries to add to the python environment. But the ability to run somewhat heavier processing than what can be achieved by a single process is often much better. Cost wise it also makes a lot of sense. Such steps may often find themselves on some large compute cluster where you have tens or hundreds of processors (or more) available. If a single step in a processing pipeline on such a cluster can be cut from 2 hours to 1 minute, it can be a large saving. Taking it from 1 minute to 10 seconds means a lot less. Btw, and with all due respect, the part about teams that require perf using JVM doesn't really match my experience. Where I come from, the Java devs tend to produce the slowest code of all, mostly because every step of the processing is serialized/deserialized as microservices talk to each other for each data element. Even python based code is often faster (sometimes by order of magnitudes). Partly because of cultural differences between teams (the python code, even when exposed as microservices, tend to work with larger blocks of code. And partly because the really is processed in C++ based libraries within python, that still have a significant edge on JVM based code. Don't get me wrong: Java has a lot of advantages for many types of business applications, where the business logic complexity can be abstracted in well organized and standardized ways. But it's not typically the go-to language when seeking maximum performance in heavy compute or massive data volume scenarios.