2 ms·
sorry for misreading that from your post! my own benching with threads vs. asyncio has never found any performance difference between the two approaches (async
by zzzeek 4y ago
sorry for misreading that from your post! my own benching with threads vs. asyncio has never found any performance difference between the two approaches (asyncio slightly slower). if you need very wide throughput, then yes asyncio is better. otherwise, it's very difficult to create equitable comparisons between threaded and asyncio code.
- heavyset_go 4y agoHaha, it's cool. I once had a problem where I needed to scrape a bunch of sites at once, and ended up using it as a benchmark for threads vs asyncio. The overhead with threads was high even when using a handful of threads, and quickly hit a plateau after scaling to 4-8 threads on a 16 core machine. I was a big fan of the concurrent.futures executor model and generally thought it was good performance anyway, but figured I'd try to implement it in asyncio, and rate limited both for a fairer comparison. It was so shockingly fast that it was the last project I reached for threads again with on Python. I can't say that it was an equitable comparison, but it certainly left an impression on me.
- zzzeek 4y agoit was probably whatever concurrent.futures does. I use plain threads and get great results for anything under 25 threads on a small laptop. you always keep the same set of threads running in a pool and use a decent queueing model. certainly, doing that kind of thing with asyncio likely has a lower learning curve because you don't need to worry about "pools of workers" so much. concurrent.futures should do that for you also but that would be the first place I don't trust