3 ms·
I'm curious why one would suggest using this over the built-in ThreadPoolExecutor class? Here's a complete, similar example from the Python docs: https://docs.
by eatonphil 6y ago
I'm curious why one would suggest using this over the built-in ThreadPoolExecutor class?
Here's a complete, similar example from the Python docs: https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor-example https://docs.python.org/3/library/concurrent.futures.html#th....
Granted that example is more lines of code than the README for this library. But the core of it is fairly straightforward:
rsps = []
with concurrent.futures.ThreadPoolExecutor(max_workers=N) as executor:
futures = [executor.submit(requests.get, url) for url in urls]
for future in concurrent.futures.as_completed(futures):
rsps.append(future.result)
- offtop5 6y agoGreat point , adding more libraries should always be a last resort. When using lambdas adding Python libraries are an absolute pain in the but, I'll use this example in the future
- joshlk 6y agoThis wont work for a large number of requests as the throughput plateaus quit rapidly with a large number of threads due to the GIL and blocking I/O. This is the main reason async was introduced into Python. Also `many-requests` has further logic to deal with retries and bad responses.
- eatonphil 6y agoYeah, I guess swapping out requests for an async version of requests would be an improvement on my example. But I think requests already handles retries on its own.
- joshlk 6y agoIts not simple to just "swap out" the requests library with an async one. This is why I wrote this package :) It deals with a lot of the headaches one would run into by trying just that.
- devxpy 6y agoDo you have empirical evidence for this? Even theoretically, threads can perform better because pythom does infact release the Gil for i/o bound operations and also, not blocking an event loop when a cpu bound task is running. In my experience at least, asyncio was almost always slower than threads, in terms of just raw throughput (as seen from nethogs)
- joshlk 6y agoI do actually - just dug it out: https://imgur.com/a/0uIQW6f https://imgur.com/a/0uIQW6f This was an experiment I did against a very slow API. It's 5 years old and so I was using Python 2.7. The graph shows number of completed requests per second against number of threads. As you can see at first the throughput linearly increases with threads but then plateaus at around 12 threads. I haven't done the same experiment with async-io but it would be quite easy using this package `many-requests`
- devxpy 6y agoHow do you know the bottleneck is python's threads, and not the server or your network? If its the latter, no amount of async programming will improve this graph.