3 ms·
There is a nice hack for multiprocessing module to use for IO-bound code without having to fork processes: from multiprocessing.dummy import Pool as ThreadP
by rdtsc 10y ago
There is a nice hack for multiprocessing module to use for IO-bound code without having to fork processes:
from multiprocessing.dummy import Pool as ThreadPool
pool = ThreadPool(16)
res = pool.map(one_arg_fun, [arg1, arg2, ...])
A 3 line IO parallelism speedup trick. Used it fetch stuff from multiple servers recently.
- ipunchghosts 10y agocan you elaborate?
- jessedhillon 10y agoIt's the equivalent of: (edit: actually an improvement over) from threading import Thread def blocking_function(arg): .... pool = [] for arg in args[:16]: f = lambda: blocking_function(arg) thread = Thread(target=f) thread.start() pool.append(thread) Edit: see below, this doesn't include job queueing but rather, hard limits your input to maximum 16 args. The point is, the code is a nice, easy way to start a number of threads working on a list of arguments.
- quinnftw 10y agoNot quite. You can submit as many jobs as you'd like to the pool and it will delegate them to 16 threads. The pool maintains a queue of work to be done and the fixed number of threads consume from the queue.
- rdtsc 10y agoAlmost but as quinnftw mentioned it does a bit more, specifically it limits the number of threads it runs to 16. I can pass 1000 url or arguments into it and it won't spawn 1000 threads but keep it at 16 max. Also I think it does a bit a better job with exception propagation.
- deleted 10y ago[deleted]
- paulddraper 10y agoMultiprocessing offers a ThreadPool for multi-threading.. It doesn't have true CPU parallelism like multiple processes would have, but it works for IO code. It's a little gold nuggest in the multiprocessing lib.
- deleted 10y ago[deleted]
- gshulegaard 10y agoI would recommend you use coroutines rather than multiprocessing.dummy for balancing blocking operations (like IO). I believe multiprocessing.dummy is just a wrapper for threading which has context switching overhead [1]. Python3 has asyncio as part of the stdlib but for Python2 there are a variety of coroutine libraries out there (I think the most widely recognized being gevent). [1] https://www.youtube.com/watch?v=Obt-vMVdM8s https://www.youtube.com/watch?v=Obt-vMVdM8s
- rdtsc 10y agoSwitching code-base to Python 3 or installing gevent/eventlet is certainly more than a 3 line hack. > I believe multiprocessing.dummy is just a wrapper for threading which has context switching overhead [1]. Yap. But I am fetching and waiting for data to come from servers half-way across the country. Not worried about too much thread switching overhead. The point it was still faster than a for loop with a fetch. It was replaced by 3 lines which was 10x faster.
- bogomipz 10y agoI'm not following. Can you explain the hack? Python releases the GIL during I/O right, isn't that desirable?
- rdtsc 10y ago> Python releases the GIL during I/O right, isn't that desirable? Python releases GIL during IO so that's why thread-based pool works for IO parallelism and there is no need to fork. Was that what you mentioned? Sorry, I don't think I completely followed.
- bogomipz 10y agoI just re-read your comment again and the one above it. I was confusing MT and MP. Thanks.