4 ms·
It looks like this downloads them one-by-one instead of downloading at the same time. Is there a simple Python equivalent to Java's .parallelStream().forEach()
by myroon5 9y ago
It looks like this downloads them one-by-one instead of downloading at the same time.
Is there a simple Python equivalent to Java's .parallelStream().forEach() that would allow these calls to easily be run in parallel?
- mynewtb 9y agoSure thing... from multiprocessing import Pool with Pool(8) as p: p.map(function, sequence))
- pddubs 9y agoYou could just use threads here too - CPython releases the GIL during IO. from concurrent.futures import ThreadPoolExecutor with ThreadPoolExecutor as executor: executor.map(function, sequence)
- myroon5 9y agoNeat. I see that you explicitly specify 8 as the number of processes here. In Java, parallelStream() will pick a sane default for you if you haven't previously specified (based on the number of available processors, I believe). Is something like that possible in Python?
- jstarfish 9y agoThe ThreadPool class picks a sane default (number of cores), but I believe it uses Python threads instead of processes.