3 ms·
To take advantage of process level parallelism you still have to have a pickle-able function, i.e. defined at the top level in a module. In [1]: import concu
by canjobear 13y ago
To take advantage of process level parallelism you still have to have a pickle-able function, i.e. defined at the top level in a module.
In [1]: import concurrent.futures
In [2]: with concurrent.futures.ProcessPoolExecutor(max_workers=8) as executor:
...: out_list = list(executor.map(lambda _: random.random(), range(1000000)))
...:
Traceback (most recent call last):
File "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 266, in _feed
send(obj)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
- e12e 13y agoGood point. Just a couple of points on futures: 1) they're backported to python 2[1], and 2) to make the example work, you need a pickleable function as you say, for example, if you have ipython running in a virtualenv: import pip pip.main(["install","futures"]) import random def l(_): return random.random() with f.ProcessPoolExecutor(max_workers=4) as ex: out_list = list(ex.map(l, range(1000))) len(out_list) #> 1000 [1] https://pypi.python.org/pypi/futures https://pypi.python.org/pypi/futures
- e12e 13y agoWhops, forgot to add a line to import futures: import futures as f #Include after pip.main(...