4 ms·
The trouble with Pool.map is that the function you pass in has to be picklable, i.e. defined at the top level in the module you are currently in. So you can't
by canjobear 13y ago
The trouble with Pool.map is that the function you pass in has to be picklable, i.e. defined at the top level in the module you are currently in.
So you can't use it in interactive sessions:
In [1]: import multiprocessing
In [2]: pool = multiprocessing.Pool(8)
In [3]: def hello(x):
....: return x+1
....:
In [5]: stuff = range(10)
In [4]: pool.map(hello, stuff)
# cryptic errors
You also can't use it with lambdas or functions defined within functions; suppose I put this in test.py:
import multiprocessing
def run_stuff():
pool = multiprocessing.Pool(8)
def hello(x):
return x+1
stuff = range(10)
return pool.map(hello, stuff)
print(run_stuff())
... I get the same errors.
This is sad because pool.map seems to be faster than anything I have written myself which might replace it without the above limitations. Unless someone out there has done better than me? :)
- deleted 13y ago[deleted]
- hcarvalhoalves 13y agoIt works on the REPL as long as you create the pool after the function has been defined: >>> import multiprocessing >>> def hello(x): ... return x + 1 ... >>> pool = multiprocessing.Pool(8) >>> pool.map(hello, range(10)) [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] It looks like the pool takes a peek at the module during instantiation. Maybe could be worked around by having a lazy pool that only inspects when you call map.
- andreasvc 13y agoDefine the function you're passing to the pool before you create the pool, and make sure you use an 'if __name__ == "__main__: main()" line at the end if you're writing a script, since it needs to be importable. If functions within functions are not supported I don't see the problem, because the thing with multiprocessing is that you need to be conscious of what you're passing between processes, so explicitly passing everything as arguments or through an initializer is the way to go.
- goostavos 13y ago(Firstly, holy moly, something I wrote is on the front page of Hacker News!? I couldn't figure out why I was suddenly getting twitter messages on my otherwise dormant account.) Yeah, the requirement that things be picklable in order to send to another process is one of the worst parts of Python in my opinion. I can write the bulk of my code without running into the limitation, but when I do hit it? Holy crap. Rage. The only work around I've found, which isn't really a work around at all, is throwing out the idea of small, side effect free functions. By which I mean, give the process everything it needs to get the resources on "its" side (thus avoiding the pickling), and then let it process stuff until it's gotten back to some serializable base object that it can finally return. Not ideal, but that's how I generally work around it. There's also `multiprocessing.Array` and `Value` which avoid the pickle requirement completely, as they're shared memory, but limit you to very base datatypes. Really only good if your operating on some homogeneous array (which for me isn't too often).
- johtso 13y agoA solution to the irritating pickling behaviour of multiprocessing.Pool is to use Pathos.multiprocessing https://github.com/uqfoundation/pathos https://github.com/uqfoundation/pathos