4 ms·
Some good alternatives, for when you just want quick and easy parallelization: Joblib[0]: 'Embarrassingly parallel for loops'. Basically just write a generator
by cossatot 10y ago
Some good alternatives, for when you just want quick and easy parallelization:
Joblib[0]: 'Embarrassingly parallel for loops'. Basically just write a generator and get multicore processing on it. Pretty straightforward.
Multiprocess[1]: This is a fork of the multiprocessing module. The biggest benefit to me (the one time I used it) is that it's easier to create shared data structures for multiprocessing (which I couldn't do with multiprocessing.Pool). Last week I had to do a really long graph calculation that only needed to return a result for a very, very small fraction of the arguments. Joblib stored all of the null results as 'None', which tore through my RAM after billions of relatively fast calculations before crashing. But using a shared dictionary and multiprocess.Pool and imap_unordered, I was able to use mutlicore processing, adding items to the dict only if the right conditions were met, and discarding the 'None's. RAM use was very minimal.
[0]: https://pythonhosted.org/joblib/index.html https://pythonhosted.org/joblib/index.html
[1]: https://pypi.python.org/pypi/multiprocess https://pypi.python.org/pypi/multiprocess
- d33 10y agoAlso, consider using Celery [0] and distribute the computation across different hosts as well. [0] http://www.celeryproject.org/ http://www.celeryproject.org/