4 ms·

This example is not too realistic and just narrows it down to the case where a job can be divided into isolated tasks with no shared data/state. Often times th

by halayli 12y ago

This example is not too realistic and just narrows it down to the case where a job can be divided into isolated tasks with no shared data/state. Often times threads need to update shared dict/list etc... With multiprocessing this cannot be done. You can use a Queue for this but it's horribly inefficient. Generally speaking if you need performance and Python is not meeting the requirements then you are better off using another language.

shogunmike 12y ago
I agree, the scope of the article is somewhat specific to the "toy" example presented. Generally I would use C++ or (gasp!) Fortran with either MPI or CUDA for these sorts of tasks if performance was the most critical factor. I'm excited by the Julia language though!
- js2 12y ago
  It's py3 but there is a back port of it for py2 that works wonderfully. I've recently begun using it and will never look back to multiprocessing (on which it is built).
  - andreasvc 12y ago
    You probably meant to reply to the thread about concurrent.futures.
pekk 12y ago
> With multiprocessing this cannot be done. You can use a Queue for this but it's horribly inefficient. So document that this is not just a problem with some program you wrote, but multiprocessing itself.
shavenwarthog2 12y ago
Multiprocessing supports "managed objects", shared across multiple processes -- example: http://johntellsall.blogspot.com/2014/05/code-multiprocessing-producerconsumer.html http://johntellsall.blogspot.com/2014/05/code-multiprocessin... I'm unsure of the performance ramifications vs using concurrent.futures