4 ms·
I've been a happy user of Python's multiprocessing package, for the following reasons: * It's really natural if you work on an OS that has fork() (i.e., not Wi
by sqrt17 16y ago
I've been a happy user of Python's multiprocessing package, for the following reasons:
* It's really natural if you work on an OS that has fork() (i.e., not Windows), you just need to create a Pool after you've initialized everything and can use Pool.imap (in much the same way as itertools.imap) after that.
* Distributing data for initialization (which is ~100-200MB in my case) would add another level of complexity if I wanted to distribute tasks among several computers automatically.
At the granularity I'm working (most single tasks use >10ms, many >100ms), a simple worker pool works just as fine as the ForkJoin model that Clojure parallel is based on.
OpenMP and OpenMPI are for a very different use case where you know (pretty much) exactly how long the different parts of an operation will take, so you can make different processors (or different nodes) take different, determined-in-advance, chunks of the problem. Usually, the "parallel map" abstraction implemented by Python's multiprocessing, Java's ForkJoin, or Hadoop/Disco is much more useful as a quick fix to parallelize something than if you start out with channels and try to build something on top of that.
TL/DR: It depends on your workload what multiprocessing workload is right for you. For most bread-and-butter workload where you can use a parallel map function, multiprocessing/ForkJoin is just fine; OpenMP is great if you have a very predictable split between processors and are willing to jump through additional hoops to squeeze out the additional 5-10% of performance. Don't use queues etc. unless you know you need them. Multi-node solutions (MPI, map/reduce) usually add another layer of complexity because you need to ship the initialization data to each node (if you need, e.g. a statistical tagging or parsing model).
- _delirium 16y agoCan't you also use OpenMP for the quick-fix "parallel map" abstraction? The typical C way of mapping an operation is to do it in a loop, and if the loop iterations don't depend on each other (as in the mapping case), you can just add a "#pragma omp parallel for" at the top of the loop to make it execute via the worker pool.
- sqrt17 16y agoIf the execution time of the function varies considerably (i.e., some of the sub-problems are faster to solve), then you want to be able to keep the other processors busy with the next sub-problems while one of them keeps gnawing at a particularly difficult one. If the timing is very predictable, OpenMP's parallel-for construct fits your bill very nicely.
- _delirium 16y agoHmm, I thought OpenMP's worker-pool management did do that? E.g. if you have an 100-iteration loop, and have your num_threads set to 10, then it'll start up the first 10 subproblems; the first thread to finish will get the 11th, etc. If one of the first 10 is really slow, the other threads will still move on to churn on the rest of the loop iterations. Or is that not how it works?
- sqrt17 16y agoActually, you're right. OpenMP allows you to choose between "static" scheduling (where it would just cut the 100-iteration loop into 0..24, 25..49, 50..74, 75..99 if you have four processors) and "dynamic" scheduling with a specifiable block size (where it would, for a block size of 10, first chop off four iteration blocks of 0..9, 10..19, 20..29, 30..39, and allocate the other six blocks depending on which processor wants more work first). According to the OpenMP tutorial, the default for OMP_SCHEDULE is implementation dependent, and the gcc OpenMP implementation uses dynamic scheduling with a chunk size of 1 by default.