3 ms·

well it allows you to share in-memory, mutable state between processes. if you have problems that don't need that then yes, parallelization is easy (the python

well it allows you to share in-memory, mutable state between processes. if you have problems that don't need that then yes, parallelization is easy (the python-only version of the above would be to use multiprocessing). but maybe i've missed the point (seem to be making error after error at the moment...) [edit: to clarify a bit more, the example given in the article linked is making each execution of a loop run in parallel, even if the loop affects some variable. so you would hope to get speedups in code that isn't written explicitly to be executed by multiple processes - without messaging etc. in particular, it guarantees safety while giving the chance (sometimes it doesn't work out) for speedups.]

TillE 14y ago
Message passing is usually a "better" method of IPC anyway. The situations that genuinely require shared memory are rare.
- deleted 14y ago
  [deleted]
- scott_s 14y ago
  Perhaps in your line of work, but in scientific computing, it's quite common. Scientific codes have parallelism at multiple levels, but loop-level parallelism is extremely common. Simply, you have large vectors or matrices that are operated on by a loop. The iterations of the loop are (mostly) independent, so you can use multiple threads to do them in parallel. Assuming shared memory, of course; the communication cost of transferring the vectors or matrices to another process over typical IPC mechanisms would kill the benefit of the parallelism. OpenMP (http://openmp.org/wp/ http://openmp.org/wp/) for C, C++ and Fortran is the most common means of exploiting such parallelism in the high performance computing world.
  - montecarl 14y ago
    I'm curious about what field of scientific computing you work in. I am only familiar with MPI and hybrid MPI/OpenMP codes. I haven't come across any OpenMP only programs ever in chemistry or physics. If you have a loop with independent iterations then you can use message passing or shared memory to approach your problem (as well as loop vectorization). The main problem with message passing is that for very large datasets you can run into memory limitations.
    - scott_s 14y ago
      I don't, really. I do system software for high performance and distributed computing. The scientific applications are given to me, and are largely benchmarks. (Although, now, I don't do much actual HPC, but high-throughput, low-latency stream processing.) I have a friend who has a galaxy simulation that uses OpenMP but not MPI. The reason is simple: she doesn't have the expertise to make it distributed. Slapping a few OpenMP directives on the most expensive loops is easier than figuring out how to make it distributed using message passing. How much parallelism you extract out of a program is often a function of how much time and effort you can invest into it. Some people get "good enough" performance improvements by scaling on a single node.
  - wtallis 14y ago
    It's worth pointing out that those algorithms are also generally the ones that most benefit from being ported to a GPU, which removes the need for a shared-memory mechanism in the host-side language.
    - scott_s 14y ago
      Yes, if the latency of transferring the data and results to and back from GPU memory is low compared to the speedup.

TillE 14y ago

Message passing is usually a "better" method of IPC anyway. The situations that genuinely require shared memory are rare.

deleted 14y ago
[deleted]
scott_s 14y ago
Perhaps in your line of work, but in scientific computing, it's quite common. Scientific codes have parallelism at multiple levels, but loop-level parallelism is extremely common. Simply, you have large vectors or matrices that are operated on by a loop. The iterations of the loop are (mostly) independent, so you can use multiple threads to do them in parallel. Assuming shared memory, of course; the communication cost of transferring the vectors or matrices to another process over typical IPC mechanisms would kill the benefit of the parallelism. OpenMP (http://openmp.org/wp/ http://openmp.org/wp/) for C, C++ and Fortran is the most common means of exploiting such parallelism in the high performance computing world.
- montecarl 14y ago
  I'm curious about what field of scientific computing you work in. I am only familiar with MPI and hybrid MPI/OpenMP codes. I haven't come across any OpenMP only programs ever in chemistry or physics. If you have a loop with independent iterations then you can use message passing or shared memory to approach your problem (as well as loop vectorization). The main problem with message passing is that for very large datasets you can run into memory limitations.
  - scott_s 14y ago
    I don't, really. I do system software for high performance and distributed computing. The scientific applications are given to me, and are largely benchmarks. (Although, now, I don't do much actual HPC, but high-throughput, low-latency stream processing.) I have a friend who has a galaxy simulation that uses OpenMP but not MPI. The reason is simple: she doesn't have the expertise to make it distributed. Slapping a few OpenMP directives on the most expensive loops is easier than figuring out how to make it distributed using message passing. How much parallelism you extract out of a program is often a function of how much time and effort you can invest into it. Some people get "good enough" performance improvements by scaling on a single node.
- wtallis 14y ago
  It's worth pointing out that those algorithms are also generally the ones that most benefit from being ported to a GPU, which removes the need for a shared-memory mechanism in the host-side language.
  - scott_s 14y ago
    Yes, if the latency of transferring the data and results to and back from GPU memory is low compared to the speedup.