4 ms·

Although it adds another piece of software on top, I've been using Python with Redis a lot lately, and it's wonderful. Here's a 'base' for building the larger

Although it adds another piece of software on top, I've been using Python with Redis a lot lately, and it's wonderful. Here's a 'base' for building the larger program. It doesn't make a single python instance multithreaded, but it easily allows me to connect other computers on my network to join in the processing fun. filePath = "/home/user/file" p = Process(target=readFile, args=(filePath,)) p.start() numProcesses = 6 for x in range(numProcesses): p = Process(target=mapper, args=()) p.start() So I have a file reader to prevent a seek storm on a large file, I read something (like separations between tags that I want to process more), I put the binary data in a "ready:NAME" key, with "NAME" going into a "readyList", and things go wonderfully. It makes my data persistent without writing to files or pickling, and I can easily share data between process instances. My question is - what advantage does a multicore programming offer that my take wouldn't? I can import process and chain things out over my 8 cores, but why would I do that if I can have persistent data with the ability to more easily add processing power when I want to?

andrewcooke 14y ago
well it allows you to share in-memory, mutable state between processes. if you have problems that don't need that then yes, parallelization is easy (the python-only version of the above would be to use multiprocessing). but maybe i've missed the point (seem to be making error after error at the moment...) [edit: to clarify a bit more, the example given in the article linked is making each execution of a loop run in parallel, even if the loop affects some variable. so you would hope to get speedups in code that isn't written explicitly to be executed by multiple processes - without messaging etc. in particular, it guarantees safety while giving the chance (sometimes it doesn't work out) for speedups.]
- TillE 14y ago
  Message passing is usually a "better" method of IPC anyway. The situations that genuinely require shared memory are rare.
  - deleted 14y ago
    [deleted]
  - scott_s 14y ago
    Perhaps in your line of work, but in scientific computing, it's quite common. Scientific codes have parallelism at multiple levels, but loop-level parallelism is extremely common. Simply, you have large vectors or matrices that are operated on by a loop. The iterations of the loop are (mostly) independent, so you can use multiple threads to do them in parallel. Assuming shared memory, of course; the communication cost of transferring the vectors or matrices to another process over typical IPC mechanisms would kill the benefit of the parallelism. OpenMP (http://openmp.org/wp/ http://openmp.org/wp/) for C, C++ and Fortran is the most common means of exploiting such parallelism in the high performance computing world.
    - montecarl 14y ago
      I'm curious about what field of scientific computing you work in. I am only familiar with MPI and hybrid MPI/OpenMP codes. I haven't come across any OpenMP only programs ever in chemistry or physics. If you have a loop with independent iterations then you can use message passing or shared memory to approach your problem (as well as loop vectorization). The main problem with message passing is that for very large datasets you can run into memory limitations.
      - scott_s 14y ago
        I don't, really. I do system software for high performance and distributed computing. The scientific applications are given to me, and are largely benchmarks. (Although, now, I don't do much actual HPC, but high-throughput, low-latency stream processing.) I have a friend who has a galaxy simulation that uses OpenMP but not MPI. The reason is simple: she doesn't have the expertise to make it distributed. Slapping a few OpenMP directives on the most expensive loops is easier than figuring out how to make it distributed using message passing. How much parallelism you extract out of a program is often a function of how much time and effort you can invest into it. Some people get "good enough" performance improvements by scaling on a single node.
spenrose 14y ago
I'm also finding pass-by-value multiprocessing with Redis to be extremely effective for going wide on I/O-bound problems. It's just so easy ...

andrewcooke 14y ago

well it allows you to share in-memory, mutable state between processes. if you have problems that don't need that then yes, parallelization is easy (the python-only version of the above would be to use multiprocessing). but maybe i've missed the point (seem to be making error after error at the moment...) [edit: to clarify a bit more, the example given in the article linked is making each execution of a loop run in parallel, even if the loop affects some variable. so you would hope to get speedups in code that isn't written explicitly to be executed by multiple processes - without messaging etc. in particular, it guarantees safety while giving the chance (sometimes it doesn't work out) for speedups.]

TillE 14y ago
Message passing is usually a "better" method of IPC anyway. The situations that genuinely require shared memory are rare.
- deleted 14y ago
  [deleted]
- scott_s 14y ago
  Perhaps in your line of work, but in scientific computing, it's quite common. Scientific codes have parallelism at multiple levels, but loop-level parallelism is extremely common. Simply, you have large vectors or matrices that are operated on by a loop. The iterations of the loop are (mostly) independent, so you can use multiple threads to do them in parallel. Assuming shared memory, of course; the communication cost of transferring the vectors or matrices to another process over typical IPC mechanisms would kill the benefit of the parallelism. OpenMP (http://openmp.org/wp/ http://openmp.org/wp/) for C, C++ and Fortran is the most common means of exploiting such parallelism in the high performance computing world.
  - montecarl 14y ago
    I'm curious about what field of scientific computing you work in. I am only familiar with MPI and hybrid MPI/OpenMP codes. I haven't come across any OpenMP only programs ever in chemistry or physics. If you have a loop with independent iterations then you can use message passing or shared memory to approach your problem (as well as loop vectorization). The main problem with message passing is that for very large datasets you can run into memory limitations.
    - scott_s 14y ago
      I don't, really. I do system software for high performance and distributed computing. The scientific applications are given to me, and are largely benchmarks. (Although, now, I don't do much actual HPC, but high-throughput, low-latency stream processing.) I have a friend who has a galaxy simulation that uses OpenMP but not MPI. The reason is simple: she doesn't have the expertise to make it distributed. Slapping a few OpenMP directives on the most expensive loops is easier than figuring out how to make it distributed using message passing. How much parallelism you extract out of a program is often a function of how much time and effort you can invest into it. Some people get "good enough" performance improvements by scaling on a single node.

spenrose 14y ago

I'm also finding pass-by-value multiprocessing with Redis to be extremely effective for going wide on I/O-bound problems. It's just so easy ...