3 ms·

Today, there are many places that have big clusters of multi-processor machines, and none of their code is threaded. They get cpu concurrency by message passing

Today, there are many places that have big clusters of multi-processor machines, and none of their code is threaded. They get cpu concurrency by message passing between heavy-weight processes and with other nodes.

openasocket 9y ago
I assume you're referencing MPI? It all depends on your problem domain. There are plenty of classes of problems where message passing incurs far more overhead than just sharing memory.
- greglindahl 9y ago
  MPI is indeed something I used to do, but I was thinking more of large NoSQL setups like blekko's search engine, which was 100% event-driven code running in separate, heavy-weight processes. (OK, in the guts, we did have some utility software that did async disk I/O using a few C threads per heavyweight process, but that was due to the limited kernel API more than us wanting threads.)
  - openasocket 9y ago
    Sure, but the point is that outside of embarrassingly parallel problems you need shared memory access. You can simulate that with message passing, but for many problems the overhead of message passing is too great. I'm specifically thinking of linear algebra routines. I'd like to see someone write a performant BLAS implementation that doesn't make extensive use of threading
    - zzzcpan 9y ago
      Message passing lets you amortize synchronization and should always be faster if properly implemented.
      - openasocket 9y ago
        Not really, no. You've still got the overhead of copying, which can be significant. There's no way you could implement something like, for example, Gaussian elimination with message passing in a way that's just as fast as using locks and barriers.
        greglindahl 9y ago
        I totally agree that zzzcpan was over-generalizing, but so are you: is it really the case that the Linpack benchmark (which uses arrays as large as memory) on the latest SGI shared-memory thing uses locks and barriers?
        zzzcpan 9y ago
        I think the cost of locks is so high, that even some primitive runtime zero-copying mechanism won't add significant overhead. After all, if you have to do something on another core, you have to copy it there, it's not really a physical shared memory.
        openasocket 9y ago
        Maybe not locks and barriers specifically, but I'd be shocked if it's using message passing, particularly because of the overhead of copying. If anyone has evidence to the contrary I'd be happy to hear it.
        greglindahl 9y ago
        Wow. I guess you aren't familiar with the history of large SGI Origin and Altix machines being mostly used as pure MPI machines because that's almost always fastest? Also, Linpack doesn't involve much data motion. Also, moving data over a shared memory interconnect is as much of a copy as moving data over a message passing interconnect. I was going to look up the current details for you, but I don't see an SGI Ultraviolet system in the current Top500. How large of a machine do you work with?
    - greglindahl 9y ago
      Well, the example I was thinking of (from after I left HPC) was a search engine, in which most of the computation took place in a NoSQL database. And it did great with event-driven programming and heavy-weight processes. You can call it embarrassingly-parallel if you like, from an HPC point of view it is. And that's a fine thing. Glad to see HPC so represented here on HN, but an HPC counter-example does not diminish an example. To put this another way: I wasn't saying that YOUR problem could be solved without threads, I said MY problem could be solved without threads.
      - openasocket 9y ago
        Completely agree, and I prefer to use something like message passing whenever possible because for a lot of problems it is the natural solution and has fewer pitfalls than using raw threads.
        greglindahl 9y ago
        Great. Nothing like a vigorous discussion that turns out to ... actually be an agreement.
tgamblin 9y ago
If you're talking about MPI and its use in HPC, you might've been mostly correct 5 years ago. Now, HPC applications at the largest sites are hybrid MPI+OpenMP (or some other threading model, or they use GPUs). On Xeon Phi systems like the latest Crays, you pretty much need OpenMP to take advantage of all the cores. The PPC A2 processors on the (formerly #1, now #4) Blue Gene/Q system at LLNL won't get full instruction issue per cycle unless you use multiple hardware threads (the threads are exposed to the hardware and each core will only pull instructions from a single process at a time, so this is somewhat unlike hyperthreading on Intel chips). There can be overheads to running very large numbers of MPI processes, depending on how scalable the MPI implementation is, and there are overheads to having too many threads contend for memory on a single node. At least in the current state of things in HPC, there is a delicate balance to choosing the right process-thread ratio, and it differs from application to application and depends on the node count for the job. People struggle to find the sweet spot, but we're definitely not running all MPI anymore.
- greglindahl 9y ago
  I wasn't talking about MPI, but if you want to talk about MPI, note that OmniPath gets faster as you have more cores talking to it, and it takes full advantage of delivering messages to the memory near the receiving process, so you'd better not move threads or processes very far. If you look in your HPC historybook, you'll notice that I was the system architect for InfiniPath.
notacoward 9y ago
> there are many places that have big clusters of multi-processor machines, and none of their code is threaded. A bit over-general, don't you think? There are many places that have big clusters full of single-threaded processes. There are also many places that have big clusters full of multi-threaded processes. I don't know which group is the biggest, but it doesn't matter as far as Osterhout's argument is concerned. What matters is that, contrary to the context in which that argument is framed, wringing performance out of multiple CPUs in a single system is an almost universal need nowadays. Many of them furthermore need high levels of intra-process as well as inter-process parallelism for that purpose. Threading is not the only way to achieve that, but he never even considers other alternatives. No matter what hairs are split or who tries to claim the biggest credentials, the OP remains seriously time-bound and its applicability to modern systems is weak.

openasocket 9y ago

I assume you're referencing MPI? It all depends on your problem domain. There are plenty of classes of problems where message passing incurs far more overhead than just sharing memory.

greglindahl 9y ago
MPI is indeed something I used to do, but I was thinking more of large NoSQL setups like blekko's search engine, which was 100% event-driven code running in separate, heavy-weight processes. (OK, in the guts, we did have some utility software that did async disk I/O using a few C threads per heavyweight process, but that was due to the limited kernel API more than us wanting threads.)
- openasocket 9y ago
  Sure, but the point is that outside of embarrassingly parallel problems you need shared memory access. You can simulate that with message passing, but for many problems the overhead of message passing is too great. I'm specifically thinking of linear algebra routines. I'd like to see someone write a performant BLAS implementation that doesn't make extensive use of threading
  - zzzcpan 9y ago
    Message passing lets you amortize synchronization and should always be faster if properly implemented.
    - openasocket 9y ago
      Not really, no. You've still got the overhead of copying, which can be significant. There's no way you could implement something like, for example, Gaussian elimination with message passing in a way that's just as fast as using locks and barriers.
      - greglindahl 9y ago
        I totally agree that zzzcpan was over-generalizing, but so are you: is it really the case that the Linpack benchmark (which uses arrays as large as memory) on the latest SGI shared-memory thing uses locks and barriers?
        zzzcpan 9y ago
        I think the cost of locks is so high, that even some primitive runtime zero-copying mechanism won't add significant overhead. After all, if you have to do something on another core, you have to copy it there, it's not really a physical shared memory.
        openasocket 9y ago
        Maybe not locks and barriers specifically, but I'd be shocked if it's using message passing, particularly because of the overhead of copying. If anyone has evidence to the contrary I'd be happy to hear it.
        greglindahl 9y ago
        Wow. I guess you aren't familiar with the history of large SGI Origin and Altix machines being mostly used as pure MPI machines because that's almost always fastest? Also, Linpack doesn't involve much data motion. Also, moving data over a shared memory interconnect is as much of a copy as moving data over a message passing interconnect. I was going to look up the current details for you, but I don't see an SGI Ultraviolet system in the current Top500. How large of a machine do you work with?
  - greglindahl 9y ago
    Well, the example I was thinking of (from after I left HPC) was a search engine, in which most of the computation took place in a NoSQL database. And it did great with event-driven programming and heavy-weight processes. You can call it embarrassingly-parallel if you like, from an HPC point of view it is. And that's a fine thing. Glad to see HPC so represented here on HN, but an HPC counter-example does not diminish an example. To put this another way: I wasn't saying that YOUR problem could be solved without threads, I said MY problem could be solved without threads.
    - openasocket 9y ago
      Completely agree, and I prefer to use something like message passing whenever possible because for a lot of problems it is the natural solution and has fewer pitfalls than using raw threads.
      - greglindahl 9y ago
        Great. Nothing like a vigorous discussion that turns out to ... actually be an agreement.

tgamblin 9y ago

If you're talking about MPI and its use in HPC, you might've been mostly correct 5 years ago. Now, HPC applications at the largest sites are hybrid MPI+OpenMP (or some other threading model, or they use GPUs). On Xeon Phi systems like the latest Crays, you pretty much need OpenMP to take advantage of all the cores. The PPC A2 processors on the (formerly #1, now #4) Blue Gene/Q system at LLNL won't get full instruction issue per cycle unless you use multiple hardware threads (the threads are exposed to the hardware and each core will only pull instructions from a single process at a time, so this is somewhat unlike hyperthreading on Intel chips). There can be overheads to running very large numbers of MPI processes, depending on how scalable the MPI implementation is, and there are overheads to having too many threads contend for memory on a single node. At least in the current state of things in HPC, there is a delicate balance to choosing the right process-thread ratio, and it differs from application to application and depends on the node count for the job. People struggle to find the sweet spot, but we're definitely not running all MPI anymore.

greglindahl 9y ago
I wasn't talking about MPI, but if you want to talk about MPI, note that OmniPath gets faster as you have more cores talking to it, and it takes full advantage of delivering messages to the memory near the receiving process, so you'd better not move threads or processes very far. If you look in your HPC historybook, you'll notice that I was the system architect for InfiniPath.

notacoward 9y ago

> there are many places that have big clusters of multi-processor machines, and none of their code is threaded. A bit over-general, don't you think? There are many places that have big clusters full of single-threaded processes. There are also many places that have big clusters full of multi-threaded processes. I don't know which group is the biggest, but it doesn't matter as far as Osterhout's argument is concerned. What matters is that, contrary to the context in which that argument is framed, wringing performance out of multiple CPUs in a single system is an almost universal need nowadays. Many of them furthermore need high levels of intra-process as well as inter-process parallelism for that purpose. Threading is not the only way to achieve that, but he never even considers other alternatives. No matter what hairs are split or who tries to claim the biggest credentials, the OP remains seriously time-bound and its applicability to modern systems is weak.