5 ms·
I assume you're referencing MPI? It all depends on your problem domain. There are plenty of classes of problems where message passing incurs far more overhead t
by openasocket 9y ago
I assume you're referencing MPI? It all depends on your problem domain. There are plenty of classes of problems where message passing incurs far more overhead than just sharing memory.
- greglindahl 9y agoMPI is indeed something I used to do, but I was thinking more of large NoSQL setups like blekko's search engine, which was 100% event-driven code running in separate, heavy-weight processes. (OK, in the guts, we did have some utility software that did async disk I/O using a few C threads per heavyweight process, but that was due to the limited kernel API more than us wanting threads.)
- openasocket 9y agoSure, but the point is that outside of embarrassingly parallel problems you need shared memory access. You can simulate that with message passing, but for many problems the overhead of message passing is too great. I'm specifically thinking of linear algebra routines. I'd like to see someone write a performant BLAS implementation that doesn't make extensive use of threading
- zzzcpan 9y agoMessage passing lets you amortize synchronization and should always be faster if properly implemented.
- openasocket 9y agoNot really, no. You've still got the overhead of copying, which can be significant. There's no way you could implement something like, for example, Gaussian elimination with message passing in a way that's just as fast as using locks and barriers.
- greglindahl 9y agoI totally agree that zzzcpan was over-generalizing, but so are you: is it really the case that the Linpack benchmark (which uses arrays as large as memory) on the latest SGI shared-memory thing uses locks and barriers?
- zzzcpan 9y agoI think the cost of locks is so high, that even some primitive runtime zero-copying mechanism won't add significant overhead. After all, if you have to do something on another core, you have to copy it there, it's not really a physical shared memory.
- openasocket 9y agoMaybe not locks and barriers specifically, but I'd be shocked if it's using message passing, particularly because of the overhead of copying. If anyone has evidence to the contrary I'd be happy to hear it.
- greglindahl 9y agoWow. I guess you aren't familiar with the history of large SGI Origin and Altix machines being mostly used as pure MPI machines because that's almost always fastest? Also, Linpack doesn't involve much data motion. Also, moving data over a shared memory interconnect is as much of a copy as moving data over a message passing interconnect. I was going to look up the current details for you, but I don't see an SGI Ultraviolet system in the current Top500. How large of a machine do you work with?
- openasocket 9y agoI was thinking about machines with a shared memory bus (I'm 90% sure that's the right term) so yeah not things in the Top500. I'm thinking about your standard 64-core servers. I know that in the large MPI is best because getting that many cores to use the same memory bus just isn't feasible. Sorry for the confusion, I didn't read your comment carefully.
- greglindahl 9y agoWell, the example I was thinking of (from after I left HPC) was a search engine, in which most of the computation took place in a NoSQL database. And it did great with event-driven programming and heavy-weight processes. You can call it embarrassingly-parallel if you like, from an HPC point of view it is. And that's a fine thing. Glad to see HPC so represented here on HN, but an HPC counter-example does not diminish an example. To put this another way: I wasn't saying that YOUR problem could be solved without threads, I said MY problem could be solved without threads.
- openasocket 9y agoCompletely agree, and I prefer to use something like message passing whenever possible because for a lot of problems it is the natural solution and has fewer pitfalls than using raw threads.
- greglindahl 9y agoGreat. Nothing like a vigorous discussion that turns out to ... actually be an agreement.