2 ms·

It's not a bandwidth issue. Functional programming (especially, say, actor systems) is better for organizing mental models of concurrency when your concurrency

It's not a bandwidth issue. Functional programming (especially, say, actor systems) is better for organizing mental models of concurrency when your concurrency is coupled with communication between the components. For hpc, you're typically optimizing for gustafson scaling (versus amdhal scaling) where you are running multiple copies of the same, computationally costly linearly organized code with no coupling between instances except statistical aggregation of results, so there is no particular benefit to functional-style concurrency. (And some FPLs,like Julia, are perfectly good at hpc anyways)

jandrewrogers 6y ago
FWIW, most supercomputing looks nothing like map-reduce; only the most trivial problems look like that. In a data model sense, a lot of supercomputing is join operation intensive, hence why they spend big bucks on high-bandwidth low-latency interconnects. STREAM benchmarks were more predictive of real-world supercomputing code performance than LAPACK in the majority of case 15+ years ago and it became more biased toward the former with time. The codes I worked on were complex graph analysis, spatiotemporal behavioral analysis, a bit of geospatial environmental modeling, and in prehistoric times thermal and mass transport modeling. These codes (pretty much anything involving reality) are intrinsically tightly coupled across compute nodes. Low-latency interconnects eventually gave way to latency-hiding software architectures but at no point did we use map-reduce as that would have been insanely inefficient given the sparsity and unpredictability of the interactions between nodes. These were the prototype software architectures for later high-performance databases. Every core is handling thousands or millions of independent shards of the larger computational model, which makes latency-hiding particularly efficient.
- physicsguy 6y ago
  This is similar to my experience too. If people can write out a single Python function and apply it to all of a large amount of data then great, but that isn’t the majority of supercomputing programming.
  - dnautics 6y ago
    I've done this across 10,000 cores, and helped people do similar tasks as well, I guess my experience is not normal. But IIRC even for things that, say use MPI to coordinate cores in the small you want to repeat across many cores, limit the spread across cores, limit your blocking coordination, lest you wind up with the old joke that "HPC is the art of turning a CPU-bound job into an I/O bound job".
- CraigJPerry 6y ago
  I have nothing to add, just wanted to say that reading all your comments this morning has been fascinating and educational. I’ve really enjoyed it so Thanks for sharing. This kind of expert insight is one of the reasons i visit HN.

jandrewrogers 6y ago

FWIW, most supercomputing looks nothing like map-reduce; only the most trivial problems look like that. In a data model sense, a lot of supercomputing is join operation intensive, hence why they spend big bucks on high-bandwidth low-latency interconnects. STREAM benchmarks were more predictive of real-world supercomputing code performance than LAPACK in the majority of case 15+ years ago and it became more biased toward the former with time. The codes I worked on were complex graph analysis, spatiotemporal behavioral analysis, a bit of geospatial environmental modeling, and in prehistoric times thermal and mass transport modeling. These codes (pretty much anything involving reality) are intrinsically tightly coupled across compute nodes. Low-latency interconnects eventually gave way to latency-hiding software architectures but at no point did we use map-reduce as that would have been insanely inefficient given the sparsity and unpredictability of the interactions between nodes. These were the prototype software architectures for later high-performance databases. Every core is handling thousands or millions of independent shards of the larger computational model, which makes latency-hiding particularly efficient.

physicsguy 6y ago
This is similar to my experience too. If people can write out a single Python function and apply it to all of a large amount of data then great, but that isn’t the majority of supercomputing programming.
- dnautics 6y ago
  I've done this across 10,000 cores, and helped people do similar tasks as well, I guess my experience is not normal. But IIRC even for things that, say use MPI to coordinate cores in the small you want to repeat across many cores, limit the spread across cores, limit your blocking coordination, lest you wind up with the old joke that "HPC is the art of turning a CPU-bound job into an I/O bound job".
CraigJPerry 6y ago
I have nothing to add, just wanted to say that reading all your comments this morning has been fascinating and educational. I’ve really enjoyed it so Thanks for sharing. This kind of expert insight is one of the reasons i visit HN.