4 ms·

Author response to “HPC is dying, and MPI is killing it” objections

37 points by cing 11y ago

nemothekid 11y ago
I read the first post, and now reading the second, and its difficult to parse because I'm not in academia. As far as I can tell, MPI is actually a standard/protocol for doing distributed computation, much like Hadoop & Spark. (Confused as to why I've never heard about it before - does it get absolutely 0 use outside of academia? Why?) If I have this right, then a more general title is "Specialized high performance machines are on the way out, commodity cluster computing is the future"
- seanmcdirmid 11y ago
  MPI is popular in the HPC community, not just in academia. If you are in the Big Data community, you might not ever have used it before...since Big Data is not really HPC. You wouldn't use Hadoop for a HPC job just like you wouldn't use MPI for a big data job! Heck, you might even use CUDA if your job is parallel enough with a small enough memory footprint (and there is interesting work on combining CUDA and MPI). MapReduce is kind of limiting, and HPC jobs require more pinned machine allocations anyways (if you schedule them like big data jobs, they perform like crap).
- tanderson92 11y ago
  It gets an incredible amount of use in the physical science and applied maths communities, and that's not only in academia. It's an extensive standard for message passing that allows distributed (non- shared-memory) computations. Multiple implementations of the standard exist. It is so widely used that entire software suites (Trilinos and PETSc, neither developed by academia but largely by the DOE) are built on top of its message passing framework. The author's point is that because it is such a large standard it is difficult to grasp and he would rather use something else. His opinion notwithstanding though, things are unlikely to change significantly, at least in the near future: even just today we saw an announcement of a massive new specialized cluster being built at Argonne. You can be sure most application programming will be done with MPI, not Spark. Commenter jeffsci on the original post is on point.
simonbyrne 11y ago
Discussion of previous article: https://news.ycombinator.com/item?id=9335441 https://news.ycombinator.com/item?id=9335441

37 points by cing 11y ago

I read the first post, and now reading the second, and its difficult to parse because I'm not in academia. As far as I can tell, MPI is actually a standard/protocol for doing distributed computation, much like Hadoop & Spark. (Confused as to why I've never heard about it before - does it get absolutely 0 use outside of academia? Why?) If I have this right, then a more general title is "Specialized high performance machines are on the way out, commodity cluster computing is the future"

seanmcdirmid 11y ago
MPI is popular in the HPC community, not just in academia. If you are in the Big Data community, you might not ever have used it before...since Big Data is not really HPC. You wouldn't use Hadoop for a HPC job just like you wouldn't use MPI for a big data job! Heck, you might even use CUDA if your job is parallel enough with a small enough memory footprint (and there is interesting work on combining CUDA and MPI). MapReduce is kind of limiting, and HPC jobs require more pinned machine allocations anyways (if you schedule them like big data jobs, they perform like crap).
tanderson92 11y ago
It gets an incredible amount of use in the physical science and applied maths communities, and that's not only in academia. It's an extensive standard for message passing that allows distributed (non- shared-memory) computations. Multiple implementations of the standard exist. It is so widely used that entire software suites (Trilinos and PETSc, neither developed by academia but largely by the DOE) are built on top of its message passing framework. The author's point is that because it is such a large standard it is difficult to grasp and he would rather use something else. His opinion notwithstanding though, things are unlikely to change significantly, at least in the near future: even just today we saw an announcement of a massive new specialized cluster being built at Argonne. You can be sure most application programming will be done with MPI, not Spark. Commenter jeffsci on the original post is on point.

simonbyrne 11y ago

Discussion of previous article: https://news.ycombinator.com/item?id=9335441 https://news.ycombinator.com/item?id=9335441