9 ms·

HPC is dying, and MPI is killing it

225 points by ljdursi 11y ago

siegela 11y ago
John: You make some solid points. Most though seem to more support the idea that more research investment/emphasis is badly needed for HPC programming models. From an application perspective, one sees such a dominant reliance on MPI+X primarily because typically the value proposition just isn't there yet for alternatives (at least in the areas where I work, where we have undertaken new developments recently and done fairly exhaustive evaluations of available options). Though the coding can be somewhat tedious and rigid, in the end these shortcomings have been outweighed by the typical practical considerations -- low risk, extremely portable, tight control over locality and thus performance, etc. It's obviously not all or nothing - as you say we could choose something even lower level and possibly get better performance, but when seen from the perspective of the near and mid-term application goals, it's hard to make a different choice unless explicitly tasked with doing so.
dalke 11y ago
I came into the essay with suspicion. A map-reduce system like Hadoop isn't a good fit for HPC problems, and I thought it would argue that MPI is old => it's stuck in the past. Instead, and to my joy, it was a well-reasoned essay with good, solid points. My only quibble is that Charm++ is not "a framework for particle simulation methods". While the molecular dynamics program NAMD has been using it for 20 years, which is why I know of Charm++, it wasn't designed specifically for particle simulation methods, nor is restricted to that topic. Quoting from http://charm.cs.illinois.edu/newPapers/08-09/paper.pdf http://charm.cs.illinois.edu/newPapers/08-09/paper.pdf : > NAMD, from our oldest collaboration, is a program for biomolecular modeling [2]; OpenAtom is a Car-Parinello MD program used for simulation of electronic structure (in nanomaterials, as well as biophysics) [3]; ChaNGa, an astronomy code [13]; and RocStar, a code for simulating solid-propellant rockets, such as those in the space shuttle solid rocket booster
- ljdursi 11y ago
  Thanks! I'm probably mischaracterizing Charm++ a bit, because I'm most familiar with it in particle context (OpenAtom, ChaNGa, NAMD). I guess it's probably particularly used in that context just because it's so good with very fine-grained distributions of work units. I'll edit that line in the article.
  - lvkale 11y ago
    (Thanks Andrew Dalke. You remember it after all these years!) Jonathan, actually, even in that set, OpenAtom is not a particle code. Its a quantum chemistry code where each electronic state is represented often by a large 3D array spread over processors.. For more recent examples of representative miniApps, see http://charmplusplus.org/benchmarks/ http://charmplusplus.org/benchmarks/ or our upcoming workshop (sorry for the plug: http://charm.cs.illinois.edu/charmWorkshop http://charm.cs.illinois.edu/charmWorkshop The broader article deserves further though, and I hope to find time to respond. But it is clear that raising the level of abstraction beyond MPI is necessary.
    - dalke 11y ago
      (Indeed I do!)
  - semi-extrinsic 11y ago
    Another nitpick: 128 cores equals ~ 4 nodes only if your problem is not bound by memory bandwidth. If it is, 128 cores equals ~ 16 cores, and then the interconnect matters a lot. Great writeup though. I do think we need to get more people into the mindset that MPI won't be the standard in 10 years, otherwise it will still be the standard in 10 years.
    - ljdursi 11y ago
      "won't be the standard in 10 years, otherwise it will still be the standard in 10 years." -- nicely said.
    - deleted 11y ago
      [deleted]
- threeseed 11y ago
  People need to be careful about talking about Hadoop as just a map-reduce system. It's YARN container system is flexible enough to run any JVM application. For example we use it to run an autoscaling ElasticSearch cluster alongside our Hadoop workloads. And we are actively investigating using it to run our Scala microservices.
  - dalke 11y ago
    Even years after Amazon started selling a lot more than just books, if people were asked "what does Amazon sell?", the answer was often "books." I looked at YARN now. I've not heard of it before. It doesn't look like it has anything to do with the topic at hand. How would one build an explicit solver for a 1D diffusion equation, corresponding to the examples given in the "HPC is dying, ..." article, using YARN? How do you do checkpointing so you can restart your 10 million atom simulation should there be a system fault after 2 weeks of run-time? (Checkpoints need about 220 MB; each atom has an x,y,z position as well as a vx,vy,vz velocity vector. Also, it needs to be at the same timestep across the entire distributed machine.) Instead, it looks like YARN is designed for service-based components, where the components are relatively independent from each other, and where failure recovery is mostly a matter of starting a new service and resending the request. If my understanding is correct, then it's certainly more capable than map-reduce. But not in a direction that's relevant for most current HPC.
    - threeseed 11y ago
      YARN is just a resource manager on top of which Hadoop jobs are run e.g. Hive, Pig. It is analogous to a set of Docker containers distributed across nodes. The same methods you would use synchronize state in that situation you could use with YARN. For example using a persistent distributed system e.g. Hazelcast to handle system failures and checkpointing. I am not saying this is some amazing solution to every HPC problem only that Hadoop is far, far more flexible than many people give it credit for.
      - dalke 11y ago
        I understand your last paragraph. Looking this time at Hazelcast, what I see is layers of code to understand before being able to do something simple. It really does look like all of the technology you are pointing to is solving a different problem. It's not related to any of the HPC needs I've heard of. Parts of my simulation are out of phase. I need some gather step to collect the data from individual nodes, when a given timestep is reached, and save the state. A simple solution is to do a barrier every ~30 minutes, send to the master node, and have it save the data. When I look at Hazelcast I see what looks to be a different sort of clustering - using clusters for redundancy, and not for CPU power. Eg, I see "Hazelcast keeps the backup of each data entry on multiple nodes", and I think "I don't care." If a node goes down, the system goes down, and I restart from a checkpoint. It's much more likely that one of the 512 compute nodes will go down than some database node. I'll withdraw my original statement that "A map-reduce system like Hadoop" and say simply "a system like Hadoop isn't a good fit for HPC problems". Here's a lovely essay which agrees with me ;) http://glennklockwood.blogspot.com.au/2014/05/hadoops-uncomfortable-fit-in-hpc.html http://glennklockwood.blogspot.com.au/2014/05/hadoops-uncomf... . It considers the questions: > Why does Hadoop remain at the fringe of high-performance computing, and what will it take for it to be a serious solution in HPC?
ToastyMallows 11y ago
MPI is Message Passing Interface, for those who don't know. Author never explicitly says what MPI stands for. https://en.wikipedia.org/wiki/Message_Passing_Interface https://en.wikipedia.org/wiki/Message_Passing_Interface
- dredmorbius 11y ago
  Thank you. Using TLAs without defining them should be a bannable offense.
- serve_yay 11y ago
  Indeed he does not.
- anigbrowl 11y ago
  Yeah, I had to go back to google after failing to find it defined anywhere on the page. If you don't use something regularly yourself, you may remember the concept but forget the acronym. For want of a couple of sentences of context at the outset, the rest of the article was quite inaccessible. Maybe there's a lesson for the author here.
  - _delirium 11y ago
    It wouldn't be hard to define up front, but in context I don't think assuming that the audience knows what MPI is poses much of a problem for the goals of the piece. This isn't a general-interest piece about HPC, but specifically an advocacy piece attempting to convince members of the HPC community that their strong attachment to MPI is detrimental to the field, and that they should refocus their efforts elsewhere. If someone doesn't know what MPI is, they are probably not strongly attached to it (and probably not in HPC), so aren't the people the author is trying to convince.
  - markhahn 11y ago
    a lesson for the author xor the reader.
- kenko 11y ago
  "MPI, the Message Passing Interface, began as a needed standardization above a dizzying array of..." The capitalization and apposition makes it pretty explicit.
Animats 11y ago
Yes, "high performance computing" is dying. There's no commercial market for it. Check the list of the top 500 supercomputers in the world.[1] The top 10 are all Government operations. In the top 25, there are a few oil companies, mostly running big arrays of Intel Xeons. CPU clock speeds maxed out between 3-4GHz a decade ago. Nobody develops special supercomputing CPUs any more. The market is tiny. Old supercomputer guys reminisce about the glory days when IBM, Cray, Control Data, and UNIVAC devoted their best R&D efforts to supercomputers. That ended 30 years ago. Supercomputers have poor price-performance. Grosch's Law [2] stopped working a long time ago. Maximum price/performance today is achieved with racks of midrange CPUs, which is why that's what every commercial data center has. Now everybody has to deal with clusters of machines. So cluster interconnection has become mainstream, not the province of supercomputing. [1] http://www.top500.org/list/2014/11/ http://www.top500.org/list/2014/11/ [2] http://en.wikipedia.org/wiki/Grosch%27s_law http://en.wikipedia.org/wiki/Grosch%27s_law
- wffurr 11y ago
  Did you RTFA? It's about MPI: "MPI is a language-independent communications protocol used to program parallel computers." Runs fine on commodity clusters.
  - spott 11y ago
    >Runs fine on commodity clusters. Kind of.... For simple, low communication jobs this is true. But when you start trying to find the eigenvectors of a large sparse matrix, communication becomes your bottleneck, at which point MPI on commodity clusters (those without a really fancy interconnect) "works", but not fast enough to be useful.
    - vonmoltke 11y ago
      I don't think "really fancy interconnects" makes a cluster not commodity, since the original post in this thread is about supercomputer processors. You can put Infiniband in any system with a PCI-e 3.0 bus.
    - 11y ago
shepardrtc 11y ago
The Fortran Standards Committee is attempting to make HPC easier through the use of coarrays, which are essentially massive abstractions over MPI. I really wish people would give Fortran a second chance. It has come a long way from the ancient, all-caps days.
- na85 11y ago
  I spent a few months learning (modern) Fortran a year or two ago. My chief obstacle was the difficulty involved in finding modern tutorials. I don't want to have to read tutorials written in 1994 whose focus is getting people used to F77 up to speed. I've yet to find a tutorial that approached teaching F08 as if it was a new language, which is what I feel is needed. Even in F08 there's a lot of backwards-compatibility cruft still left in the language, too. The IO model still provides very little abstraction and is based on tape drives. You can/have to "rewind" files. There are obscure "unit descriptors" that manifest themselves as integer literals in most code posted online which makes it a chore to learn from. As far as I can tell there is no functionality that approximates the behaviour of C++'s streams. It's fast as hell, and the GNU compiler is mature and well-developed, but Fortran remains a horrid language for doing any sort of interactive programming. It's best used if you just give it some arguments, let it run free, and then have it return some object or value that a more sane language can then interpret and present to the user for a decision. There is little reason to learn a language where the only sane choice for doing input/output involves calling your Fortran module from a python script and letting the python handle i/o.
  - duskwuff 11y ago
    > You can/have to "rewind" files. This isn't necessarily a Fortran-specific thing. The standard C library includes a rewind(fd) function, equivalent to lseek(fd, 0, SEEK_SET).
  - semi-extrinsic 11y ago
    I also think we need someone to put some real effort into making gdb usable for Fortran dynamic/automatic arrays. That's a real PITA currently.
acadien 11y ago
I've heard there is a new effort being led by Torsten Hoefler to modernize MPI and address a number of the issues mentioned in this article. http://htor.inf.ethz.ch/ http://htor.inf.ethz.ch/ I was at a talk of his last year and there are a number of fault tolerant MP algorithms being drawn in. MPI hasn't been updated in ages, I don't think that necessarily means we need to ditch it, it just means the standard needs to be modernized. I don't feel very strongly about this since working with MPI is a huge pain in the ass and it seems like the challenge of modernizing it is just gargantuan. Also I'm not familiar with spark, but isn't Chapel a decade old at this point and barely works at all? I tried their compiler last summer and it took 5 minutes to compile hello world, hopefully its improving.
- tormeh 11y ago
  A decade is no age for a language, though. Creating a language with compiler can be done very quickly, but creating a good language with a good compiler and a good standard library takes time. And then it needs to catch on. This requires about a decade++. Scala is 12 years old, Go is 6 years old and Clojure is 8 years old.
  - acadien 11y ago
    Yeah good point. I just felt it might be a misleading of the author to suggest Chapel as an alternative when you cannot possibly write a useful program with it.
    - vandot 11y ago
      There are numerous benchmarks implemented in Chapel, some of which are competitive with other implementations (see paper reference in article). There is a growing standard library and literally thousands of test codes that represent a broad set of functionality. That said, Chapel is not yet a product grade language, nor is it promoted as such. Chapel may not be an appropriate replacement for all MPI programs, but it can be used for some programs today.
      - 11y ago
rz2k 11y ago
The article briefly mentions Erlang with its focus on message passing, and the focus on being fault tolerant should be another benefit. I could find a few mentions of Erlang used for simulations, such as here[1], but I am curious whether there is much actual usage in scientific computing, or whether there are some problems in practice. [1] https://books.google.com/books?id=p0h9vAb1m7IC&pg=PA365#v=onepage&q&f=false https://books.google.com/books?id=p0h9vAb1m7IC&pg=PA365#v=on...
- shepardrtc 11y ago
  Erlang itself is not good for the type of numerical computation typical done in HPC. It is amazing as a backend, and there are several ways it can call code written in other languages through the use of ports or NIFS, but if you try to do massive number-crunching using its own libraries then you're going to be unhappy with the results.
  - rz2k 11y ago
    How about in simulations where a large number of cellular automata are interacting with each other, but individually only carrying out simple computations?
    - sampo 11y ago
      You cannot make an efficient fluid mechanics simulation on a 4000x4000x4000 grid if you set up a separate process for each individual gridcell. More efficient to just store your numbers in 3d arrays.
      - noobermin 11y ago
        why not?
        shepardrtc 11y ago
        If you store the data in arrays, you can use matrix multiplication libraries such as Intel's MKL or OpenBLAS, which are written to be exceptionally optimized for use on multiple cores. I cannot emphasize enough how much time and effort has been put into these libraries to multiply matrices as fast as can possibly be done. If you use processes such as in the Erlang VM, they're doing calculations, sure, but they're also sending messages back and forth, and they're acting as supervisors, and they're being shuffled around by the VM. There's a lot going on. And that extra stuff that's going on takes away from the time you could be multiplying stuff. And even then, there's been no optimization done for this sort of calculation. There are a lot of tricks you can do. Heck, the better matrix multiplication libraries have individual optimizations for CPUs.
MichaelCrawford 11y ago
This happened with Smalltalk. I was a Smalltalk coder. I thought it was the best thing since sliced bread. It has always been clear to me that Smalltalk is far superior to Java. I left the company after a little while, to do C++ graphics. I later heard that my former employer rewrote their Smalltalk application in Java. Now no one uses Smalltalk anymore. While Objective-C is based on Smalltalk, Smalltalk was far easier to use, however lots of people use Objective-C. No one uses Smalltalk. How could it have been different? My friend Kurt Thames once said that "Smalltalk is the way object-oriented programming SHOULD be done." I have always agreed with that. But when new methods (!) of OOP arose, all the Smalltalk crowd did was gripe about how Smalltalk was far better than Java or Objective-C.
- wglb 11y ago
  Ah, but I think that the comparison doesn't hold. In my opinion, Smalltalk was the right way to do all of what we do, and Java took the enterprise mindset by storm. There was an enormous project at a very large Insurance company near Chicago that was written in Smalltalk, but got abandoned for some obscure reason. I think that Smalltalk has the right level of abstraction and a lot of very good other things about it. MPI was, as the article points out, the wrong abstraction for the problem. If MPI dies, I am ok with that. I am sad that Smalltalk is not more widely used.
  - MichaelCrawford 11y ago
    "Java took the enterprise by storm" Smalltalk's demise no doubt had a lot to do with Sun's marketing people convincing a bunch of Pointy-Haired Bosses that garbage collection means that you have no memory leaks, as well as that Java was the only way to do cross-platform development. IMHO Java is one of the very worst ways to do cross-platform, however when I ported a a Mac OS Pascal program to Java so that it could be run on both Windows and Mac, the client was completely convinced that Java was the only way that could possibly be done - this despite my loud and frequent protests that the state of Java at the time was quite poor, that the Java interpreter was dog-slow, that Java sucked the memory dry, and that I knew a whole bunch of ways to write cross-platform native code that would be far faster and use far-less memory. The reason that Smalltalk specifically suffered from this, is that Sun made most of its money by selling servers to the enterprise. Sun Workstations were favored by scientists and engineers, however Sun's real money came from enterprise servers. Right around that time, Smalltalk was largely used for enterprise applications. Enterprise application developers loved Smalltalk absolutely to death however the bean counters and the PHBs were more inclined to listen to marketdroids promises about garbage collection being immune to memory leaks. Garbage collection and memory leaks are orthogonal.
sampo 11y ago
in High Performance Computing, there is (1) 3-dimensional simulations (weather, fluid dynamics, structural mechanics, all kinds of physics simulations, like magnetic storms in space or nuclear reactors etc.) and then there is (2) everything else, like data mining, machine learning, genomics etc. Some of the sparse matrix computations in structural mechanics and in some machine learning algorithms have some overlap. But mostly, group 2 has little reason to be interested in what group 1 is doing. Now, group 2 obviously has more modern tools than the 3d-simulation community, because machine learning came to common use much later that numerical fluid mechanics. But do 3d-simulation people also have much reason to be interested in what the machine learning people are doing? The "machine learning / big data" people are probably not doing anything that makes a weather prediction model to run faster? Or are they?
- sgt101 11y ago
  They are doing it (interesting things) for lower capex and lower development costs. On opex, good for operations, bad for power consumption (relatively). In terms of absolute performance HPC is absolutely faster. In terms of bangs for bucks, Big Data is hands down faster. Also in terms of accessibility Big Data is hugely easier - I can build you a 100 core big data system for $300k
  - markhahn 11y ago
    that's silly: HPC has been pinching pennies before big data was a thing. and the computer industry is biz: you get what you pay for. if you can live with Gb performance, you can drop around $2k (IB card, cables, switches) off your price. But it's not as if the hardware is any different, faster or more accessible.
    - sgt101 11y ago
      I think it's economics, GPU's are sold by the million, super computers interconnects are sold by the thousands. Commodity kit is mass produced spreading design, vvt and manufacturing tooling costs. The hardware is different in terms of the layout. Aggregations of small cores on boards (gpus) vs. very high speed large cores with lots of local memory. Highly localised connections vs. an interconnect fabric. And it is more accessible because it's affordable, and you can get at it in the cloud; this means that skills building is easier for more people and it also means that a wider user base is possible.
east2west 11y ago
I am trying to get into distributed computing so this article is particularly interesting to me. I may be mistaken so please excuse my naivety if my points are off marks. I thought MPI was mainly geared towards communication-heavy tasks where the underlying network is specialized, for example infiniband or bus between CPUs. One use of MPI is to manage distributed memory tasks between different physical CPUs while threads run on multiple cores of same CPU. Spark, I believe, doesn't handle cases like this well because JVM hides low level details. I have read papers that propose to layer MPI over RDMA rather than expose a flat memory model, which came as a surprise to me but it shows the flexibility of MPI. One thing unclear to me what performance we can expect from MPI when we use commodity network gears, and how it compares to Spark. The article is absolutely correct MPI leaves robustness to user and that is today an oversight. Modern Hadoop ecosystem is designed for different workload from MPI's. It emphasizes co-localizing date and computation, seamless robustness,and trades off raw power for simple programmingmodels. MapReduce turns out too simple, so Spark implements graph execution, which is nothing new to HPC. As far I know Spark's authors don't believe it is ready for distributed numerical linear algebra yet. But a counterpoint is that I am seeing machine learning libraries using Spark, so perhaps things are improving. One thing I have learnt today is that MPI isn't gaining popularity. I just have a hard time picturing a JVM language in overall control in HPC where precise control of memory is paramount to performance.
- ljdursi 11y ago
  I agree that languages that rely on tracing GC seem like they're fundamentally at a disadvantage when it comes to pushing the envelope of single-node performance; the best article I've read arguing this was actually in the context of mobile games, rather than HPC, but I can't for the life of me find the article now. I don't know if Spark itself is the right way forward; but it's an example of a very productive high-level language for certain forms of distributed memory computing. And some of these issues - like the JVM - aren't fundamental to Spark's approach; there's no inherent reason why something similar couldn't be built based on C++ or the like.
  - east2west 11y ago
    I totally agree that the programming model of Spark is the right direction. I dream of the day when compiler and OS cooperate to expose a simple interface to distributed memory and an optimal execution-communication system, kind of like Cilk but for clusters. BTW, thanks for a thought provoking article. You have given me a lot to ponder.
bjwbell 11y ago
Too bad he didn't talk about GPGPU killing MPI too or not. I don't know enough to say. I'm not familiar with the HPC space but I thought a lot of new work, at least in machine learning, was migrating to GPGPU instead of traditional CPUs. The compute per $ or per watt payoff is too large to ignore.
- ljdursi 11y ago
  I think with GPGPU, the issue is a little different; it's local computation, so a bit orthogonal to distributed-memory coordination. But it is interesting to see how many higher-level libraries and other tools (like OpenMP4/OpenACC) are springing up around GPU usage. It's hard not to be a bit jealous...
- markhahn 11y ago
  GPUs have a large advantage in a very narrow niche: regular and very intensive ops on regular and compact data. ML is not completely ideal (because it's not that compute-intensive), but MC simulations often are. Most things are not ideal for GPUs, especially since it's often not obvious how to nicely scale across GPUs. MPI's strongest point is that it lets you take very good advantage of any topology of general-purpose computers: multicore, NUMA, distributed-memory. Models that emphasize data-parallel programming (co-array fortran, etc) suit GPUs much better. But nothing is going to change the fact that GPU registers are much faster than GPU (card) memory, which is faster than any possible interconnect.
- stephen272 11y ago
  I wouldn't say GPUs are killing off MPI. You still need some way to pass data between nodes/GPUs (most of these datasets can't fit within a single GPU). What you are seeing though, is less and less use of the CPU. If code developers decide to use GPUs, they try and move their data onto the GPU and keep it there for as long as possible (data movement across PCIe is a killer for performance). ORNL's new machine Sumiit [1] will have 1/4 the nodes of their current machine, Titan, but multiple GPUs per node. Thus emphasizing the move away from CPUs and onto GPUs. Again though, there are still 3K nodes and you need some efficient way to pass data between those nodes. [1] https://www.olcf.ornl.gov/summit/ https://www.olcf.ornl.gov/summit/
pjmlp 11y ago
This made me feel old, as I remember the days when we got to learn PVM at distributed computing classes and MPI was presented as something that some people were kind of working on.
- dalke 11y ago
  If it makes you feel better, I started with PVM 2.
Dylan16807 11y ago
W-why are they using that picture at the start when their source is an essay on exactly the way they're misusing it?
acomjean 11y ago
I used MPI (Message Passing Interface...) back in the day. But it was a pain, especially since our code was a mix of c (which was easy to mpi) and ada (not so easy). Its pretty low level stuff (I think we used Open MPI). All the nodes need to have MPI set up and configured, fine if you have a team willing to do it but these days.... mpiexec -n 10 myprocess I think we liked it because the processes would be put to sleep by the mpi daemon until a message arrived. You can sleep and wait for a message with sockets now I think. Its been a while since I've used the unix IPC (Interprocess communication). I don't think I'll miss it.
wglb 11y ago
This is a good article. I built a tiny cluster in my basement (a prototype) and looked at MPI and decided that it was way to complicated, so I just built something that pushes the essential bits, pretty much without abstraction, to the nodes and was done. The cluster is a very specific solution, so I felt justified in not looking at MPI. And now the decision feels even more justified.
lego12 11y ago
The sad thing is that MPI-3 doesn't have wide acceptance yet. A lot of organizations mainly use MPI-1/2. While using RHEL 6.x. Shoutout to red hat for maintaining ancient software
kxyvr 11y ago
The lion in the room is that the DOE National Laboratories have a huge amount of code tied up in MPI and they continue to spend millions of dollars both on hardware and software to support this infrastructure. If you look at the top 500 list: http://www.top500.org/lists/2014/11/ http://www.top500.org/lists/2014/11/ Four out the ten computers are owned by DOE. That's a pretty significant investment, so they're going to be reluctant to change over to a different system. And, to be clear, a different software setup could be used on these systems, but they were almost certainly purchased with the idea that their existing MPI codes would work well on them. Hell, MPICH was partially authored by Argonne: http://www.mcs.anl.gov/project/mpich-high-performance-portable-implementation-mpi http://www.mcs.anl.gov/project/mpich-high-performance-portab... so they've a vested interest in seeing this community stay consistent. Now, on the technical merits, is it possible to do better? Of course. That being said, part of the reason that DOE invested so heavily in this infrastructure is that they often solve physics based problems based on PDE formulations. Here, we're basically using either a finite element, finite difference, or finite volume based method and it turns out that there's quite a bit of experience writing these codes with MPI. Certainly, GPUs have made a big impact on things like finite difference codes, but you still have to distribute data for these problems across a cluster of computers because they require too much memory to store locally. Right now, this can be done in a moderately straight forward way with MPI. Well, more specifically, people end up using DOE libraries like PETSc or Trilinos to do this for them and they're based on MPI. It's not perfect, but it works and scales well. Thus far, I've not seen anything that improves upon this enough to convince these teams to abandon their MPI infrastructure. Again, this is not to say that this setup is perfect. I also believe that this setup has caused a certain amount of stagnation (read huge amount) in the HPC community and that's bad. However, in order to convince DOE that there's something better than MPI, someone has to put together some scalable codes that vastly outperform (or are vastly easier to use, code, or maintain) the problems that they care about. Very specifically, these are PDE discretizations of continuum mechanics based problems using either finite different, finite element, or finite volume methods in 3D. The 1-D diffusion problem in the article is nice, but 3-D is a pain in the ass, everyone knows it, and you can not get even a casual glance shy of 3-D problems. That sucks and is not fair, but that's the reality of the community. By the way, the oil industry basically mirrors the sentiment of DOE as well. They're huge consumers of the same technology and the same sort of problems. If someone is curious, check out reverse time migration or full wave inversion. There are billions of dollars tied up in these two problems and they have a huge amount of MPI code. If someone can solve these problems better using a new technology, there's a huge amount of money in it. So far, no one has done it because that's a huge investment and hard.
Xcelerate 11y ago
Thank goodness someone said it. I get tired of using ancient software that's older than me for relatively simply tasks that should have long ago been coded into a higher level of abstraction. I programmed a pair-correlation function calculation routine using MPI once -- yech. Fortran, MPI, even C to an extent -- can we please move on? I don't understand why the scientific community is so reluctant to embrace change. It seriously doesn't take that long to learn a new language or a platform like Github (yeah, that's still considered "new" in the scientific community), and the time investment more than pays itself back many times over.
- pron 11y ago
  > I don't understand why the scientific community is so reluctant to embrace change. Let's assume the opposite were true, and it was fast to embrace change. How much time would be spent on this change -- relearning, rewriting, refighting old bugs -- vs. actual work done? Change is overhead. You do as little of it as necessary, and only when not changing starts costing a lot. Which means you change, but slowly. As to Fortran, it will go away when something better comes along, and then it will do so slowly, for the aforementioned reasons.

225 points by ljdursi 11y ago

John: You make some solid points. Most though seem to more support the idea that more research investment/emphasis is badly needed for HPC programming models. From an application perspective, one sees such a dominant reliance on MPI+X primarily because typically the value proposition just isn't there yet for alternatives (at least in the areas where I work, where we have undertaken new developments recently and done fairly exhaustive evaluations of available options). Though the coding can be somewhat tedious and rigid, in the end these shortcomings have been outweighed by the typical practical considerations -- low risk, extremely portable, tight control over locality and thus performance, etc. It's obviously not all or nothing - as you say we could choose something even lower level and possibly get better performance, but when seen from the perspective of the near and mid-term application goals, it's hard to make a different choice unless explicitly tasked with doing so.

dalke 11y ago

I came into the essay with suspicion. A map-reduce system like Hadoop isn't a good fit for HPC problems, and I thought it would argue that MPI is old => it's stuck in the past. Instead, and to my joy, it was a well-reasoned essay with good, solid points. My only quibble is that Charm++ is not "a framework for particle simulation methods". While the molecular dynamics program NAMD has been using it for 20 years, which is why I know of Charm++, it wasn't designed specifically for particle simulation methods, nor is restricted to that topic. Quoting from http://charm.cs.illinois.edu/newPapers/08-09/paper.pdf http://charm.cs.illinois.edu/newPapers/08-09/paper.pdf : > NAMD, from our oldest collaboration, is a program for biomolecular modeling [2]; OpenAtom is a Car-Parinello MD program used for simulation of electronic structure (in nanomaterials, as well as biophysics) [3]; ChaNGa, an astronomy code [13]; and RocStar, a code for simulating solid-propellant rockets, such as those in the space shuttle solid rocket booster

ljdursi 11y ago
Thanks! I'm probably mischaracterizing Charm++ a bit, because I'm most familiar with it in particle context (OpenAtom, ChaNGa, NAMD). I guess it's probably particularly used in that context just because it's so good with very fine-grained distributions of work units. I'll edit that line in the article.
- lvkale 11y ago
  (Thanks Andrew Dalke. You remember it after all these years!) Jonathan, actually, even in that set, OpenAtom is not a particle code. Its a quantum chemistry code where each electronic state is represented often by a large 3D array spread over processors.. For more recent examples of representative miniApps, see http://charmplusplus.org/benchmarks/ http://charmplusplus.org/benchmarks/ or our upcoming workshop (sorry for the plug: http://charm.cs.illinois.edu/charmWorkshop http://charm.cs.illinois.edu/charmWorkshop The broader article deserves further though, and I hope to find time to respond. But it is clear that raising the level of abstraction beyond MPI is necessary.
  - dalke 11y ago
    (Indeed I do!)
- semi-extrinsic 11y ago
  Another nitpick: 128 cores equals ~ 4 nodes only if your problem is not bound by memory bandwidth. If it is, 128 cores equals ~ 16 cores, and then the interconnect matters a lot. Great writeup though. I do think we need to get more people into the mindset that MPI won't be the standard in 10 years, otherwise it will still be the standard in 10 years.
  - ljdursi 11y ago
    "won't be the standard in 10 years, otherwise it will still be the standard in 10 years." -- nicely said.
  - deleted 11y ago
    [deleted]
threeseed 11y ago
People need to be careful about talking about Hadoop as just a map-reduce system. It's YARN container system is flexible enough to run any JVM application. For example we use it to run an autoscaling ElasticSearch cluster alongside our Hadoop workloads. And we are actively investigating using it to run our Scala microservices.
- dalke 11y ago
  Even years after Amazon started selling a lot more than just books, if people were asked "what does Amazon sell?", the answer was often "books." I looked at YARN now. I've not heard of it before. It doesn't look like it has anything to do with the topic at hand. How would one build an explicit solver for a 1D diffusion equation, corresponding to the examples given in the "HPC is dying, ..." article, using YARN? How do you do checkpointing so you can restart your 10 million atom simulation should there be a system fault after 2 weeks of run-time? (Checkpoints need about 220 MB; each atom has an x,y,z position as well as a vx,vy,vz velocity vector. Also, it needs to be at the same timestep across the entire distributed machine.) Instead, it looks like YARN is designed for service-based components, where the components are relatively independent from each other, and where failure recovery is mostly a matter of starting a new service and resending the request. If my understanding is correct, then it's certainly more capable than map-reduce. But not in a direction that's relevant for most current HPC.
  - threeseed 11y ago
    YARN is just a resource manager on top of which Hadoop jobs are run e.g. Hive, Pig. It is analogous to a set of Docker containers distributed across nodes. The same methods you would use synchronize state in that situation you could use with YARN. For example using a persistent distributed system e.g. Hazelcast to handle system failures and checkpointing. I am not saying this is some amazing solution to every HPC problem only that Hadoop is far, far more flexible than many people give it credit for.
    - dalke 11y ago
      I understand your last paragraph. Looking this time at Hazelcast, what I see is layers of code to understand before being able to do something simple. It really does look like all of the technology you are pointing to is solving a different problem. It's not related to any of the HPC needs I've heard of. Parts of my simulation are out of phase. I need some gather step to collect the data from individual nodes, when a given timestep is reached, and save the state. A simple solution is to do a barrier every ~30 minutes, send to the master node, and have it save the data. When I look at Hazelcast I see what looks to be a different sort of clustering - using clusters for redundancy, and not for CPU power. Eg, I see "Hazelcast keeps the backup of each data entry on multiple nodes", and I think "I don't care." If a node goes down, the system goes down, and I restart from a checkpoint. It's much more likely that one of the 512 compute nodes will go down than some database node. I'll withdraw my original statement that "A map-reduce system like Hadoop" and say simply "a system like Hadoop isn't a good fit for HPC problems". Here's a lovely essay which agrees with me ;) http://glennklockwood.blogspot.com.au/2014/05/hadoops-uncomfortable-fit-in-hpc.html http://glennklockwood.blogspot.com.au/2014/05/hadoops-uncomf... . It considers the questions: > Why does Hadoop remain at the fringe of high-performance computing, and what will it take for it to be a serious solution in HPC?

ToastyMallows 11y ago

MPI is Message Passing Interface, for those who don't know. Author never explicitly says what MPI stands for. https://en.wikipedia.org/wiki/Message_Passing_Interface https://en.wikipedia.org/wiki/Message_Passing_Interface

dredmorbius 11y ago
Thank you. Using TLAs without defining them should be a bannable offense.
serve_yay 11y ago
Indeed he does not.
anigbrowl 11y ago
Yeah, I had to go back to google after failing to find it defined anywhere on the page. If you don't use something regularly yourself, you may remember the concept but forget the acronym. For want of a couple of sentences of context at the outset, the rest of the article was quite inaccessible. Maybe there's a lesson for the author here.
- _delirium 11y ago
  It wouldn't be hard to define up front, but in context I don't think assuming that the audience knows what MPI is poses much of a problem for the goals of the piece. This isn't a general-interest piece about HPC, but specifically an advocacy piece attempting to convince members of the HPC community that their strong attachment to MPI is detrimental to the field, and that they should refocus their efforts elsewhere. If someone doesn't know what MPI is, they are probably not strongly attached to it (and probably not in HPC), so aren't the people the author is trying to convince.
- markhahn 11y ago
  a lesson for the author xor the reader.
kenko 11y ago
"MPI, the Message Passing Interface, began as a needed standardization above a dizzying array of..." The capitalization and apposition makes it pretty explicit.

Animats 11y ago

Yes, "high performance computing" is dying. There's no commercial market for it. Check the list of the top 500 supercomputers in the world.[1] The top 10 are all Government operations. In the top 25, there are a few oil companies, mostly running big arrays of Intel Xeons. CPU clock speeds maxed out between 3-4GHz a decade ago. Nobody develops special supercomputing CPUs any more. The market is tiny. Old supercomputer guys reminisce about the glory days when IBM, Cray, Control Data, and UNIVAC devoted their best R&D efforts to supercomputers. That ended 30 years ago. Supercomputers have poor price-performance. Grosch's Law [2] stopped working a long time ago. Maximum price/performance today is achieved with racks of midrange CPUs, which is why that's what every commercial data center has. Now everybody has to deal with clusters of machines. So cluster interconnection has become mainstream, not the province of supercomputing. [1] http://www.top500.org/list/2014/11/ http://www.top500.org/list/2014/11/ [2] http://en.wikipedia.org/wiki/Grosch%27s_law http://en.wikipedia.org/wiki/Grosch%27s_law

wffurr 11y ago
Did you RTFA? It's about MPI: "MPI is a language-independent communications protocol used to program parallel computers." Runs fine on commodity clusters.
- spott 11y ago
  >Runs fine on commodity clusters. Kind of.... For simple, low communication jobs this is true. But when you start trying to find the eigenvectors of a large sparse matrix, communication becomes your bottleneck, at which point MPI on commodity clusters (those without a really fancy interconnect) "works", but not fast enough to be useful.
  - vonmoltke 11y ago
    I don't think "really fancy interconnects" makes a cluster not commodity, since the original post in this thread is about supercomputer processors. You can put Infiniband in any system with a PCI-e 3.0 bus.
  - 11y ago

shepardrtc 11y ago

The Fortran Standards Committee is attempting to make HPC easier through the use of coarrays, which are essentially massive abstractions over MPI. I really wish people would give Fortran a second chance. It has come a long way from the ancient, all-caps days.

na85 11y ago
I spent a few months learning (modern) Fortran a year or two ago. My chief obstacle was the difficulty involved in finding modern tutorials. I don't want to have to read tutorials written in 1994 whose focus is getting people used to F77 up to speed. I've yet to find a tutorial that approached teaching F08 as if it was a new language, which is what I feel is needed. Even in F08 there's a lot of backwards-compatibility cruft still left in the language, too. The IO model still provides very little abstraction and is based on tape drives. You can/have to "rewind" files. There are obscure "unit descriptors" that manifest themselves as integer literals in most code posted online which makes it a chore to learn from. As far as I can tell there is no functionality that approximates the behaviour of C++'s streams. It's fast as hell, and the GNU compiler is mature and well-developed, but Fortran remains a horrid language for doing any sort of interactive programming. It's best used if you just give it some arguments, let it run free, and then have it return some object or value that a more sane language can then interpret and present to the user for a decision. There is little reason to learn a language where the only sane choice for doing input/output involves calling your Fortran module from a python script and letting the python handle i/o.
- duskwuff 11y ago
  > You can/have to "rewind" files. This isn't necessarily a Fortran-specific thing. The standard C library includes a rewind(fd) function, equivalent to lseek(fd, 0, SEEK_SET).
- semi-extrinsic 11y ago
  I also think we need someone to put some real effort into making gdb usable for Fortran dynamic/automatic arrays. That's a real PITA currently.

acadien 11y ago

I've heard there is a new effort being led by Torsten Hoefler to modernize MPI and address a number of the issues mentioned in this article. http://htor.inf.ethz.ch/ http://htor.inf.ethz.ch/ I was at a talk of his last year and there are a number of fault tolerant MP algorithms being drawn in. MPI hasn't been updated in ages, I don't think that necessarily means we need to ditch it, it just means the standard needs to be modernized. I don't feel very strongly about this since working with MPI is a huge pain in the ass and it seems like the challenge of modernizing it is just gargantuan. Also I'm not familiar with spark, but isn't Chapel a decade old at this point and barely works at all? I tried their compiler last summer and it took 5 minutes to compile hello world, hopefully its improving.

tormeh 11y ago
A decade is no age for a language, though. Creating a language with compiler can be done very quickly, but creating a good language with a good compiler and a good standard library takes time. And then it needs to catch on. This requires about a decade++. Scala is 12 years old, Go is 6 years old and Clojure is 8 years old.
- acadien 11y ago
  Yeah good point. I just felt it might be a misleading of the author to suggest Chapel as an alternative when you cannot possibly write a useful program with it.
  - vandot 11y ago
    There are numerous benchmarks implemented in Chapel, some of which are competitive with other implementations (see paper reference in article). There is a growing standard library and literally thousands of test codes that represent a broad set of functionality. That said, Chapel is not yet a product grade language, nor is it promoted as such. Chapel may not be an appropriate replacement for all MPI programs, but it can be used for some programs today.
    - 11y ago

rz2k 11y ago

The article briefly mentions Erlang with its focus on message passing, and the focus on being fault tolerant should be another benefit. I could find a few mentions of Erlang used for simulations, such as here[1], but I am curious whether there is much actual usage in scientific computing, or whether there are some problems in practice. [1] https://books.google.com/books?id=p0h9vAb1m7IC&pg=PA365#v=onepage&q&f=false https://books.google.com/books?id=p0h9vAb1m7IC&pg=PA365#v=on...

shepardrtc 11y ago
Erlang itself is not good for the type of numerical computation typical done in HPC. It is amazing as a backend, and there are several ways it can call code written in other languages through the use of ports or NIFS, but if you try to do massive number-crunching using its own libraries then you're going to be unhappy with the results.
- rz2k 11y ago
  How about in simulations where a large number of cellular automata are interacting with each other, but individually only carrying out simple computations?
  - sampo 11y ago
    You cannot make an efficient fluid mechanics simulation on a 4000x4000x4000 grid if you set up a separate process for each individual gridcell. More efficient to just store your numbers in 3d arrays.
    - noobermin 11y ago
      why not?
      - shepardrtc 11y ago
        If you store the data in arrays, you can use matrix multiplication libraries such as Intel's MKL or OpenBLAS, which are written to be exceptionally optimized for use on multiple cores. I cannot emphasize enough how much time and effort has been put into these libraries to multiply matrices as fast as can possibly be done. If you use processes such as in the Erlang VM, they're doing calculations, sure, but they're also sending messages back and forth, and they're acting as supervisors, and they're being shuffled around by the VM. There's a lot going on. And that extra stuff that's going on takes away from the time you could be multiplying stuff. And even then, there's been no optimization done for this sort of calculation. There are a lot of tricks you can do. Heck, the better matrix multiplication libraries have individual optimizations for CPUs.

MichaelCrawford 11y ago

This happened with Smalltalk. I was a Smalltalk coder. I thought it was the best thing since sliced bread. It has always been clear to me that Smalltalk is far superior to Java. I left the company after a little while, to do C++ graphics. I later heard that my former employer rewrote their Smalltalk application in Java. Now no one uses Smalltalk anymore. While Objective-C is based on Smalltalk, Smalltalk was far easier to use, however lots of people use Objective-C. No one uses Smalltalk. How could it have been different? My friend Kurt Thames once said that "Smalltalk is the way object-oriented programming SHOULD be done." I have always agreed with that. But when new methods (!) of OOP arose, all the Smalltalk crowd did was gripe about how Smalltalk was far better than Java or Objective-C.

wglb 11y ago
Ah, but I think that the comparison doesn't hold. In my opinion, Smalltalk was the right way to do all of what we do, and Java took the enterprise mindset by storm. There was an enormous project at a very large Insurance company near Chicago that was written in Smalltalk, but got abandoned for some obscure reason. I think that Smalltalk has the right level of abstraction and a lot of very good other things about it. MPI was, as the article points out, the wrong abstraction for the problem. If MPI dies, I am ok with that. I am sad that Smalltalk is not more widely used.
- MichaelCrawford 11y ago
  "Java took the enterprise by storm" Smalltalk's demise no doubt had a lot to do with Sun's marketing people convincing a bunch of Pointy-Haired Bosses that garbage collection means that you have no memory leaks, as well as that Java was the only way to do cross-platform development. IMHO Java is one of the very worst ways to do cross-platform, however when I ported a a Mac OS Pascal program to Java so that it could be run on both Windows and Mac, the client was completely convinced that Java was the only way that could possibly be done - this despite my loud and frequent protests that the state of Java at the time was quite poor, that the Java interpreter was dog-slow, that Java sucked the memory dry, and that I knew a whole bunch of ways to write cross-platform native code that would be far faster and use far-less memory. The reason that Smalltalk specifically suffered from this, is that Sun made most of its money by selling servers to the enterprise. Sun Workstations were favored by scientists and engineers, however Sun's real money came from enterprise servers. Right around that time, Smalltalk was largely used for enterprise applications. Enterprise application developers loved Smalltalk absolutely to death however the bean counters and the PHBs were more inclined to listen to marketdroids promises about garbage collection being immune to memory leaks. Garbage collection and memory leaks are orthogonal.

sampo 11y ago

in High Performance Computing, there is (1) 3-dimensional simulations (weather, fluid dynamics, structural mechanics, all kinds of physics simulations, like magnetic storms in space or nuclear reactors etc.) and then there is (2) everything else, like data mining, machine learning, genomics etc. Some of the sparse matrix computations in structural mechanics and in some machine learning algorithms have some overlap. But mostly, group 2 has little reason to be interested in what group 1 is doing. Now, group 2 obviously has more modern tools than the 3d-simulation community, because machine learning came to common use much later that numerical fluid mechanics. But do 3d-simulation people also have much reason to be interested in what the machine learning people are doing? The "machine learning / big data" people are probably not doing anything that makes a weather prediction model to run faster? Or are they?

sgt101 11y ago
They are doing it (interesting things) for lower capex and lower development costs. On opex, good for operations, bad for power consumption (relatively). In terms of absolute performance HPC is absolutely faster. In terms of bangs for bucks, Big Data is hands down faster. Also in terms of accessibility Big Data is hugely easier - I can build you a 100 core big data system for $300k
- markhahn 11y ago
  that's silly: HPC has been pinching pennies before big data was a thing. and the computer industry is biz: you get what you pay for. if you can live with Gb performance, you can drop around $2k (IB card, cables, switches) off your price. But it's not as if the hardware is any different, faster or more accessible.
  - sgt101 11y ago
    I think it's economics, GPU's are sold by the million, super computers interconnects are sold by the thousands. Commodity kit is mass produced spreading design, vvt and manufacturing tooling costs. The hardware is different in terms of the layout. Aggregations of small cores on boards (gpus) vs. very high speed large cores with lots of local memory. Highly localised connections vs. an interconnect fabric. And it is more accessible because it's affordable, and you can get at it in the cloud; this means that skills building is easier for more people and it also means that a wider user base is possible.

east2west 11y ago

I am trying to get into distributed computing so this article is particularly interesting to me. I may be mistaken so please excuse my naivety if my points are off marks. I thought MPI was mainly geared towards communication-heavy tasks where the underlying network is specialized, for example infiniband or bus between CPUs. One use of MPI is to manage distributed memory tasks between different physical CPUs while threads run on multiple cores of same CPU. Spark, I believe, doesn't handle cases like this well because JVM hides low level details. I have read papers that propose to layer MPI over RDMA rather than expose a flat memory model, which came as a surprise to me but it shows the flexibility of MPI. One thing unclear to me what performance we can expect from MPI when we use commodity network gears, and how it compares to Spark. The article is absolutely correct MPI leaves robustness to user and that is today an oversight. Modern Hadoop ecosystem is designed for different workload from MPI's. It emphasizes co-localizing date and computation, seamless robustness,and trades off raw power for simple programmingmodels. MapReduce turns out too simple, so Spark implements graph execution, which is nothing new to HPC. As far I know Spark's authors don't believe it is ready for distributed numerical linear algebra yet. But a counterpoint is that I am seeing machine learning libraries using Spark, so perhaps things are improving. One thing I have learnt today is that MPI isn't gaining popularity. I just have a hard time picturing a JVM language in overall control in HPC where precise control of memory is paramount to performance.

ljdursi 11y ago
I agree that languages that rely on tracing GC seem like they're fundamentally at a disadvantage when it comes to pushing the envelope of single-node performance; the best article I've read arguing this was actually in the context of mobile games, rather than HPC, but I can't for the life of me find the article now. I don't know if Spark itself is the right way forward; but it's an example of a very productive high-level language for certain forms of distributed memory computing. And some of these issues - like the JVM - aren't fundamental to Spark's approach; there's no inherent reason why something similar couldn't be built based on C++ or the like.
- east2west 11y ago
  I totally agree that the programming model of Spark is the right direction. I dream of the day when compiler and OS cooperate to expose a simple interface to distributed memory and an optimal execution-communication system, kind of like Cilk but for clusters. BTW, thanks for a thought provoking article. You have given me a lot to ponder.

bjwbell 11y ago

Too bad he didn't talk about GPGPU killing MPI too or not. I don't know enough to say. I'm not familiar with the HPC space but I thought a lot of new work, at least in machine learning, was migrating to GPGPU instead of traditional CPUs. The compute per $ or per watt payoff is too large to ignore.

ljdursi 11y ago
I think with GPGPU, the issue is a little different; it's local computation, so a bit orthogonal to distributed-memory coordination. But it is interesting to see how many higher-level libraries and other tools (like OpenMP4/OpenACC) are springing up around GPU usage. It's hard not to be a bit jealous...
markhahn 11y ago
GPUs have a large advantage in a very narrow niche: regular and very intensive ops on regular and compact data. ML is not completely ideal (because it's not that compute-intensive), but MC simulations often are. Most things are not ideal for GPUs, especially since it's often not obvious how to nicely scale across GPUs. MPI's strongest point is that it lets you take very good advantage of any topology of general-purpose computers: multicore, NUMA, distributed-memory. Models that emphasize data-parallel programming (co-array fortran, etc) suit GPUs much better. But nothing is going to change the fact that GPU registers are much faster than GPU (card) memory, which is faster than any possible interconnect.
stephen272 11y ago
I wouldn't say GPUs are killing off MPI. You still need some way to pass data between nodes/GPUs (most of these datasets can't fit within a single GPU). What you are seeing though, is less and less use of the CPU. If code developers decide to use GPUs, they try and move their data onto the GPU and keep it there for as long as possible (data movement across PCIe is a killer for performance). ORNL's new machine Sumiit [1] will have 1/4 the nodes of their current machine, Titan, but multiple GPUs per node. Thus emphasizing the move away from CPUs and onto GPUs. Again though, there are still 3K nodes and you need some efficient way to pass data between those nodes. [1] https://www.olcf.ornl.gov/summit/ https://www.olcf.ornl.gov/summit/

pjmlp 11y ago

This made me feel old, as I remember the days when we got to learn PVM at distributed computing classes and MPI was presented as something that some people were kind of working on.

dalke 11y ago
If it makes you feel better, I started with PVM 2.

Dylan16807 11y ago

W-why are they using that picture at the start when their source is an essay on exactly the way they're misusing it?

acomjean 11y ago

I used MPI (Message Passing Interface...) back in the day. But it was a pain, especially since our code was a mix of c (which was easy to mpi) and ada (not so easy). Its pretty low level stuff (I think we used Open MPI). All the nodes need to have MPI set up and configured, fine if you have a team willing to do it but these days.... mpiexec -n 10 myprocess I think we liked it because the processes would be put to sleep by the mpi daemon until a message arrived. You can sleep and wait for a message with sockets now I think. Its been a while since I've used the unix IPC (Interprocess communication). I don't think I'll miss it.

wglb 11y ago

This is a good article. I built a tiny cluster in my basement (a prototype) and looked at MPI and decided that it was way to complicated, so I just built something that pushes the essential bits, pretty much without abstraction, to the nodes and was done. The cluster is a very specific solution, so I felt justified in not looking at MPI. And now the decision feels even more justified.

lego12 11y ago

The sad thing is that MPI-3 doesn't have wide acceptance yet. A lot of organizations mainly use MPI-1/2. While using RHEL 6.x. Shoutout to red hat for maintaining ancient software

kxyvr 11y ago

The lion in the room is that the DOE National Laboratories have a huge amount of code tied up in MPI and they continue to spend millions of dollars both on hardware and software to support this infrastructure. If you look at the top 500 list: http://www.top500.org/lists/2014/11/ http://www.top500.org/lists/2014/11/ Four out the ten computers are owned by DOE. That's a pretty significant investment, so they're going to be reluctant to change over to a different system. And, to be clear, a different software setup could be used on these systems, but they were almost certainly purchased with the idea that their existing MPI codes would work well on them. Hell, MPICH was partially authored by Argonne: http://www.mcs.anl.gov/project/mpich-high-performance-portable-implementation-mpi http://www.mcs.anl.gov/project/mpich-high-performance-portab... so they've a vested interest in seeing this community stay consistent. Now, on the technical merits, is it possible to do better? Of course. That being said, part of the reason that DOE invested so heavily in this infrastructure is that they often solve physics based problems based on PDE formulations. Here, we're basically using either a finite element, finite difference, or finite volume based method and it turns out that there's quite a bit of experience writing these codes with MPI. Certainly, GPUs have made a big impact on things like finite difference codes, but you still have to distribute data for these problems across a cluster of computers because they require too much memory to store locally. Right now, this can be done in a moderately straight forward way with MPI. Well, more specifically, people end up using DOE libraries like PETSc or Trilinos to do this for them and they're based on MPI. It's not perfect, but it works and scales well. Thus far, I've not seen anything that improves upon this enough to convince these teams to abandon their MPI infrastructure. Again, this is not to say that this setup is perfect. I also believe that this setup has caused a certain amount of stagnation (read huge amount) in the HPC community and that's bad. However, in order to convince DOE that there's something better than MPI, someone has to put together some scalable codes that vastly outperform (or are vastly easier to use, code, or maintain) the problems that they care about. Very specifically, these are PDE discretizations of continuum mechanics based problems using either finite different, finite element, or finite volume methods in 3D. The 1-D diffusion problem in the article is nice, but 3-D is a pain in the ass, everyone knows it, and you can not get even a casual glance shy of 3-D problems. That sucks and is not fair, but that's the reality of the community. By the way, the oil industry basically mirrors the sentiment of DOE as well. They're huge consumers of the same technology and the same sort of problems. If someone is curious, check out reverse time migration or full wave inversion. There are billions of dollars tied up in these two problems and they have a huge amount of MPI code. If someone can solve these problems better using a new technology, there's a huge amount of money in it. So far, no one has done it because that's a huge investment and hard.

Xcelerate 11y ago

Thank goodness someone said it. I get tired of using ancient software that's older than me for relatively simply tasks that should have long ago been coded into a higher level of abstraction. I programmed a pair-correlation function calculation routine using MPI once -- yech. Fortran, MPI, even C to an extent -- can we please move on? I don't understand why the scientific community is so reluctant to embrace change. It seriously doesn't take that long to learn a new language or a platform like Github (yeah, that's still considered "new" in the scientific community), and the time investment more than pays itself back many times over.

pron 11y ago
> I don't understand why the scientific community is so reluctant to embrace change. Let's assume the opposite were true, and it was fast to embrace change. How much time would be spent on this change -- relearning, rewriting, refighting old bugs -- vs. actual work done? Change is overhead. You do as little of it as necessary, and only when not changing starts costing a lot. Which means you change, but slowly. As to Fortran, it will go away when something better comes along, and then it will do so slowly, for the aforementioned reasons.