4 ms·

I came into the essay with suspicion. A map-reduce system like Hadoop isn't a good fit for HPC problems, and I thought it would argue that MPI is old => it's st

by dalke 12y ago

I came into the essay with suspicion. A map-reduce system like Hadoop isn't a good fit for HPC problems, and I thought it would argue that MPI is old => it's stuck in the past. Instead, and to my joy, it was a well-reasoned essay with good, solid points. My only quibble is that Charm++ is not "a framework for particle simulation methods". While the molecular dynamics program NAMD has been using it for 20 years, which is why I know of Charm++, it wasn't designed specifically for particle simulation methods, nor is restricted to that topic. Quoting from http://charm.cs.illinois.edu/newPapers/08-09/paper.pdf http://charm.cs.illinois.edu/newPapers/08-09/paper.pdf : > NAMD, from our oldest collaboration, is a program for biomolecular modeling [2]; OpenAtom is a Car-Parinello MD program used for simulation of electronic structure (in nanomaterials, as well as biophysics) [3]; ChaNGa, an astronomy code [13]; and RocStar, a code for simulating solid-propellant rockets, such as those in the space shuttle solid rocket booster

ljdursi 12y ago
Thanks! I'm probably mischaracterizing Charm++ a bit, because I'm most familiar with it in particle context (OpenAtom, ChaNGa, NAMD). I guess it's probably particularly used in that context just because it's so good with very fine-grained distributions of work units. I'll edit that line in the article.
- lvkale 12y ago
  (Thanks Andrew Dalke. You remember it after all these years!) Jonathan, actually, even in that set, OpenAtom is not a particle code. Its a quantum chemistry code where each electronic state is represented often by a large 3D array spread over processors.. For more recent examples of representative miniApps, see http://charmplusplus.org/benchmarks/ http://charmplusplus.org/benchmarks/ or our upcoming workshop (sorry for the plug: http://charm.cs.illinois.edu/charmWorkshop http://charm.cs.illinois.edu/charmWorkshop The broader article deserves further though, and I hope to find time to respond. But it is clear that raising the level of abstraction beyond MPI is necessary.
  - dalke 12y ago
    (Indeed I do!)
- semi-extrinsic 12y ago
  Another nitpick: 128 cores equals ~ 4 nodes only if your problem is not bound by memory bandwidth. If it is, 128 cores equals ~ 16 cores, and then the interconnect matters a lot. Great writeup though. I do think we need to get more people into the mindset that MPI won't be the standard in 10 years, otherwise it will still be the standard in 10 years.
  - ljdursi 12y ago
    "won't be the standard in 10 years, otherwise it will still be the standard in 10 years." -- nicely said.
  - deleted 12y ago
    [deleted]
threeseed 12y ago
People need to be careful about talking about Hadoop as just a map-reduce system. It's YARN container system is flexible enough to run any JVM application. For example we use it to run an autoscaling ElasticSearch cluster alongside our Hadoop workloads. And we are actively investigating using it to run our Scala microservices.
- dalke 12y ago
  Even years after Amazon started selling a lot more than just books, if people were asked "what does Amazon sell?", the answer was often "books." I looked at YARN now. I've not heard of it before. It doesn't look like it has anything to do with the topic at hand. How would one build an explicit solver for a 1D diffusion equation, corresponding to the examples given in the "HPC is dying, ..." article, using YARN? How do you do checkpointing so you can restart your 10 million atom simulation should there be a system fault after 2 weeks of run-time? (Checkpoints need about 220 MB; each atom has an x,y,z position as well as a vx,vy,vz velocity vector. Also, it needs to be at the same timestep across the entire distributed machine.) Instead, it looks like YARN is designed for service-based components, where the components are relatively independent from each other, and where failure recovery is mostly a matter of starting a new service and resending the request. If my understanding is correct, then it's certainly more capable than map-reduce. But not in a direction that's relevant for most current HPC.
  - threeseed 12y ago
    YARN is just a resource manager on top of which Hadoop jobs are run e.g. Hive, Pig. It is analogous to a set of Docker containers distributed across nodes. The same methods you would use synchronize state in that situation you could use with YARN. For example using a persistent distributed system e.g. Hazelcast to handle system failures and checkpointing. I am not saying this is some amazing solution to every HPC problem only that Hadoop is far, far more flexible than many people give it credit for.
    - dalke 12y ago
      I understand your last paragraph. Looking this time at Hazelcast, what I see is layers of code to understand before being able to do something simple. It really does look like all of the technology you are pointing to is solving a different problem. It's not related to any of the HPC needs I've heard of. Parts of my simulation are out of phase. I need some gather step to collect the data from individual nodes, when a given timestep is reached, and save the state. A simple solution is to do a barrier every ~30 minutes, send to the master node, and have it save the data. When I look at Hazelcast I see what looks to be a different sort of clustering - using clusters for redundancy, and not for CPU power. Eg, I see "Hazelcast keeps the backup of each data entry on multiple nodes", and I think "I don't care." If a node goes down, the system goes down, and I restart from a checkpoint. It's much more likely that one of the 512 compute nodes will go down than some database node. I'll withdraw my original statement that "A map-reduce system like Hadoop" and say simply "a system like Hadoop isn't a good fit for HPC problems". Here's a lovely essay which agrees with me ;) http://glennklockwood.blogspot.com.au/2014/05/hadoops-uncomfortable-fit-in-hpc.html http://glennklockwood.blogspot.com.au/2014/05/hadoops-uncomf... . It considers the questions: > Why does Hadoop remain at the fringe of high-performance computing, and what will it take for it to be a serious solution in HPC?

by dalke 12y ago

ljdursi 12y ago

Thanks! I'm probably mischaracterizing Charm++ a bit, because I'm most familiar with it in particle context (OpenAtom, ChaNGa, NAMD). I guess it's probably particularly used in that context just because it's so good with very fine-grained distributions of work units. I'll edit that line in the article.

lvkale 12y ago
(Thanks Andrew Dalke. You remember it after all these years!) Jonathan, actually, even in that set, OpenAtom is not a particle code. Its a quantum chemistry code where each electronic state is represented often by a large 3D array spread over processors.. For more recent examples of representative miniApps, see http://charmplusplus.org/benchmarks/ http://charmplusplus.org/benchmarks/ or our upcoming workshop (sorry for the plug: http://charm.cs.illinois.edu/charmWorkshop http://charm.cs.illinois.edu/charmWorkshop The broader article deserves further though, and I hope to find time to respond. But it is clear that raising the level of abstraction beyond MPI is necessary.
- dalke 12y ago
  (Indeed I do!)
semi-extrinsic 12y ago
Another nitpick: 128 cores equals ~ 4 nodes only if your problem is not bound by memory bandwidth. If it is, 128 cores equals ~ 16 cores, and then the interconnect matters a lot. Great writeup though. I do think we need to get more people into the mindset that MPI won't be the standard in 10 years, otherwise it will still be the standard in 10 years.
- ljdursi 12y ago
  "won't be the standard in 10 years, otherwise it will still be the standard in 10 years." -- nicely said.
- deleted 12y ago
  [deleted]

threeseed 12y ago

People need to be careful about talking about Hadoop as just a map-reduce system. It's YARN container system is flexible enough to run any JVM application. For example we use it to run an autoscaling ElasticSearch cluster alongside our Hadoop workloads. And we are actively investigating using it to run our Scala microservices.

dalke 12y ago
Even years after Amazon started selling a lot more than just books, if people were asked "what does Amazon sell?", the answer was often "books." I looked at YARN now. I've not heard of it before. It doesn't look like it has anything to do with the topic at hand. How would one build an explicit solver for a 1D diffusion equation, corresponding to the examples given in the "HPC is dying, ..." article, using YARN? How do you do checkpointing so you can restart your 10 million atom simulation should there be a system fault after 2 weeks of run-time? (Checkpoints need about 220 MB; each atom has an x,y,z position as well as a vx,vy,vz velocity vector. Also, it needs to be at the same timestep across the entire distributed machine.) Instead, it looks like YARN is designed for service-based components, where the components are relatively independent from each other, and where failure recovery is mostly a matter of starting a new service and resending the request. If my understanding is correct, then it's certainly more capable than map-reduce. But not in a direction that's relevant for most current HPC.
- threeseed 12y ago
  YARN is just a resource manager on top of which Hadoop jobs are run e.g. Hive, Pig. It is analogous to a set of Docker containers distributed across nodes. The same methods you would use synchronize state in that situation you could use with YARN. For example using a persistent distributed system e.g. Hazelcast to handle system failures and checkpointing. I am not saying this is some amazing solution to every HPC problem only that Hadoop is far, far more flexible than many people give it credit for.
  - dalke 12y ago
    I understand your last paragraph. Looking this time at Hazelcast, what I see is layers of code to understand before being able to do something simple. It really does look like all of the technology you are pointing to is solving a different problem. It's not related to any of the HPC needs I've heard of. Parts of my simulation are out of phase. I need some gather step to collect the data from individual nodes, when a given timestep is reached, and save the state. A simple solution is to do a barrier every ~30 minutes, send to the master node, and have it save the data. When I look at Hazelcast I see what looks to be a different sort of clustering - using clusters for redundancy, and not for CPU power. Eg, I see "Hazelcast keeps the backup of each data entry on multiple nodes", and I think "I don't care." If a node goes down, the system goes down, and I restart from a checkpoint. It's much more likely that one of the 512 compute nodes will go down than some database node. I'll withdraw my original statement that "A map-reduce system like Hadoop" and say simply "a system like Hadoop isn't a good fit for HPC problems". Here's a lovely essay which agrees with me ;) http://glennklockwood.blogspot.com.au/2014/05/hadoops-uncomfortable-fit-in-hpc.html http://glennklockwood.blogspot.com.au/2014/05/hadoops-uncomf... . It considers the questions: > Why does Hadoop remain at the fringe of high-performance computing, and what will it take for it to be a serious solution in HPC?