4 ms·
An MPI implementation is free to spread traffic over available transports; there's nothing in the design of MPI that prevents such a configuration. The improve
by dekhn 10y ago
An MPI implementation is free to spread traffic over available transports; there's nothing in the design of MPI that prevents such a configuration.
The improved performance of HDFS over Infiniband comes at a price (the aggregation switches are much more expensive and challenging to configure; there are far fewer network engineers who understand them).
As for the paper you linked to- I see numerous problems. First is the fact that HPC codes spend 60% of their time checkpointing to deal with MTBFs. This is not a problem in data warehouse applications- failures of components do not affect the job's progress (except in the form of slower progress). Second, the paper has the checkpoint data buffered in local RAM then spilled out to local or remote storage. That's fine, except that you're still at danger if the node crashes before the data is committed to true backing storage (or has a way to recover the spill process after repair/reboot).
- gnufx 10y agoThe question was how you'd configure a concrete MPI implementation to separate MPI-IO traffic from the rest if it was a good idea. We have the Infiniband (or similar) fabric, and we understand it -- in my case better than the Ethernet one we're proposed to have to understand as well, and we're typically not "network engineers" doing this stuff. I disagree that Infiniband is more challenging anyway, having had and known about bad problems with cluster Ethernet. Then this 10G Ethernet get a whole 10% extra bandwidth on top of current HPC fabrics while presumably adding jitter. HPC codes don't generally spend 60% checkpointing, and "data warehouse applications" typically aren't of interest to us, but we can do mapreduce-type things fast. See the bits about redundancy and the advantage of RDMA in the checkpointing filesystem.