3 ms·
1000 10Gb ethernet doesn't usually make sense in the situations that are using Lustre and Infiniband; they actually need the reduced latency for inter-process c
by epistasis 10y ago
1000 10Gb ethernet doesn't usually make sense in the situations that are using Lustre and Infiniband; they actually need the reduced latency for inter-process communication. It may be fine for ethernet in front of the disks, but they still need to get the bits to the processors that are connected by infiniband.
Rewriting large existing codebases to use a different paradigm for storage and distribution may make economic sense, but it's a risky proposition. And when you get one shot at building a Top 500 cluster, taking that risk and delivering a cluster that can't run the code you need it to may mean that all that economic spend was wasted.
- dekhn 10y agoThe 10Gb's are for node to storage. The IB is for low latency. Having both in the same machine is a win. Wasting IB for storage is dumb, because storage times are orders of magnitude higher than the IB latency. Ultimately, rewriting your application to not require low-latency is not a risky proposition. it's in fact the least risky.
- gnufx 10y agoThat makes no sense to an HPC person (who has actually been evaluating low-latency Ethernet recently). How do you even separate the MPI-IO traffic over a different fabric from other MPI in, say, Open MPI, and how could maybe doubling network cost and complexity be a win if you could? And then we have to re-write the applications somehow? There's literature on improved performance running HDFS over Infiniband and Hadoop over Lustre and PVFS at least. And from up the thread, for a discussion of file systems for high performance checkpointing of MPI jobs (which is what's relevant to HPC) see, for instance http://nowlab.cse.ohio-state.edu/static/media/publications/abstract/rajachan-hpdc13.pdf http://nowlab.cse.ohio-state.edu/static/media/publications/a...
- dekhn 10y agoAn MPI implementation is free to spread traffic over available transports; there's nothing in the design of MPI that prevents such a configuration. The improved performance of HDFS over Infiniband comes at a price (the aggregation switches are much more expensive and challenging to configure; there are far fewer network engineers who understand them). As for the paper you linked to- I see numerous problems. First is the fact that HPC codes spend 60% of their time checkpointing to deal with MTBFs. This is not a problem in data warehouse applications- failures of components do not affect the job's progress (except in the form of slower progress). Second, the paper has the checkpoint data buffered in local RAM then spilled out to local or remote storage. That's fine, except that you're still at danger if the node crashes before the data is committed to true backing storage (or has a way to recover the spill process after repair/reboot).
- gnufx 10y agoThe question was how you'd configure a concrete MPI implementation to separate MPI-IO traffic from the rest if it was a good idea. We have the Infiniband (or similar) fabric, and we understand it -- in my case better than the Ethernet one we're proposed to have to understand as well, and we're typically not "network engineers" doing this stuff. I disagree that Infiniband is more challenging anyway, having had and known about bad problems with cluster Ethernet. Then this 10G Ethernet get a whole 10% extra bandwidth on top of current HPC fabrics while presumably adding jitter. HPC codes don't generally spend 60% checkpointing, and "data warehouse applications" typically aren't of interest to us, but we can do mapreduce-type things fast. See the bits about redundancy and the advantage of RDMA in the checkpointing filesystem.