3 ms·

My impression, having worked with Infiniband for several years a few years ago, is that the drive was always to make it a supercomputing fabric rather than a ge

My impression, having worked with Infiniband for several years a few years ago, is that the drive was always to make it a supercomputing fabric rather than a general-purpose cluster fabric. The goal was to unseat Myrinet and other low-level technologies for MPI jobs where the job wouldn't be using TCP or IP for the communication. IPoIB got tacked on to placate the market segment that wanted to use tools that communicated with TCP/IP but wanted something faster than gigabit ethernet. Unfortunately for the few players in the space, the cluster ecosystem is moving away from those tightly-coupled systems to more flexible systems that can tolerate higher communication overheads.

greglindahl 11y ago
Actually, IPoIB was a customer requirement. In the Myrinet era, people frequently built clusters with a very poor ethernet, and used IPoverMyrinet to access storage etc. I don't agree with you about the way the ecosystem is headed. Tightly-coupled problems are tightly-coupled, period. It's true that there is a lot of interest in more loosely-connected clusters, but that doesn't get rid of the existing market.
- vonmoltke 11y ago
  > Actually, IPoIB was a customer requirement. In the Myrinet era, people frequently built clusters with a very poor ethernet, and used IPoverMyrinet to access storage etc. Interesting. I know we had a general-purpose cluster wired together with IPoIB just because it was the fastest option at the time. The production systems we were building would have IB for main communications using RDMA primitives in our middleware and a gigabit ethernet maintenance network for coordination and monitoring. > I don't agree with you about the way the ecosystem is headed. Tightly-coupled problems are tightly-coupled, period. It's true that there is a lot of interest in more loosely-connected clusters, but that doesn't get rid of the existing market. My point was more that MPI forces a greater degree of coupling to cluster architecture than some problems require, just because it was the way MPI was designed. Maybe I am overestimating the portion, but some number of problems solved in MPI prior to the development of more modern MapReduce frameworks was done simply because the problem needed the throughput of a cluster and MPI was the only way to use it without writing your own network middleware. That portion of the market is going away.
  - greglindahl 11y ago
    In my HPC career, I never met such a customer, other than some oil and gas folks doing the usual prestack depth migration algorithm, which is not a tightly-coupled problem.

greglindahl 11y ago

Actually, IPoIB was a customer requirement. In the Myrinet era, people frequently built clusters with a very poor ethernet, and used IPoverMyrinet to access storage etc. I don't agree with you about the way the ecosystem is headed. Tightly-coupled problems are tightly-coupled, period. It's true that there is a lot of interest in more loosely-connected clusters, but that doesn't get rid of the existing market.

vonmoltke 11y ago
> Actually, IPoIB was a customer requirement. In the Myrinet era, people frequently built clusters with a very poor ethernet, and used IPoverMyrinet to access storage etc. Interesting. I know we had a general-purpose cluster wired together with IPoIB just because it was the fastest option at the time. The production systems we were building would have IB for main communications using RDMA primitives in our middleware and a gigabit ethernet maintenance network for coordination and monitoring. > I don't agree with you about the way the ecosystem is headed. Tightly-coupled problems are tightly-coupled, period. It's true that there is a lot of interest in more loosely-connected clusters, but that doesn't get rid of the existing market. My point was more that MPI forces a greater degree of coupling to cluster architecture than some problems require, just because it was the way MPI was designed. Maybe I am overestimating the portion, but some number of problems solved in MPI prior to the development of more modern MapReduce frameworks was done simply because the problem needed the throughput of a cluster and MPI was the only way to use it without writing your own network middleware. That portion of the market is going away.
- greglindahl 11y ago
  In my HPC career, I never met such a customer, other than some oil and gas folks doing the usual prestack depth migration algorithm, which is not a tightly-coupled problem.