5 ms·

This line in that post is mystifying: they peddled Infiniband, an inferior design based on bus technology, but repositioned as a point to point network

detaro 5y ago
I think that's a confusing reference to the fact that InfiniBand initially was supposed to be also a replacement for system-internal buses (i.e. a PCI replacement), before PCIe took that place instead.
- monocasa 5y ago
  Yeah that makes sense, it's an RDMA protocol, and PCIe has wayyyy more in common with infiniband then it does with legacy PCI.
- gnufx 5y ago
  I wasn't there, but understood it was intended for general machine-room -- I dislike "datacentres" -- use; Wikipedia implies it was intended for ~everything... Anyway, IB at least had lower latency than Myrinet, which is what counts in a lot of HPC. I don't remember the numbers, but I got quite close to Myrinet latency with the on-board 1GbE NICs on Sun x2200s, an un-managed switch, and Open-MX, which was interoperable with the Myrinet MX protocol. (I remember MX being slagged off by Mellanox; I don't know how similar MXM later was for IB.)
dboreham 5y ago
You can use a bus PHY in a p2p configuration.
- wmf 5y ago
  But Infiniband didn't do that.
  - paulmd 5y ago
    you can do point-to-point connections with infiniband, it's a pretty common homelab use-case since adapters are so cheap these days.
    - detaro 5y ago
      InfiniBand is always point-to-point, that's the point. The talk about "bus" is confusing/wrong since it never is a bus.
      - paulmd 5y ago
        oh, "bus" in the context of an actual bus, like CAN. I thought p2p just meaning direct-attach. But yeah it's switched, it's not a shared bus.
NavinF 5y ago
Yeah that makes no sense.
drewg123 5y ago
At least initially (~2005 or so?) we had customers who were given free IB for their HPC clusters ripping the IB out and replacing it with with paid Myrinet installations because they could never get their IB clusters to work reliably. Although we had the superior product, we could not market our way out of a wet paper bag. Our CEO counted on our (then) superior engineering & expected our products to market themselves. Eventually the IB engineering caught up and surpassed ours, but our marketing never even came close to theirs, so they were the clear winner.
- convolvatron 5y ago
  unless I'm thinking of the wrong vendor, the myrinet devices supported programming the nic. for some applications that was a really big win over IB verbs
  - drewg123 5y ago
    Yes and no. The original firmware / host driver stack was open (shared?) source called MyriAPI. It was so terrible that spawned an ecosystem of uni and research lab projects to implement replacements. I worked on one such project at Duke (Trapeze). Others included BIP, Fast Messages, Active Messages. MyriAPI was later replaced by a new firmware / host stack called GM. I think at some point we stopped shipping firmware source. I think that point was the transition from MyriAPI to GM...
- bee_rider 5y ago
  Where'd the name Myricom/Myrinet come from? "Infiniband" is a pretty cool name.
- sliken 5y ago
  I build a myrinet HPC cluster or two, vague memory that it was 2.5Gbit. Worked well, I particularly liked the cables (thin and optical, way nicer than IB cables). When I went to replace them the Myrinet 10G was repeatedly delayed, and seemed to be focusing on 10G ethernet, not MPI/HPC. We ended up going with a new upstart called Pathscale, which then was bought by qlogic, and then Intel. Our benchmarks showed a clear win over Mellanox at the time. In particular on larger jobs the Mellanox drivers seemed to consume a fair bit of ram for each client you were talking to, so even 32 nodes jobs would consume a fair bit of ram. We even built a cluster out of hypertransport connected infinipath cards in a "HTX" slow instead of PCIe. Managed I believe 1.4us latency, which is still reasonably competitive today. We built SDR (first public cluster), DDR (first public cluster), QDR based clusters. But then Intel said the new Infinipath would not be supported on AMD nodes, or even previous gen Intel nodes, I said it's a non starter and switched to Mellanox. I talked to several other HPC sites that said the same thing, months later Intel killed off the product. Sad and short sited on Intel's part.
deleted 5y ago
[deleted]

detaro 5y ago

I think that's a confusing reference to the fact that InfiniBand initially was supposed to be also a replacement for system-internal buses (i.e. a PCI replacement), before PCIe took that place instead.

monocasa 5y ago
Yeah that makes sense, it's an RDMA protocol, and PCIe has wayyyy more in common with infiniband then it does with legacy PCI.
gnufx 5y ago
I wasn't there, but understood it was intended for general machine-room -- I dislike "datacentres" -- use; Wikipedia implies it was intended for ~everything... Anyway, IB at least had lower latency than Myrinet, which is what counts in a lot of HPC. I don't remember the numbers, but I got quite close to Myrinet latency with the on-board 1GbE NICs on Sun x2200s, an un-managed switch, and Open-MX, which was interoperable with the Myrinet MX protocol. (I remember MX being slagged off by Mellanox; I don't know how similar MXM later was for IB.)

dboreham 5y ago

You can use a bus PHY in a p2p configuration.

wmf 5y ago
But Infiniband didn't do that.
- paulmd 5y ago
  you can do point-to-point connections with infiniband, it's a pretty common homelab use-case since adapters are so cheap these days.
  - detaro 5y ago
    InfiniBand is always point-to-point, that's the point. The talk about "bus" is confusing/wrong since it never is a bus.
    - paulmd 5y ago
      oh, "bus" in the context of an actual bus, like CAN. I thought p2p just meaning direct-attach. But yeah it's switched, it's not a shared bus.

NavinF 5y ago

Yeah that makes no sense.

drewg123 5y ago

At least initially (~2005 or so?) we had customers who were given free IB for their HPC clusters ripping the IB out and replacing it with with paid Myrinet installations because they could never get their IB clusters to work reliably. Although we had the superior product, we could not market our way out of a wet paper bag. Our CEO counted on our (then) superior engineering & expected our products to market themselves. Eventually the IB engineering caught up and surpassed ours, but our marketing never even came close to theirs, so they were the clear winner.

convolvatron 5y ago
unless I'm thinking of the wrong vendor, the myrinet devices supported programming the nic. for some applications that was a really big win over IB verbs
- drewg123 5y ago
  Yes and no. The original firmware / host driver stack was open (shared?) source called MyriAPI. It was so terrible that spawned an ecosystem of uni and research lab projects to implement replacements. I worked on one such project at Duke (Trapeze). Others included BIP, Fast Messages, Active Messages. MyriAPI was later replaced by a new firmware / host stack called GM. I think at some point we stopped shipping firmware source. I think that point was the transition from MyriAPI to GM...
bee_rider 5y ago
Where'd the name Myricom/Myrinet come from? "Infiniband" is a pretty cool name.
sliken 5y ago
I build a myrinet HPC cluster or two, vague memory that it was 2.5Gbit. Worked well, I particularly liked the cables (thin and optical, way nicer than IB cables). When I went to replace them the Myrinet 10G was repeatedly delayed, and seemed to be focusing on 10G ethernet, not MPI/HPC. We ended up going with a new upstart called Pathscale, which then was bought by qlogic, and then Intel. Our benchmarks showed a clear win over Mellanox at the time. In particular on larger jobs the Mellanox drivers seemed to consume a fair bit of ram for each client you were talking to, so even 32 nodes jobs would consume a fair bit of ram. We even built a cluster out of hypertransport connected infinipath cards in a "HTX" slow instead of PCIe. Managed I believe 1.4us latency, which is still reasonably competitive today. We built SDR (first public cluster), DDR (first public cluster), QDR based clusters. But then Intel said the new Infinipath would not be supported on AMD nodes, or even previous gen Intel nodes, I said it's a non starter and switched to Mellanox. I talked to several other HPC sites that said the same thing, months later Intel killed off the product. Sad and short sited on Intel's part.

deleted 5y ago

[deleted]