3 ms·

I think that Infiniband was previously an open standard, but over time only Mellanox was in the market ( as other companies stopped making product) and they wer

I think that Infiniband was previously an open standard, but over time only Mellanox was in the market ( as other companies stopped making product) and they were then bought by Nvidia. I don't agree with Jim Keller in that absolute highest performance is found in Infiniband, not only with bandwidth but latency.

justinclift 3y ago
> I don't agree [...] absolute highest performance is found in Infiniband, not only with bandwidth but latency. What's the alternative(s) you're thinking of? There are some specialist things inside supercomputers (in the cases that Infiniband isn't used), but there doesn't seem to be anything else reasonably available commercially. Omni-path (a descendant of Infiniband) doesn't seem to have worked out either. --- Infiniband also has advantages over Ethernet in other things, like bundling links together. With (standardised) ethernet, adding more cables/links between two hosts gives more overall bandwidth in total. But the additional links can't be used to increase the bandwidth of a single stream. With Infiniband, when you add links they just combine together into a single bigger pipe, which can transfer more traffic in a single stream. Less need to jump up to higher tier adapters and switches with IB for that, you can just run more links between the hosts.
- shrubble 3y ago
  I re-worded what I wrote for clarity, it's an obscure sort of sentence construction. "I don't agree with Jim Keller in that absolute highest performance is found in Infiniband, not only with bandwidth but latency. " I don't agree with Jim Keller who says that Ethernet would be as good or better, in that when you use Infiniband, you get the absolute highest performance from IB, since IB does best on bandwidth and latency considerations.
  - justinclift 3y ago
    No worries, my misunderstanding. :)
- oasisaimlessly 3y ago
  You can definitely bond multiple Ethernet links together ("IEEE 802.3ad link aggregation").
  - justinclift 3y ago
    Sure, but ethernet doesn't let you combine the entire bandwidth of those links into a bigger fat pipe for a single stream. It more designed for higher bandwidth going to multiple destinations. A super common example of this is when consumers want to upgrade their existing home infrastructure to use a new NAS, and they start hitting the limits of 1GbE. Their first question is often along the lines of "can't I just add more links together to give more bandwidth?". Then people need to explain "No, ethernet doesn't work that way. You need to move up to 2.5/5/10GbE". (etc) Infiniband doesn't have that problem.
    - sheepdestroyer 3y ago
      It, not Ethernet that doesn't support the bandwidth aggregation you want, but the protocol over it. Multi Stream TCP seems to be one solution to this problem
    - saltcured 3y ago
      This used to be done, using what was called round-robin mode in bonded links. I think you are right that this mode never found its way into 802.11ad. Instead, parallel stacks of switches would be deployed, so that hosts were configured to bond links to N switches but the switches were not aware of the bonding. It was a commonly used Linux option to modestly scale up with low cost hardware. A single stream would have its frames striped over all the links in the bond, due to the configuration of the sending host. The flows would remain parallel and arrive over N switches to N NIC ports on the destination host. Care had to be taken with topology to keep parallel paths from end to end such as in one datacenter. Generally, flows could fan-in somewhere on the path and then go over a higher speed uplink, but would never fan-out again to parallel links. The drawback is more frame (packet) reordering, so it is even more sensitive to buffering and congestion control effects. If there were switch failures, the flows could become unbalanced if the topology offered redundant cross-links, since this would cause some fan-in on the surviving switches. Otherwise, completely independent switch stacks could be used, so a failure means traffic is dropped. Then, the host needs to do some kind of remote peer connectivity check to decide whether each link is healthy or not, and drop unhealthy links from the bond. Ironically, this could be done in consumer-grade LANs, since it does not depend on bonding features of the switches. However, careful setup is needed with consideration of the topology and switch behavior, to avoid pathology.

justinclift 3y ago

> I don't agree [...] absolute highest performance is found in Infiniband, not only with bandwidth but latency. What's the alternative(s) you're thinking of? There are some specialist things inside supercomputers (in the cases that Infiniband isn't used), but there doesn't seem to be anything else reasonably available commercially. Omni-path (a descendant of Infiniband) doesn't seem to have worked out either. --- Infiniband also has advantages over Ethernet in other things, like bundling links together. With (standardised) ethernet, adding more cables/links between two hosts gives more overall bandwidth in total. But the additional links can't be used to increase the bandwidth of a single stream. With Infiniband, when you add links they just combine together into a single bigger pipe, which can transfer more traffic in a single stream. Less need to jump up to higher tier adapters and switches with IB for that, you can just run more links between the hosts.

shrubble 3y ago
I re-worded what I wrote for clarity, it's an obscure sort of sentence construction. "I don't agree with Jim Keller in that absolute highest performance is found in Infiniband, not only with bandwidth but latency. " I don't agree with Jim Keller who says that Ethernet would be as good or better, in that when you use Infiniband, you get the absolute highest performance from IB, since IB does best on bandwidth and latency considerations.
- justinclift 3y ago
  No worries, my misunderstanding. :)
oasisaimlessly 3y ago
You can definitely bond multiple Ethernet links together ("IEEE 802.3ad link aggregation").
- justinclift 3y ago
  Sure, but ethernet doesn't let you combine the entire bandwidth of those links into a bigger fat pipe for a single stream. It more designed for higher bandwidth going to multiple destinations. A super common example of this is when consumers want to upgrade their existing home infrastructure to use a new NAS, and they start hitting the limits of 1GbE. Their first question is often along the lines of "can't I just add more links together to give more bandwidth?". Then people need to explain "No, ethernet doesn't work that way. You need to move up to 2.5/5/10GbE". (etc) Infiniband doesn't have that problem.
  - sheepdestroyer 3y ago
    It, not Ethernet that doesn't support the bandwidth aggregation you want, but the protocol over it. Multi Stream TCP seems to be one solution to this problem
  - saltcured 3y ago
    This used to be done, using what was called round-robin mode in bonded links. I think you are right that this mode never found its way into 802.11ad. Instead, parallel stacks of switches would be deployed, so that hosts were configured to bond links to N switches but the switches were not aware of the bonding. It was a commonly used Linux option to modestly scale up with low cost hardware. A single stream would have its frames striped over all the links in the bond, due to the configuration of the sending host. The flows would remain parallel and arrive over N switches to N NIC ports on the destination host. Care had to be taken with topology to keep parallel paths from end to end such as in one datacenter. Generally, flows could fan-in somewhere on the path and then go over a higher speed uplink, but would never fan-out again to parallel links. The drawback is more frame (packet) reordering, so it is even more sensitive to buffering and congestion control effects. If there were switch failures, the flows could become unbalanced if the topology offered redundant cross-links, since this would cause some fan-in on the surviving switches. Otherwise, completely independent switch stacks could be used, so a failure means traffic is dropped. Then, the host needs to do some kind of remote peer connectivity check to decide whether each link is healthy or not, and drop unhealthy links from the bond. Ironically, this could be done in consumer-grade LANs, since it does not depend on bonding features of the switches. However, careful setup is needed with consideration of the topology and switch behavior, to avoid pathology.