4 ms·

You can definitely bond multiple Ethernet links together ("IEEE 802.3ad link aggregation").

justinclift 2y ago
Sure, but ethernet doesn't let you combine the entire bandwidth of those links into a bigger fat pipe for a single stream. It more designed for higher bandwidth going to multiple destinations. A super common example of this is when consumers want to upgrade their existing home infrastructure to use a new NAS, and they start hitting the limits of 1GbE. Their first question is often along the lines of "can't I just add more links together to give more bandwidth?". Then people need to explain "No, ethernet doesn't work that way. You need to move up to 2.5/5/10GbE". (etc) Infiniband doesn't have that problem.
- sheepdestroyer 2y ago
  It, not Ethernet that doesn't support the bandwidth aggregation you want, but the protocol over it. Multi Stream TCP seems to be one solution to this problem
- saltcured 2y ago
  This used to be done, using what was called round-robin mode in bonded links. I think you are right that this mode never found its way into 802.11ad. Instead, parallel stacks of switches would be deployed, so that hosts were configured to bond links to N switches but the switches were not aware of the bonding. It was a commonly used Linux option to modestly scale up with low cost hardware. A single stream would have its frames striped over all the links in the bond, due to the configuration of the sending host. The flows would remain parallel and arrive over N switches to N NIC ports on the destination host. Care had to be taken with topology to keep parallel paths from end to end such as in one datacenter. Generally, flows could fan-in somewhere on the path and then go over a higher speed uplink, but would never fan-out again to parallel links. The drawback is more frame (packet) reordering, so it is even more sensitive to buffering and congestion control effects. If there were switch failures, the flows could become unbalanced if the topology offered redundant cross-links, since this would cause some fan-in on the surviving switches. Otherwise, completely independent switch stacks could be used, so a failure means traffic is dropped. Then, the host needs to do some kind of remote peer connectivity check to decide whether each link is healthy or not, and drop unhealthy links from the bond. Ironically, this could be done in consumer-grade LANs, since it does not depend on bonding features of the switches. However, careful setup is needed with consideration of the topology and switch behavior, to avoid pathology.

Sure, but ethernet doesn't let you combine the entire bandwidth of those links into a bigger fat pipe for a single stream. It more designed for higher bandwidth going to multiple destinations. A super common example of this is when consumers want to upgrade their existing home infrastructure to use a new NAS, and they start hitting the limits of 1GbE. Their first question is often along the lines of "can't I just add more links together to give more bandwidth?". Then people need to explain "No, ethernet doesn't work that way. You need to move up to 2.5/5/10GbE". (etc) Infiniband doesn't have that problem.

sheepdestroyer 2y ago
It, not Ethernet that doesn't support the bandwidth aggregation you want, but the protocol over it. Multi Stream TCP seems to be one solution to this problem
saltcured 2y ago
This used to be done, using what was called round-robin mode in bonded links. I think you are right that this mode never found its way into 802.11ad. Instead, parallel stacks of switches would be deployed, so that hosts were configured to bond links to N switches but the switches were not aware of the bonding. It was a commonly used Linux option to modestly scale up with low cost hardware. A single stream would have its frames striped over all the links in the bond, due to the configuration of the sending host. The flows would remain parallel and arrive over N switches to N NIC ports on the destination host. Care had to be taken with topology to keep parallel paths from end to end such as in one datacenter. Generally, flows could fan-in somewhere on the path and then go over a higher speed uplink, but would never fan-out again to parallel links. The drawback is more frame (packet) reordering, so it is even more sensitive to buffering and congestion control effects. If there were switch failures, the flows could become unbalanced if the topology offered redundant cross-links, since this would cause some fan-in on the surviving switches. Otherwise, completely independent switch stacks could be used, so a failure means traffic is dropped. Then, the host needs to do some kind of remote peer connectivity check to decide whether each link is healthy or not, and drop unhealthy links from the bond. Ironically, this could be done in consumer-grade LANs, since it does not depend on bonding features of the switches. However, careful setup is needed with consideration of the topology and switch behavior, to avoid pathology.