6 ms·
What router/switch one would use for such speed?
by riku_iki 3y ago
What router/switch one would use for such speed?
- KeplerBoy 3y ago800Gbps via OSFP and QSFP-DD are already a thing. Multiple vendors have NICs and switches for that.
- _zoltan_ 3y agocan you show me a 800G NIC? the switch is fine, I'm buying 64x800G switches, but NIC wise I'm limited to 400Gbit.
- KeplerBoy 3y agofair enough, it seems I was mistaken about the NIC. I guess that has to wait for PCIe 6 and should arrive soon-ish.
- CyberDildonics 3y ago16x PCIe 4.0 is 32GB/s 16x PCIe 5.0 should be 64 GB/s, how is any computer using 100 GB/s ?
- KeplerBoy 3y agoI was talking about Gigabit/s, not Gigabyte/s. The article however actually talks about Terabyte/s scale, albeit not over a single node.
- CyberDildonics 3y ago800 gigabits is 100 gigabytes which is still more than PCIe 5.0 16x 64 gigabyte per second bandwidth. You said there were 800 gigabit network cards, I'm wondering how that much bandwidth makes it to the card in the first place. The article however actually talks about Terabyte/s scale, albeit not over a single node. This does not have anything to do with what you originally said, you were talking about 800gb single ports.
- KeplerBoy 3y agoYes, apparently I was mistaken about the NICs. They don't seem to be available yet. But it's not a PCIe limitation. There are PCIe devices out there which use 32 lanes, so you could achieve the bandwidth even on PCIe5. https://www.servethehome.com/ocp-nic-3-0-form-factors-quick-guide-intel-broadcom-nvidia-meta-inspur-dell-emc-hpe-lenovo-gigabyte-supermicro/ https://www.servethehome.com/ocp-nic-3-0-form-factors-quick-...
- NavinF 3y agoI'm not aware of any 800G cards, but FYI a single Mellanox card can use two PCIe x16 slots to avoid NUMA issues on dual-socket servers: https://www.nvidia.com/en-us/networking/ethernet/socket-direct/ https://www.nvidia.com/en-us/networking/ethernet/socket-dire... So the software infra for using multiple slots already exists and doesn't require any special config. Oh and some cards can use PCIe slots across multiple hosts. No idea why you'd want to do that, but you can.
- epistasis 3y agoGiven their configuration of just 4U spread across 17 racks, there's likely a bunch of compute in the rest of the rack, and 1-2 top of rack switches like this: https://www.qct.io/product/index/Switch/Ethernet-Switch/T7000-Series/QuantaMesh-T7032-IX7D https://www.qct.io/product/index/Switch/Ethernet-Switch/T700... And then you connect the TOR switches to higher level switches in something like a Clos distribution to get the desired bandwidth between any two nodes: https://www.techtarget.com/searchnetworking/definition/Clos-network https://www.techtarget.com/searchnetworking/definition/Clos-...
- NavinF 3y agoLinked article says they used 68 machines with 2 x 100GbE Mellanox ConnectX-6 cards. So any 100G pizza box switches should work. Note that 36 port 56G switches are dirt cheap on eBay and 4tbps is good enough for most homelab use cases
- riku_iki 3y ago> So any 100G pizza box switches should work. but will it be able to handle combined TB/s traffic?
- baq 3y agoany switch which can't handle full load on all ports isn't worthy of the name 'switch', it's more like 'toy network appliance'
- birdman3131 3y agoI will forever be scarred by the "Gigabit" switches of old that were 2 gigabit ports and 22 100mb ports. Coworker bought it missing the nuance.
- bombcar 3y agoStill happens, gotta see if the top speed mentioned is an uplink or normal ports.
- aaronax 3y agoYes. Most network switches can handle all ports at 100% utilization in both directions simultaneously. Take for example the Mellanox SX6790 available for less than $100 on eBay. It has 36 56gbps ports. 36 * 2 * 56 = 4032gbps and it is stated to have a switching capacity of 4.032Tbps. Edit: I guess you are asking how one would possibly sip 1TiB/s of data into a given client. You would need multiple clients spread across several switches to generate such load. Or maybe some freaky link aggregation. 10x 800gbps links for your client, plus at least 10x 800gbps links out to the servers.