Both Intel (Gaudi 3) and Tenstorrent (Wormhole) have built NIC's into the GPUs. I think that we are going to see more of that in the future, but the only proble

Both Intel (Gaudi 3) and Tenstorrent (Wormhole) have built NIC's into the GPUs. I think that we are going to see more of that in the future, but the only problem with this is that your failure cases become more extreme. NIC dies, I swap a NIC... GPUNIC dies... I swap both. 128 NICs/GPUs is the largest single non-blocking 400G switch size today using a broadcom t5 chip [0]. If you want to go larger than that, you have to go with 6x switches in a spine/leaf configuration, which greatly increases the costs. We're (Hot Aisle) in the process of deploying a cluster of 128 MI300x right now with this configuration using a Dell Z9864F switch [1]. https://hotaisle.xyz/networking/ https://hotaisle.xyz/networking/ and https://hotaisle.xyz/compute/ https://hotaisle.xyz/compute/ [0] https://www.broadcom.com/products/ethernet-connectivity/switching/strataxgs/bcm78900-series https://www.broadcom.com/products/ethernet-connectivity/swit... [1] https://www.delltechnologies.com/asset/en-us/products/networking/technical-support/dell-powerswitch-z9864f-on-spec-sheet.pdf https://www.delltechnologies.com/asset/en-us/products/networ...