In the case of llama.cpp's RPC mode, the network isn't the limiting factor for inference, but for distributing layers to nodes. I was monitoring the network wh

In the case of llama.cpp's RPC mode, the network isn't the limiting factor for inference, but for distributing layers to nodes. I was monitoring the network while running various models, and for all models, the first step was to copy over layers (a few gigabytes to 100 or so GB for the huge models), and that would max out the 5 Gbps connection. But then while warming up and processing, there were only 5-10 Mbps of traffic, so you could do it over a string and tin cans, almost. But that's a limitation of the current RPC architecture, it can't really parallelize processing, so as I noted in the post and in my video, it kinda uses resources round-robin style, and you can only get worse performance across the entire cluster than on a single node for any model you can fit on the single node.