Ideally those many GPUs could be on different hosts connected with a commodity interconnect like 10gbe. If MOE models do well it could be great for commodity h

Ideally those many GPUs could be on different hosts connected with a commodity interconnect like 10gbe. If MOE models do well it could be great for commodity hw based distributed inference approaches.