I think as a rule of thumb, latest generation hardware makes the most sense for cloud compute providers supporting AI. But while yes, training can be bottleneck

I think as a rule of thumb, latest generation hardware makes the most sense for cloud compute providers supporting AI. But while yes, training can be bottlenecked easily by internode training speed, it really depends on the model, model size, how you’re doing the training (DDP? FSDP? Custom sharding?). I’ve seen bottlenecks there, but usually it’s on the latency side of things, and you won’t really see an improvement there moving from Infiniband HDR to NDR. Preparing hardware for a generic cluster with changing or unknown workloads, yeah max it out and build the most flexible fabric you can. But if you know your model, you can optimize your hardware to it.