Thats what makes ML training challenging, the nodes are constantly all-reducing the weights to each other and the step time is in milliseconds. Thats why you s

Thats what makes ML training challenging, the nodes are constantly all-reducing the weights to each other and the step time is in milliseconds. Thats why you see cloud vendors coming out with really beefy individual hosts with lots of GPU's strapped to them, as datacenter networking sucks for these applications.