Training needs insane amounts of inter node bandwidth, to the extent that training clusters use specially built hardware for it. Decentralised training isn't ph

Training needs insane amounts of inter node bandwidth, to the extent that training clusters use specially built hardware for it. Decentralised training isn't physically possible anytime soon, maybe never.