Unfortunately, I don’t think distributed training is feasible. My understanding is that these models are trained using some form of gradient descent. This requ

Unfortunately, I don’t think distributed training is feasible. My understanding is that these models are trained using some form of gradient descent. This requires knowledge of the model’s prior state and current training error to adjust parameter values. With distributed training it’s difficult, to point of impossible, to communicate the state of the model across large distances because these models are so large. Take my comment with a grain of salt because I’m not an expert in this area. I’d very much enjoy being wrong about this because distributed training would unlock so much potential.