There's still a lot of data that needs to be passed between GPUs. SageMaker is "minimizing the communication," but it's still way more than nothing, and all the

There's still a lot of data that needs to be passed between GPUs. SageMaker is "minimizing the communication," but it's still way more than nothing, and all the gradients need to be communicated roughly every iteration. That's ok to send between computers with high speed datacenter links, but much more than you would ever want to send across the internet repeatedly.