Some observations: * for synchronous model-based distributed training to scale linearly, the time required to broadcast the model must be much larger than the

Some observations: * for synchronous model-based distributed training to scale linearly, the time required to broadcast the model must be much larger than the time required for a worker (GPU) to process a batch * it's not strict synchronous training, as when gradients are computed at a worker, they are transmitted to all workers - so the driver doesn't have to send models to all 32 workers at the same time (8 GPUs per worker makes 256 GPUs in total). * there are extremely large batch sizes (8196) * it's a good network (50 Gb Ethernet, albeit not infiniband) So, the relative amount of work done training at each worker is much higher than the time spent broadcasting the model (which is quite small (~100 MB, i think)) to the workers for each iteration. For larger models with smaller batch sizes, this relationship would break down. The interesting contribution here is that you can have massive batch sizes and Facebook provided a heuristic for adjusting the learning rate to converge with such massive batch sizes.