The paper: https://arxiv.org/pdf/1602.05629.pdf https://arxiv.org/pdf/1602.05629.pdf The key algorithmic detail: it seems they have each device perform multipl

The paper: https://arxiv.org/pdf/1602.05629.pdf https://arxiv.org/pdf/1602.05629.pdf The key algorithmic detail: it seems they have each device perform multiple batch updates to the model, and then average all the multi-batch updates. "That is, each client locally takes one step of gradient descent on the current model using its local data, and the server then takes a weighted average of the resulting models. Once the algorithm is written this way, we can add more computation to each client by iterating the local update. " They do some sensible things with model initialization to make sure weight update averaging works, and show in practice this way of doing things requires less communication and gets to the goal faster than a more naive approach. It seems like a fairly straighforward idea from the baseline SGD, so the contribution is mostly in actually doing it.