You are averaging weights in distributed training? That seems like it would be rife with pitfalls unless you average after every batch. I always thought the pr

You are averaging weights in distributed training? That seems like it would be rife with pitfalls unless you average after every batch. I always thought the preferred method was to average the gradient updates, and pass that to update the single mother-model.