3 ms·
The weights arent averaged unless the training step is synchronous. Even then most of the times it is the gradients that are added up rather than the actual wei
by pavanky 7y ago
The weights arent averaged unless the training step is synchronous. Even then most of the times it is the gradients that are added up rather than the actual weights.
For inference, I dont think there are many papers that claim direct average of weights perform better than any single model. It is usually the output that is accumulated in some way.
- sillysaurusx 7y agohttps://arxiv.org/abs/1803.05407 https://arxiv.org/abs/1803.05407 "Averaging Weights Leads to Wider Optima and Better Generalization" Weirdly, the averaging doesn't have to be synchronous.