3 ms·
Cranking up the batch size kills convergence.
by brrrrrm 3y ago
Cranking up the batch size kills convergence.
- FeepingCreature 3y agoWonder if that can be avoided by modifying the training approach. Ideas offhand: group by topic, train a subset of weights per node; figure out which layers have the most divergence and reduce lr on those only.
- brrrrrm 3y agoA provable way to recover convergence is to calculate the hessian. It’s computationally expensive but there are approximation methods.