4 ms·
Regularisation should not be done with the optimiser but with the loss function and the architecture.
by WithinReason 2y ago
Regularisation should not be done with the optimiser but with the loss function and the architecture.
- redox99 2y agoRegularization only helps you so much.
- staunton 2y agoMaybe it should not be done but the large neutral networks this decade absolutely rely on this. A network at the global minimum of any of the (regularized) loss functions that are used these days would be waaay overfitted.
- uoaei 2y agoThe entire reason SGD works is because the stochastic nature of updates on minibatches is an implicit regularizer. This one perspective built the foundations for all of modern machine learning. I completely agree that the most effective regularization is inductive bias in the architecture. But bang for buck, given all the memory/compute savings it accomplishes, SGD is the exemplar of implicit regularization techniques.