4 ms·
Gradient descent finds the minimum L2 norm solution of the least squares? So adding a L2 regularization term should do nothing if the objective is convex, right
by esafak 4y ago
Gradient descent finds the minimum L2 norm solution of the least squares? So adding a L2 regularization term should do nothing if the objective is convex, right? Is this common knowledge? I must have missed the memo, or be getting rusty.
- canjobear 4y agoThe formal claim is in Appendix A here: https://arxiv.org/pdf/2303.14151.pdf https://arxiv.org/pdf/2303.14151.pdf
- svantana 4y agoIt finds the minimum with regards to the training data. The double descent phenomenon is about error on unseen data.
- jwilber 4y agoThere’s definitely more to the picture than that - it’s really about how models in the interpolation region behave (see https://mlu-explain.github.io/double-descent/ https://mlu-explain.github.io/double-descent/ or https://mlu-explain.github.io/double-descent2/ https://mlu-explain.github.io/double-descent2/)
- deleted 4y ago[deleted]
- macleginn 4y agoIt depends on the inputs. E.g., if two elements of the input are highly correlated, the network can assign an arbitrarily high positive weight to one element and then correct this by assigning an arbitrarily high negative weight to the other, with the total contribution of the two elements being reasonable. However, if these input elements are not as tightly correlated in the test data, the results may be bad. L2 regularisation ensures that this does not happen. (In practice, the effect will be mitigated by the activation function, but weights do blow up in training, leading to "saturated" models, which exhibit weird behaviours.)
- elcomet 4y agoThe objective is usually not convex in deep learning.
- elcomet 4y agoYou don't necessarily want the optimal solution, L2 regularization can give you a slightly worse solution on the training set but that will be better at generalization on unseen data.