4 ms·
Yes, it does. You might have been thrown off by the fact that L1 regularization is not L0 regularization, i.e. it doesn't explicitly limit the number of nonzero
by makeset 10y ago
Yes, it does. You might have been thrown off by the fact that L1 regularization is not L0 regularization, i.e. it doesn't explicitly limit the number of nonzero coefficients. Still, the linearity of L1 constraint boundaries creates spikes in directions with zero components, thus forcing constrained solutions to occur where many variables are driven to exactly zero. See here:
https://en.wikipedia.org/wiki/Lasso_(statistics)#Geometric_interpretation https://en.wikipedia.org/wiki/Lasso_(statistics)#Geometric_i...
- ekelsen 10y agoIf we have a parameter x, and some cost function J(x), then with L1 regularization the cost function would be J(x) + beta * abs(x). The derivative of that loss with respect to x would be J'(x) + beta * sgn(x). So using some variant of SGD (which is what basically all neural network training does these days) we would essentially update x as: x = x - alpha * (J'(x) + beta). (The specifics depend on the algorithm, but it doesn't change the result). So for x to end up as _exactly_ 0, we have to be extremely lucky, which in practice I have never observed. Using L1 regularization definitely leads to small weights, but not to ones that are _exactly_ 0.
- argonaut 10y agoYour conclusion is theoretically false. You can prove that L1 regularization is equivalent to taking the optimal unregularized parameters, setting to parameters below a threshold to 0 (the threshold depends on the regularization parameter), and penalizing the other parameters.
- ekelsen 10y agoYes, but how do you actually optimize that loss in practice? I'm not saying that a perfect solution with an L1 penalty wouldn't have weights exactly equal to 0. I'm saying that with the optimization techniques that are commonly used, you don't end up with exact zeros.
- argonaut 10y agoYou're not making sense. If the loss function is convex, adding L1 regularization is still convex. So iterative methods for convex problems (which include SVMs, linear regression, and logistic regression) will find the global optimum.
- ekelsen 10y ago1) Neural Network Loss functions are not convex. But that isn't the issue here. 2) When you use actual numerical optimization techniques with floating point arithmetic, you don't find an exact minimum (global or local). And you don't get exact zeros. Have you tried this on a real problem? I wouldn't consider MNIST a real problem, but even there you will not get _exact_ zeros. Try it.
- makeset 10y agoIf you rolled your own naive numerical approximation of L1 regularization, you might not have gotten exact zeros. If you use e.g. LARS or cyclical coordinate descent for the L1-regularized parameter cohort, as suited to the problem, you will get exact zeros, as prescribed by the mathematics of L1.
- ekelsen 10y agoI've never seen anyone optimize a neural network using LARS or cyclical coordinate descent. I thought that's what this entire discussion was about - not arbitrary optimization theory.
- deleted 10y ago[deleted]