3 ms·
Do you have any sources that expands on how that LASSO optimization problem was graphed? My thoughts when I looked at it: Where did beta come from? Whats the b
by yelnatz 10y ago
Do you have any sources that expands on how that LASSO optimization problem was graphed?
My thoughts when I looked at it: Where did beta come from? Whats the beta hat supposed to represent? Where'd the contours come from? How come they stopped there? Oh they're supposed to represent the weights? How do they represent the weights? Why is LASSO diamond, the other circular?
I remember seeing that graph a while ago, didn't understand it then, still don't understand it now.
- achompas 10y agoThanks for your comment. I note several references throughout the article, and they do a better job of explaining than I can in an HN comment. You should check them out! I'll try to explain briefly though. Fitting a model is an optimization problem: for a given loss function and a training set, find the parameter vector W that minimizes this loss function over the training set. Once that's done, you can estimate labels/values for unseen data using this model. Regularization is a constrained optimization problem: we still seek to choose the W that minimizes a loss function over a training set, but we have now limited the values of W we can choose from. Ok, great. What the heck is up with that plot? It's a visualization of the space of all W (we're assuming W is 2-dimensional here), and we're searching it for the optimal W according to the training set. The red contours represent the loss function, \hat{\beta} -- our estimate of W -- represents a minimum for that loss function, and the blue region represents values of W allowed by regularization. We basically want to find the point where the lowest contour lies tangent to the blue region -- that'll be our estimate of W. Answering your questions directly: * The blue region is centered at zero because we are penalizing the loss function for higher lengths of W. * LASSO maps to the diamond-shaped region because it aggressively pushes elements of W to zero (visually, keep drawing red contours and note that the contours are more likely to hit a corner of the diamond). * L2 maps to the unit circle because it penalizes the loss function using the total length of W; this is represented visually as a circle centered at zero.
- marketforlemmas 10y agoTo be a little more clear about the diamond and circle... LASSO is a diamond because it represents the constraint that w_1 + w_2 <= 1. The region of (w_1,w_2) that satisfy that inequality is a square. Ridge is a circle because it represents the constraint that w_1^2 + w_2^2 <= 1. The region of (w_1,w_2) that satisfy that inequality is a circle.