4 ms·

In Bayesian terms, ridge regression is equivalent to putting a Gaussian prior on the weights, which gets wider the weaker the penalty term. The lasso is the sa

In Bayesian terms, ridge regression is equivalent to putting a Gaussian prior on the weights, which gets wider the weaker the penalty term. The lasso is the same, but with a Laplace distribution as the prior. The "elastic net", which combines the l1 and l2 penalties, even has a Bayesian interpretation, with a fairly weird prior [1, 2]. Anyone know if there's an equivalent Bayesian interpretation of group lasso? Maybe it's just a Gaussian prior but with block-wise correlation between the variables? [1]: https://stats.stackexchange.com/questions/283238/is-there-a-bayesian-interpretation-of-linear-regression-with-simultaneous-l1-and https://stats.stackexchange.com/questions/283238/is-there-a-... [2]: https://www.tandfonline.com/doi/abs/10.1198/jasa.2011.tm09241 https://www.tandfonline.com/doi/abs/10.1198/jasa.2011.tm0924...

laGrenouille 6y ago
From a Bayesian perspective, it’s not entirely accurate to say that there is the same relationship between ridge/Gaussian-prior and lasso/Laplacian-prior. In the first case, ridge regression yields a "proper" Bayesian estimator (e.g., mean of the posterior); the latter is only the MAP estimator, which is not really recommended from a purely Bayesian perspective. Ridge is also a MAP estimator, but that’s in addition to the stronger condition of minimizing the Bayes risk. There is a way to compute a "true" Bayesian Lasso, but it doesn’t yield a sparse model [0]. [0] http://archived.stat.ufl.edu/casella/Papers/Lasso.pdf http://archived.stat.ufl.edu/casella/Papers/Lasso.pdf
unemphysbro 6y ago
It's similar to lasso where the Laplace distribution is decomposed using a gaussian mixture scaled with an exponential/gamma distribution. In group lasso, there is one variance term per group which is generated by the exponential/gamma prior. See https://www.cs.ubc.ca/~murphyk/MLbook/ https://www.cs.ubc.ca/~murphyk/MLbook/ for the mathematical decomp. It should be noted that obtained using that prior in a MAP estimation per the other comment.

laGrenouille 6y ago

From a Bayesian perspective, it’s not entirely accurate to say that there is the same relationship between ridge/Gaussian-prior and lasso/Laplacian-prior. In the first case, ridge regression yields a "proper" Bayesian estimator (e.g., mean of the posterior); the latter is only the MAP estimator, which is not really recommended from a purely Bayesian perspective. Ridge is also a MAP estimator, but that’s in addition to the stronger condition of minimizing the Bayes risk. There is a way to compute a "true" Bayesian Lasso, but it doesn’t yield a sparse model [0]. [0] http://archived.stat.ufl.edu/casella/Papers/Lasso.pdf http://archived.stat.ufl.edu/casella/Papers/Lasso.pdf

unemphysbro 6y ago

It's similar to lasso where the Laplace distribution is decomposed using a gaussian mixture scaled with an exponential/gamma distribution. In group lasso, there is one variance term per group which is generated by the exponential/gamma prior. See https://www.cs.ubc.ca/~murphyk/MLbook/ https://www.cs.ubc.ca/~murphyk/MLbook/ for the mathematical decomp. It should be noted that obtained using that prior in a MAP estimation per the other comment.