3 ms·

Group Lasso Regularization

51 points by keyboardman 6y ago

6gvONxR4sf7o 6y ago
Group Lasso is one of those things I'm really surprised not to see used more often. Great post.
- 0xddd 6y ago
  Why is that? The article doesn't say much about the benefits of using this type of regularization.
  - 6gvONxR4sf7o 6y ago
    Say you want to represent a categorical feature that has many levels. You're likely to one-hot encode it. This can lead to a lack of parity between how sparsely you treat your continuous features versus your categorical ones, where continuous features can't be partially included, but one-hot-encoded categoricals can. Sometimes you want to be sparse on your original variables, rather than sparse on dimensions of their encoding. It's a good inductive bias that if, for example, profession is relevant to your model, then all professions are relevant (and you still get to ridge penalize the individual professions). Nice for interpretability too.
abhgh 6y ago
Not very commonly discussed, but great topic to be aware of. One particularly powerful use of this is in learning models with interaction terms: more powerful than a linear model but still interpretable, [1]. [1] R package https://cran.r-project.org/web/packages/glinternet/index.html https://cran.r-project.org/web/packages/glinternet/index.htm... (paper linked therein)
nerdponx 6y ago
Nice post. There's also at least one CrossValidated question on this topic, with good answers attached. For example https://stats.stackexchange.com/q/214325/36229 https://stats.stackexchange.com/q/214325/36229
deleted 6y ago
[deleted]
currymj 6y ago
In Bayesian terms, ridge regression is equivalent to putting a Gaussian prior on the weights, which gets wider the weaker the penalty term. The lasso is the same, but with a Laplace distribution as the prior. The "elastic net", which combines the l1 and l2 penalties, even has a Bayesian interpretation, with a fairly weird prior [1, 2]. Anyone know if there's an equivalent Bayesian interpretation of group lasso? Maybe it's just a Gaussian prior but with block-wise correlation between the variables? [1]: https://stats.stackexchange.com/questions/283238/is-there-a-bayesian-interpretation-of-linear-regression-with-simultaneous-l1-and https://stats.stackexchange.com/questions/283238/is-there-a-... [2]: https://www.tandfonline.com/doi/abs/10.1198/jasa.2011.tm09241 https://www.tandfonline.com/doi/abs/10.1198/jasa.2011.tm0924...
- laGrenouille 6y ago
  From a Bayesian perspective, it’s not entirely accurate to say that there is the same relationship between ridge/Gaussian-prior and lasso/Laplacian-prior. In the first case, ridge regression yields a "proper" Bayesian estimator (e.g., mean of the posterior); the latter is only the MAP estimator, which is not really recommended from a purely Bayesian perspective. Ridge is also a MAP estimator, but that’s in addition to the stronger condition of minimizing the Bayes risk. There is a way to compute a "true" Bayesian Lasso, but it doesn’t yield a sparse model [0]. [0] http://archived.stat.ufl.edu/casella/Papers/Lasso.pdf http://archived.stat.ufl.edu/casella/Papers/Lasso.pdf
- unemphysbro 6y ago
  It's similar to lasso where the Laplace distribution is decomposed using a gaussian mixture scaled with an exponential/gamma distribution. In group lasso, there is one variance term per group which is generated by the exponential/gamma prior. See https://www.cs.ubc.ca/~murphyk/MLbook/ https://www.cs.ubc.ca/~murphyk/MLbook/ for the mathematical decomp. It should be noted that obtained using that prior in a MAP estimation per the other comment.
deleted 6y ago
[deleted]

51 points by keyboardman 6y ago

6gvONxR4sf7o 6y ago

Group Lasso is one of those things I'm really surprised not to see used more often. Great post.

0xddd 6y ago
Why is that? The article doesn't say much about the benefits of using this type of regularization.
- 6gvONxR4sf7o 6y ago
  Say you want to represent a categorical feature that has many levels. You're likely to one-hot encode it. This can lead to a lack of parity between how sparsely you treat your continuous features versus your categorical ones, where continuous features can't be partially included, but one-hot-encoded categoricals can. Sometimes you want to be sparse on your original variables, rather than sparse on dimensions of their encoding. It's a good inductive bias that if, for example, profession is relevant to your model, then all professions are relevant (and you still get to ridge penalize the individual professions). Nice for interpretability too.

abhgh 6y ago

Not very commonly discussed, but great topic to be aware of. One particularly powerful use of this is in learning models with interaction terms: more powerful than a linear model but still interpretable, [1]. [1] R package https://cran.r-project.org/web/packages/glinternet/index.html https://cran.r-project.org/web/packages/glinternet/index.htm... (paper linked therein)

nerdponx 6y ago

Nice post. There's also at least one CrossValidated question on this topic, with good answers attached. For example https://stats.stackexchange.com/q/214325/36229 https://stats.stackexchange.com/q/214325/36229

deleted 6y ago

[deleted]

currymj 6y ago

In Bayesian terms, ridge regression is equivalent to putting a Gaussian prior on the weights, which gets wider the weaker the penalty term. The lasso is the same, but with a Laplace distribution as the prior. The "elastic net", which combines the l1 and l2 penalties, even has a Bayesian interpretation, with a fairly weird prior [1, 2]. Anyone know if there's an equivalent Bayesian interpretation of group lasso? Maybe it's just a Gaussian prior but with block-wise correlation between the variables? [1]: https://stats.stackexchange.com/questions/283238/is-there-a-bayesian-interpretation-of-linear-regression-with-simultaneous-l1-and https://stats.stackexchange.com/questions/283238/is-there-a-... [2]: https://www.tandfonline.com/doi/abs/10.1198/jasa.2011.tm09241 https://www.tandfonline.com/doi/abs/10.1198/jasa.2011.tm0924...

laGrenouille 6y ago
From a Bayesian perspective, it’s not entirely accurate to say that there is the same relationship between ridge/Gaussian-prior and lasso/Laplacian-prior. In the first case, ridge regression yields a "proper" Bayesian estimator (e.g., mean of the posterior); the latter is only the MAP estimator, which is not really recommended from a purely Bayesian perspective. Ridge is also a MAP estimator, but that’s in addition to the stronger condition of minimizing the Bayes risk. There is a way to compute a "true" Bayesian Lasso, but it doesn’t yield a sparse model [0]. [0] http://archived.stat.ufl.edu/casella/Papers/Lasso.pdf http://archived.stat.ufl.edu/casella/Papers/Lasso.pdf
unemphysbro 6y ago
It's similar to lasso where the Laplace distribution is decomposed using a gaussian mixture scaled with an exponential/gamma distribution. In group lasso, there is one variance term per group which is generated by the exponential/gamma prior. See https://www.cs.ubc.ca/~murphyk/MLbook/ https://www.cs.ubc.ca/~murphyk/MLbook/ for the mathematical decomp. It should be noted that obtained using that prior in a MAP estimation per the other comment.

deleted 6y ago

[deleted]