4 ms·

Logistic regression without regularization blows up if your data is linearly separable. This is an issue you don't really encounter with linear regression. EDI

Logistic regression without regularization blows up if your data is linearly separable. This is an issue you don't really encounter with linear regression. EDIT: Apparently this is getting downvoted. To explain a bit more clearly: if you know nothing about the data, then your default should be to run with regularization, since you won't get a useful result if the data turns out to be linearly separable. In the case of linear regression, it's often sensible to start with non-regularized fitting, since that won't mess up your coefficients. But logistic regression is quite simply a different beast altogether, and this is covered in any intro on logistic regression you care to consult.

gbrown 7y ago
I think you're taking a perspective which is too focused on ML, ignoring the widespread statistical use of logistic regression in lower dimensional settings with reasonable a-priori covariate structures. > Logistic regression without regularization blows up if your data is linearly separable. This is an issue you don't really encounter with linear regression. Finding that your data is linearly separable is usually quite important - I do see students mess this up though, to be fair. If you want frequentist inference, non-penalized logistic regression is the way to go. (disclaimer, I'm predominantly Bayesian, so with proper priors the point is actually moot for me.)
- missosoup 7y ago
  > I think you're taking a perspective which is too focused on ML scikit-learn is an ML library. It's explicitly not a stats library. The assumptions and defaults are correct for ML use cases. GP comment is entirely correct. If you want to do widespread statistics, use StatsModels which is designed for that.
  - gbrown 7y ago
    The post I was replying to was attempting to make a much more general point - I understand the default as applied to Scikit Learn.
  - 6gvONxR4sf7o 7y ago
    >scikit-learn is an ML library. It's explicitly not a stats library. Yeah but logistic regression is from stats. If you're going to implement ClassicalThingFromNearbyField in an ML library, people are right to complain when the definition doesn't match. Not surprising anyone, there's a lot of overlap between stats and ML. If I implemented a "SimpleFraction" model in sklearn, it would be useful to use "[# this class] / ([# all classes] + epsilon)" using epsilon to regularize, but that would be a crazy name. Or at least it would be crazy if I used a default other than epsilon=0.
    - zwaps 7y ago
      I am a stats guy, and sklearn did confuse the hell out of me when I tried to use it as such, but I have to agree with others here. The issue is not how regression works, the issue is that sklearn does not do stats at all! The use case you think about is not what sklearn does. You cant even use it, because it returns no model metrics you need for a specified model structure. No one uses sklearn for stats, and logistic regression here is really not stats, it is a not a ML estimated conditional expectation, it is simply a consequence of a ML loss function. Ane in that sense, regularization is useful. People expecting to get a stats model out of sklearn will fail immediately, since it does not even return errors, covariances or statistics.
mattkrause 7y ago
If you know nothing about your data, you shouldn’t be fitting either kind of model; you should be learning about it instead.
deleted 7y ago
[deleted]

gbrown 7y ago

I think you're taking a perspective which is too focused on ML, ignoring the widespread statistical use of logistic regression in lower dimensional settings with reasonable a-priori covariate structures. > Logistic regression without regularization blows up if your data is linearly separable. This is an issue you don't really encounter with linear regression. Finding that your data is linearly separable is usually quite important - I do see students mess this up though, to be fair. If you want frequentist inference, non-penalized logistic regression is the way to go. (disclaimer, I'm predominantly Bayesian, so with proper priors the point is actually moot for me.)

missosoup 7y ago
> I think you're taking a perspective which is too focused on ML scikit-learn is an ML library. It's explicitly not a stats library. The assumptions and defaults are correct for ML use cases. GP comment is entirely correct. If you want to do widespread statistics, use StatsModels which is designed for that.
- gbrown 7y ago
  The post I was replying to was attempting to make a much more general point - I understand the default as applied to Scikit Learn.
- 6gvONxR4sf7o 7y ago
  >scikit-learn is an ML library. It's explicitly not a stats library. Yeah but logistic regression is from stats. If you're going to implement ClassicalThingFromNearbyField in an ML library, people are right to complain when the definition doesn't match. Not surprising anyone, there's a lot of overlap between stats and ML. If I implemented a "SimpleFraction" model in sklearn, it would be useful to use "[# this class] / ([# all classes] + epsilon)" using epsilon to regularize, but that would be a crazy name. Or at least it would be crazy if I used a default other than epsilon=0.
  - zwaps 7y ago
    I am a stats guy, and sklearn did confuse the hell out of me when I tried to use it as such, but I have to agree with others here. The issue is not how regression works, the issue is that sklearn does not do stats at all! The use case you think about is not what sklearn does. You cant even use it, because it returns no model metrics you need for a specified model structure. No one uses sklearn for stats, and logistic regression here is really not stats, it is a not a ML estimated conditional expectation, it is simply a consequence of a ML loss function. Ane in that sense, regularization is useful. People expecting to get a stats model out of sklearn will fail immediately, since it does not even return errors, covariances or statistics.

mattkrause 7y ago

If you know nothing about your data, you shouldn’t be fitting either kind of model; you should be learning about it instead.

deleted 7y ago

[deleted]