5 ms·

If you're not using regularisation for logistic regression, you're most likely doing it wrong. If you don't know to normalise your data prior to that, then you

If you're not using regularisation for logistic regression, you're most likely doing it wrong. If you don't know to normalise your data prior to that, then you need to go back to the basics. The default makes sense. It's also clearly stated as the very first parameter in the constructor which defaults to L2. The docs also state in BOLD: "Note that regularization is applied by default." This post is just a case of pebcac.

Macuyiko 7y ago
I agree. The only thing I believe the article gets right however is that the naming doesn't expose this default behavior, which might cause problems for people coming from other software. Still, any good data scientist I know actually knows to read the documentation carefully and has learnt regularisation is the default.
- missosoup 7y ago
  The call signature does expose it though. Am I a minority in terms of using jupyter notebook and an IDE that shows call signatures and docstrings?
- paulgb 7y ago
  One thing that concerns me about the "you should read the documentation" argument is that even if the person writing the code did that for every function they used, anyone reading the code is likely to make (reasonable) assumptions. So bugs like accidentally regularizing, for example, could slip past code review.
  - deleted 7y ago
    [deleted]
  - ad404b8a372f2b9 7y ago
    It's a fair point for standard software engineering libraries, that the defaults should be obvious, but I don't think that it's possible to hold scientific libraries to the same standards. They require expert knowledge, and the assumptions you make about the models that are being used should be very carefully checked. There is no obvious defaults in that case because we're not writing software, we're building scientific models. For example, look at the neural network classifier model in scikit-learn: https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html https://scikit-learn.org/stable/modules/generated/sklearn.ne... None of these defaults can be assumed. If you showed me a function call with no optional arguments I couldn't tell you, nor anybody else that hasn't read the documentation, what activation is being used, what optimizer, learning rate, etc... In fact when coming across these kind of models during code reviews the nature of these parameters is the first thing I ask about.
    - rfeather 7y ago
      Agree. It's naive to assume your use case is the basic one when using these libraries or that you know the underlying implementation and all of the parameters, since in ML the implementations vary enough to affect the outcome and will have different controls. To the point of exploratory analysis in some of the parents ,I prefer statsmodels for that purpose. It's not quite up to where the similarly purposed tools are in other languages, but for most of my work where I care about interpretation, it hits the right spot between usability and providing the standard statistical outputs.
    - zenexer 7y ago
      Then there shouldn’t be any defaults, right? All of these issues could easily be avoided by forcing users to provide explicit parameters. It only makes sense to offer defaults if they’re intuitive.
darawk 7y ago
> If you're not using regularisation for logistic regression, you're most likely doing it wrong Why's that? Logistic regressions are perfectly reasonable to run for exploratory purposes. There are tons of situations where regularization is not what you want. > If you don't know to normalise your data prior to that It's not a question of not knowing. In an unregularized regression, you don't need to standardize your data beforehand. People who knows this but don't realize regularization is being applied by default can easily make this mistake.
- funklute 7y ago
  Logistic regression without regularization blows up if your data is linearly separable. This is an issue you don't really encounter with linear regression. EDIT: Apparently this is getting downvoted. To explain a bit more clearly: if you know nothing about the data, then your default should be to run with regularization, since you won't get a useful result if the data turns out to be linearly separable. In the case of linear regression, it's often sensible to start with non-regularized fitting, since that won't mess up your coefficients. But logistic regression is quite simply a different beast altogether, and this is covered in any intro on logistic regression you care to consult.
  - gbrown 7y ago
    I think you're taking a perspective which is too focused on ML, ignoring the widespread statistical use of logistic regression in lower dimensional settings with reasonable a-priori covariate structures. > Logistic regression without regularization blows up if your data is linearly separable. This is an issue you don't really encounter with linear regression. Finding that your data is linearly separable is usually quite important - I do see students mess this up though, to be fair. If you want frequentist inference, non-penalized logistic regression is the way to go. (disclaimer, I'm predominantly Bayesian, so with proper priors the point is actually moot for me.)
    - missosoup 7y ago
      > I think you're taking a perspective which is too focused on ML scikit-learn is an ML library. It's explicitly not a stats library. The assumptions and defaults are correct for ML use cases. GP comment is entirely correct. If you want to do widespread statistics, use StatsModels which is designed for that.
CretinDesAlpes 7y ago
Completely agree. If you can't read documentation's software, you shouldn't do data science/ML in any cases.
alexfromapex 7y ago
Came here to say this. I can think of a lot of use cases where the feature data numbers will be dramatically different scales and the regularization helps to prevent that “noise” from dramatically affecting your model.
6gvONxR4sf7o 7y ago
Are you seriously arguing that R's implementation and statsmodels's implementations are doing it wrong? What if I had a model called "OrdinaryLeastSquares" in sklearn and had it regularize by default. After all, it's sklearn, so people probably wanted a penalized least squares model. That would be bad, because it's not what that means in most other libraries and textbooks where users have encountered the term.
bart_spoon 7y ago
The docs only state that because of this pushback. If I remember correctly, that is a relatively recent addition (within the last year I believe) that was added due to the communities dislike for both the defaults and the previous documentation.

Macuyiko 7y ago

I agree. The only thing I believe the article gets right however is that the naming doesn't expose this default behavior, which might cause problems for people coming from other software. Still, any good data scientist I know actually knows to read the documentation carefully and has learnt regularisation is the default.

missosoup 7y ago
The call signature does expose it though. Am I a minority in terms of using jupyter notebook and an IDE that shows call signatures and docstrings?
paulgb 7y ago
One thing that concerns me about the "you should read the documentation" argument is that even if the person writing the code did that for every function they used, anyone reading the code is likely to make (reasonable) assumptions. So bugs like accidentally regularizing, for example, could slip past code review.
- deleted 7y ago
  [deleted]
- ad404b8a372f2b9 7y ago
  It's a fair point for standard software engineering libraries, that the defaults should be obvious, but I don't think that it's possible to hold scientific libraries to the same standards. They require expert knowledge, and the assumptions you make about the models that are being used should be very carefully checked. There is no obvious defaults in that case because we're not writing software, we're building scientific models. For example, look at the neural network classifier model in scikit-learn: https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html https://scikit-learn.org/stable/modules/generated/sklearn.ne... None of these defaults can be assumed. If you showed me a function call with no optional arguments I couldn't tell you, nor anybody else that hasn't read the documentation, what activation is being used, what optimizer, learning rate, etc... In fact when coming across these kind of models during code reviews the nature of these parameters is the first thing I ask about.
  - rfeather 7y ago
    Agree. It's naive to assume your use case is the basic one when using these libraries or that you know the underlying implementation and all of the parameters, since in ML the implementations vary enough to affect the outcome and will have different controls. To the point of exploratory analysis in some of the parents ,I prefer statsmodels for that purpose. It's not quite up to where the similarly purposed tools are in other languages, but for most of my work where I care about interpretation, it hits the right spot between usability and providing the standard statistical outputs.
  - zenexer 7y ago
    Then there shouldn’t be any defaults, right? All of these issues could easily be avoided by forcing users to provide explicit parameters. It only makes sense to offer defaults if they’re intuitive.

darawk 7y ago

> If you're not using regularisation for logistic regression, you're most likely doing it wrong Why's that? Logistic regressions are perfectly reasonable to run for exploratory purposes. There are tons of situations where regularization is not what you want. > If you don't know to normalise your data prior to that It's not a question of not knowing. In an unregularized regression, you don't need to standardize your data beforehand. People who knows this but don't realize regularization is being applied by default can easily make this mistake.

funklute 7y ago
Logistic regression without regularization blows up if your data is linearly separable. This is an issue you don't really encounter with linear regression. EDIT: Apparently this is getting downvoted. To explain a bit more clearly: if you know nothing about the data, then your default should be to run with regularization, since you won't get a useful result if the data turns out to be linearly separable. In the case of linear regression, it's often sensible to start with non-regularized fitting, since that won't mess up your coefficients. But logistic regression is quite simply a different beast altogether, and this is covered in any intro on logistic regression you care to consult.
- gbrown 7y ago
  I think you're taking a perspective which is too focused on ML, ignoring the widespread statistical use of logistic regression in lower dimensional settings with reasonable a-priori covariate structures. > Logistic regression without regularization blows up if your data is linearly separable. This is an issue you don't really encounter with linear regression. Finding that your data is linearly separable is usually quite important - I do see students mess this up though, to be fair. If you want frequentist inference, non-penalized logistic regression is the way to go. (disclaimer, I'm predominantly Bayesian, so with proper priors the point is actually moot for me.)
  - missosoup 7y ago
    > I think you're taking a perspective which is too focused on ML scikit-learn is an ML library. It's explicitly not a stats library. The assumptions and defaults are correct for ML use cases. GP comment is entirely correct. If you want to do widespread statistics, use StatsModels which is designed for that.

CretinDesAlpes 7y ago

Completely agree. If you can't read documentation's software, you shouldn't do data science/ML in any cases.

alexfromapex 7y ago

Came here to say this. I can think of a lot of use cases where the feature data numbers will be dramatically different scales and the regularization helps to prevent that “noise” from dramatically affecting your model.

6gvONxR4sf7o 7y ago

Are you seriously arguing that R's implementation and statsmodels's implementations are doing it wrong? What if I had a model called "OrdinaryLeastSquares" in sklearn and had it regularize by default. After all, it's sklearn, so people probably wanted a penalized least squares model. That would be bad, because it's not what that means in most other libraries and textbooks where users have encountered the term.

bart_spoon 7y ago

The docs only state that because of this pushback. If I remember correctly, that is a relatively recent addition (within the last year I believe) that was added due to the communities dislike for both the defaults and the previous documentation.