3 ms·

Curious- why would you ever use linear regression for binary classification? Logistic regression can be implemented as linear regression predicting the input t

Curious- why would you ever use linear regression for binary classification? Logistic regression can be implemented as linear regression predicting the input to the logistic function.

oldskoolbob2000 13y ago
From my experience, it's easier to explain linear coefficients. Also, at least in R, linear regression tends to run faster than logistic regression.
- sgy 13y ago
  Might be a good read http://mrvar.fdv.uni-lj.si/pub/mz/mz1.1/pohar.pdf http://mrvar.fdv.uni-lj.si/pub/mz/mz1.1/pohar.pdf
michaelochurch 13y ago
Linear regression has a closed-form solution and, even when that is impractical (50k+ features, at which point you start not wanting to do the O(p^3) operations involved) and you have to use gradient descent, it still tends to run faster. If most of your data are in the p ∈ (0.2, 0.8) range, then the logistic curve is approximately linear anyway. Where linear regression fails you on a logistic problem is if you have a lot of data where the linear model will give p's outside of (0, 1). While the linear model's mathematical foundations are "incorrect"-- more precisely, it's not the maximum-likelihood model and may have a conditional likelihood of zero, that is, being impossible as "the correct" model on the data-- the truth is that, at 100k+ features, getting "the right" model is effectively impossible with most real-world data sets and you have to use simplifying assumptions and techniques (e.g. regularization, early stopping) anyway. Linear regression doesn't usually do as well as logistic regression, but sometimes it can, and most people are going to try both.
- dannypgh 13y ago
  I think I see the source of the confusion. Linear regression does not imply any specific method for solving for your coefficients. You can solve a linear regression by inverting your matrix or you can use gradient descent, to give two mechanisms. Yes, solving for your coefficents based on inverting the matrix can be O(n^3) (although wikipedia tells me there's ways of doing it that are O(n^2.373), but at the end of the day, it's slow for large n) but gradient descent doesn't have this problem, as gradient descent is O(n) with regards to your training set. And if you're implementing gradient descent to build a classifier, you can equivalently do logistic regression by just changing your cost function (that is, you'll be solving a linear regression to give you f(x) such that S(f(x)) is the probability you care about, with S(x) being the sigmoid function). So linear regression can perhaps be easier than logistic regression, but only when n is so small that you're solving your linear regression by inverting the matrix. Once you've switched to gradient descent (or anything that can solve for solutions to arbitrary cost functions) there is no difference.
  - michaelochurch 13y ago
    Once you've switched to gradient descent (or anything that can solve for solutions to arbitrary cost functions) there is no difference. I'd have to work this out on paper. I think that's essentially right, but most of your CPU cost is going to be in evaluating that function (or, more accurately, its derivative) and not all functions are, in terms of computational cost, equal. I think the logistic model might take longer to train, but I'd have to work it out to be sure. You're right that the code isn't much different. So linear regression can perhaps be easier than logistic regression, but only when n is so small that you're solving your linear regression by inverting the matrix. The matrix grows with the number of features, so if you have 1 billion observations but only 100 features (dimensions) you can do that.
    - dannypgh 13y ago
      I doubt the CPU costs will be measurably different. I suggest watching: https://class.coursera.org/ml-003/lecture/36 https://class.coursera.org/ml-003/lecture/36 and you can jump to 7 minutes if you don't want to watch the whole thing. The only difference is instead of merely doing the matrix multiplication of your observation times your coefficients (the scalar product of θ and x) you have to take the output and use it as the input to the logistic function. The matrix multiplication will grow in complexity with the number of non-zero features (either non-zero coefficients in your model or non-zero features in your observation, presumably the latter) and the evaluation of the sigmoid function will be constant time, and on many architectures will be a very small number of instructions. If you're operating in very small numbers of dimensions what you're doing may still be interesting but I'm not sure it qualifies as "Big Data," regardless of the number of available observations. After all, if you're producing something with actual utility, you'll probably converge sooner rather than later...

Curious- why would you ever use linear regression for binary classification? Logistic regression can be implemented as linear regression predicting the input to the logistic function.

oldskoolbob2000 13y ago

From my experience, it's easier to explain linear coefficients. Also, at least in R, linear regression tends to run faster than logistic regression.

sgy 13y ago
Might be a good read http://mrvar.fdv.uni-lj.si/pub/mz/mz1.1/pohar.pdf http://mrvar.fdv.uni-lj.si/pub/mz/mz1.1/pohar.pdf

michaelochurch 13y ago

Linear regression has a closed-form solution and, even when that is impractical (50k+ features, at which point you start not wanting to do the O(p^3) operations involved) and you have to use gradient descent, it still tends to run faster. If most of your data are in the p ∈ (0.2, 0.8) range, then the logistic curve is approximately linear anyway. Where linear regression fails you on a logistic problem is if you have a lot of data where the linear model will give p's outside of (0, 1). While the linear model's mathematical foundations are "incorrect"-- more precisely, it's not the maximum-likelihood model and may have a conditional likelihood of zero, that is, being impossible as "the correct" model on the data-- the truth is that, at 100k+ features, getting "the right" model is effectively impossible with most real-world data sets and you have to use simplifying assumptions and techniques (e.g. regularization, early stopping) anyway. Linear regression doesn't usually do as well as logistic regression, but sometimes it can, and most people are going to try both.

dannypgh 13y ago
I think I see the source of the confusion. Linear regression does not imply any specific method for solving for your coefficients. You can solve a linear regression by inverting your matrix or you can use gradient descent, to give two mechanisms. Yes, solving for your coefficents based on inverting the matrix can be O(n^3) (although wikipedia tells me there's ways of doing it that are O(n^2.373), but at the end of the day, it's slow for large n) but gradient descent doesn't have this problem, as gradient descent is O(n) with regards to your training set. And if you're implementing gradient descent to build a classifier, you can equivalently do logistic regression by just changing your cost function (that is, you'll be solving a linear regression to give you f(x) such that S(f(x)) is the probability you care about, with S(x) being the sigmoid function). So linear regression can perhaps be easier than logistic regression, but only when n is so small that you're solving your linear regression by inverting the matrix. Once you've switched to gradient descent (or anything that can solve for solutions to arbitrary cost functions) there is no difference.
- michaelochurch 13y ago
  Once you've switched to gradient descent (or anything that can solve for solutions to arbitrary cost functions) there is no difference. I'd have to work this out on paper. I think that's essentially right, but most of your CPU cost is going to be in evaluating that function (or, more accurately, its derivative) and not all functions are, in terms of computational cost, equal. I think the logistic model might take longer to train, but I'd have to work it out to be sure. You're right that the code isn't much different. So linear regression can perhaps be easier than logistic regression, but only when n is so small that you're solving your linear regression by inverting the matrix. The matrix grows with the number of features, so if you have 1 billion observations but only 100 features (dimensions) you can do that.
  - dannypgh 13y ago
    I doubt the CPU costs will be measurably different. I suggest watching: https://class.coursera.org/ml-003/lecture/36 https://class.coursera.org/ml-003/lecture/36 and you can jump to 7 minutes if you don't want to watch the whole thing. The only difference is instead of merely doing the matrix multiplication of your observation times your coefficients (the scalar product of θ and x) you have to take the output and use it as the input to the logistic function. The matrix multiplication will grow in complexity with the number of non-zero features (either non-zero coefficients in your model or non-zero features in your observation, presumably the latter) and the evaluation of the sigmoid function will be constant time, and on many architectures will be a very small number of instructions. If you're operating in very small numbers of dimensions what you're doing may still be interesting but I'm not sure it qualifies as "Big Data," regardless of the number of available observations. After all, if you're producing something with actual utility, you'll probably converge sooner rather than later...