9 ms·

The Sigmoid Function in Logistic Regression

97 points by krosaen 10y ago

nabla9 10y ago
y = a0 + Σ(ai×xi) Use linear regression when 1 unit increase in x increases y by a. Use logistic regression when 1 unit increases x increases log odds by a. Use Poisson regression when 1 unit increase in x increases y by y×e^a (see also negative binomial regression)
- nimish 10y ago
  Important that these are on average. https://en.wikipedia.org/wiki/Generalized_linear_model https://en.wikipedia.org/wiki/Generalized_linear_model
- gbrown 10y ago
  Binomial and Poisson regression can both be performed with non-canonical link functions, which sometimes provide a better fit or more interpretable results (see: canonical link for gamma regression).
MidsizeBlowfish 10y ago
There is also the interesting interpretation of logistic regression as a latent choice/utility model, where unobserved variation in utility is modeled using the logistic distribution. In this interpretation, the sigmoid function arises as the CDF of the logistic distribution. This interpretation is also nice as it shows the natural relationship between logistic and probit regression. Probit regression arises when the unobserved utility is modeled with a Gaussian distribution.
- thanatropism 10y ago
  In this interpretation we expect decision-makers to be utility-maximizers - for example, as whether I buy or not Thing X; so, we expect Gumbel (an extreme value distribution) errors. But we're looking for Utility(1) minus Utility(0) (buying vs. not buying) and the difference between two Gumbels is a logistic. More detail: http://fisher.osu.edu/~schroeder.9/AMIS900/ch5.pdf http://fisher.osu.edu/~schroeder.9/AMIS900/ch5.pdf
  - MidsizeBlowfish 10y ago
    Ah, right you are! Thanks for correcting my mistake :)
ponderingHplus 10y ago
Your learning sabbatical seems very well thought out and I'm glad your documenting your journey publicly so others can learn with you. I actually started down a similar path to self-learn AI, but ended up going the conventional Masters degree route. I even setup a skeleton of a blog to document the journey. The home page has some courses you may want to add to your resource list. I've taken quite a few courses on Udacity and have enjoyed almost all of them. http://cole-maclean.github.io/About/ http://cole-maclean.github.io/About/ Good luck on your adventure!
- krosaen 10y ago
  Thanks! That graphic at http://cole-maclean.github.io/ http://cole-maclean.github.io/ is pretty neat, I take it that was your WIP on your self-study before you opted for the masters program? Good luck to you as well.
  - gtani 10y ago
    There's lots of preparatory ML reading lists, e.g. metacademy.org roadmap, Goodfellow et al. DL book, Shalev-Shwartz/Ben-David ML text (both books freely available content), review materials on Stanford course syllabuses by Socher, Ng and Karpathy. Usual recs are Strang or Axler Linear Algebra, Ross or Tsitsiklis for Probability, Spivak Calculus, Boyd Convex Optimization, and other folks will recommend real analysis, diff eq's, topology, etc.
oddthink 10y ago
I'm sorry, that seems like the most backwards explanation of logistic regression I've ever seen. It uses the sigmoid function because it's logistic regression. The statistical model is a Bernoulli distribution. Write down the likelihood, maximize it, and you get the sigmoid.
- llasram 10y ago
  I think this is a good example of the problems with autodidactism. For all its flaws, structured education is what makes common knowledge common. When you study on your own, you don't know what you don't know, and there's no one there to point out your obvious-with-the-right-knowledge lapses.
  - brudgers 10y ago
    To me it's an example of what Hacker News offers people interested in learning [the human, not machine type]. This very article shows how it is a place where there are no long term consequences equivalent to grades: just a wee bit of standard internet asshattery and those complaints about the youth of today that have been in the Western cannon since Socrates was drinking Hemlock.
- krosaen 10y ago
  Yes, it uses a sigmoid function because it's logistic regression and therefore you are using the inverse of the logistic function, the sigmoid, as the notebook explains. But I think it's worth running through that and exploring why it's useful to use a logistic function in the first place (maps linear combo to (-1, 1) range). It was useful to me anyways, perhaps obvious to others. Thanks for pointing out the relation to the Bernoulli distribution.
  - ryanmonroe 10y ago
    The logistic function maps to (0,1). The hyperbolic tangent tanh(x)=2*logistic(2x)-1 maps to (-1,1).
- brudgers 10y ago
mjw 10y ago
Their answer is pretty much 'because it's based on the log-odds', which to me is still only very mild motivation. There are other non-linearities which people use to map onto (0, 1), for example probit regression uses the Normal CDF. In fact you can use the CDF of any distribution supported on the whole real line, and the sigmoid is an example of this -- it's the CDF of a standard logistic distribution [1]. There's a nice interpretation for this using an extra latent variable: for probit regression, you take your linear predictor, add a standard normal noise term, and the response is determined by the sign of the result. For logistic regression, same thing except make it a standard logistic instead. This then extends nicely to ordinal regression too. [0] https://en.wikipedia.org/wiki/Probit_model https://en.wikipedia.org/wiki/Probit_model [1] https://en.wikipedia.org/wiki/Logistic_distribution https://en.wikipedia.org/wiki/Logistic_distribution
- gbrown 10y ago
  There are other nice properties. For example, because the logit link is canonical for the binomial GLM, inference about unknown parameters using it is based on sufficient statistics. It's certainly not the only option though, and not always the best fit.
  - mjw 10y ago
    Ah yep, I forgot it's the canonical link. That's more of a small computational convenience though, right, at least when fitting a straightforward GLM -- it should be very cheap to fit regardless. I suppose the logistic having heavier tails than the normal is probably the main consideration in motivating one or the other as the better model for a given situation. Logistic being is heavier-tailed, is potentially more robust to outliers. Which in terms of binary data, means that it might be a better choice in cases where an unexpected outcome is possible even in the most clear-cut cases. Probit regression with its heavier normal tails, might be a better fit in cases where the response is expected to be pretty much deterministic in clear-cut cases, and where quite severe inferences can be drawn from unexpected outcomes in those cases. Sound fair?
- jules 10y ago
  Is there a natural justification for the logistic distribution though?
pflats 10y ago
Perhaps I was educated wrong, or perhaps I've forgotten almost everything from my math degrees, but I can't wrap my head around this blog post, let alone how it was upvoted so highly. Asking why the logistic/sigmoid function is used in logistic regression is like asking why the quadratic x^2 function is used in quadratic regression or why The Beatles are the band whose recording is on the White Album. A regression fits discrete data to a curve. A logistic regression fits discrete data to the logistic curve. Asking where these functions come from is interesting, but that's another question entirely. I had always looked at the logistic function as having an easy source, the ODE: y' = y(1-y) As for how that relates to probability (p' = p[1-p]) I leave as an exercise to the reader. >Note: the log of the odds function is often called "the logistic" function. I was taught that this (the inverse of the logistic function) is the logit function. My brain is admittedly a little fuzzier than it used to be these days due to medical issues, so if anyone can clarify for me, I'd appreciate it.
- TDL 10y ago
  "let alone how it was upvoted so highly. The reason I upvoted this post was with the hope that comments like this would be made. I'm still solidifying my grasp of ML/math topics so when these posts come up I always hope there are commenters with more experience/education in these topics to further enlighten me.
  - argonaut 10y ago
    Yikes. Unfortunately this is why (IMO) I see a lot of mediocre ML articles upvoted very highly on HN that have either very little discussion, or generally poor quality discussion, especially around hot topics like neural networks. The discussion I've seen here is honestly the exception to the norm (probably due to the statistical nature of the topic). Personally this is one of the few areas where Reddit does better.
    - TDL 10y ago
      Any suggestions which subreddits I should be looking at?
tacos 10y ago
Agreed. This was painful to read until I clicked "About" and realized the author was on some sort of... self-exploration mission. The need to publish this stuff is debatable. I don't think promoting it -- without clear caveats -- is particularly helpful. Self-submitting it to HN borders on naive arrogance. Frankly you don't know what you don't know and the amount of cultural damage you can do in hopes of finding the one person who'll risk downvotes to say "nope, that's sorta wrong" doesn't seem like a great use of everyone's time. I'd prefer we reserve that for bigger, broader, riskier, innovative concepts. Not e^-x.
- brudgers 10y ago
  For me, one of the great things about Hacker News is that every person on the site is a new comer to some of the topics presented. Perhaps more importantly, for any technical topic there is a high probability that there is a user for whom the article is so basic as to be technically uninformative...I mean there is a reasonable probability that one or more of the people who use HN actually rode in Feynman's van at some point. To me, the risk to Hacker News is that comments inconsistent with HN's ethos of civility, dissuade people from the vulnerability that comes when one posts original work. And I'd prefer people put a bit more effort into writing thoughtful and reflective comments.
  - tacos 10y ago
    Agreed. However I draw the line at obvious amateurs submitting their own articles. I fail to see why we need to celebrate every dope making a lounge chair out of packing crates. Coincidentally I've driven Feynman's van. In fact it's the quality of Feynman's highly accessible writing that has me so disappointed in half-assed articles such as this one. A young, knowledge-hungry, enthusiastic crowd such as HN deserves BETTER input, not junk like this. Surely we can retain the positivity while culling the crap.
    - skybrian 10y ago
      If there are better articles, how about submitting some and hopefully they'll be upvoted more.
hammock 10y ago
What regression do you use when modeling total views collected by a YouTube video over time?
- gbrown 10y ago
  Probably some sort of growth model.
Animats 10y ago
Sigmoid functions also come up in neural nets. But that's because you need something that's differentiable. If you just used a step function, backpropagation wouldn't work, because small changes in the tuning parameters wouldn't do anything. You can think of a sigmoid as a step function pushed through a low-pass filter. It would be interesting to try step functions run through low-pass filters with different cutoffs to see which work best for logistic regression. Anything that produces a monotonically increasing value ought to converge. The only question is how fast.
- gcp 10y ago
  >backpropagation wouldn't work, because small changes in the tuning parameters wouldn't do anything cough Rectified Linear Units cough
- eggy 10y ago
  I was going to chime in about neural nets too, since I thought that was what the article was going to mention instead of just defining it for logistic regression. It is used because it is continuous and differentiable as you say, and it keeps the bounds neat between 0 and 1, although most modern deep neural nets use an ReLU, or rectifier function of some sort. The softplus is an ReLU, that has the nice property of its derivative being the sigmoid function.
- argonaut 10y ago
  To expand on what gcp is saying, that's not true. Rectified linear units do better than sigmoids but are not differentiable (but are sub-differentiable), and ReLUs have the problem where small changes don't do anything (in the negative regime).
moultano 10y ago
There are much better motivations for log-odds and the logistic function than just ensuring the predicted values are bounded probabilities. If you want to do bayesian updating of your model upon seeing new data, and you want to express this as a linear function, a sum of log-odds values is the way to do it. The best way to understand this is to learn about "naive bayes" first. If you start with naive bayes, but replace the values that are based on observed frequencies with values that you can just optimize for your objective, you get logistic regression.

97 points by krosaen 10y ago

y = a0 + Σ(ai×xi) Use linear regression when 1 unit increase in x increases y by a. Use logistic regression when 1 unit increases x increases log odds by a. Use Poisson regression when 1 unit increase in x increases y by y×e^a (see also negative binomial regression)

nimish 10y ago
Important that these are on average. https://en.wikipedia.org/wiki/Generalized_linear_model https://en.wikipedia.org/wiki/Generalized_linear_model
gbrown 10y ago
Binomial and Poisson regression can both be performed with non-canonical link functions, which sometimes provide a better fit or more interpretable results (see: canonical link for gamma regression).

MidsizeBlowfish 10y ago

There is also the interesting interpretation of logistic regression as a latent choice/utility model, where unobserved variation in utility is modeled using the logistic distribution. In this interpretation, the sigmoid function arises as the CDF of the logistic distribution. This interpretation is also nice as it shows the natural relationship between logistic and probit regression. Probit regression arises when the unobserved utility is modeled with a Gaussian distribution.

thanatropism 10y ago
In this interpretation we expect decision-makers to be utility-maximizers - for example, as whether I buy or not Thing X; so, we expect Gumbel (an extreme value distribution) errors. But we're looking for Utility(1) minus Utility(0) (buying vs. not buying) and the difference between two Gumbels is a logistic. More detail: http://fisher.osu.edu/~schroeder.9/AMIS900/ch5.pdf http://fisher.osu.edu/~schroeder.9/AMIS900/ch5.pdf
- MidsizeBlowfish 10y ago
  Ah, right you are! Thanks for correcting my mistake :)

ponderingHplus 10y ago

Your learning sabbatical seems very well thought out and I'm glad your documenting your journey publicly so others can learn with you. I actually started down a similar path to self-learn AI, but ended up going the conventional Masters degree route. I even setup a skeleton of a blog to document the journey. The home page has some courses you may want to add to your resource list. I've taken quite a few courses on Udacity and have enjoyed almost all of them. http://cole-maclean.github.io/About/ http://cole-maclean.github.io/About/ Good luck on your adventure!

krosaen 10y ago
Thanks! That graphic at http://cole-maclean.github.io/ http://cole-maclean.github.io/ is pretty neat, I take it that was your WIP on your self-study before you opted for the masters program? Good luck to you as well.
- gtani 10y ago
  There's lots of preparatory ML reading lists, e.g. metacademy.org roadmap, Goodfellow et al. DL book, Shalev-Shwartz/Ben-David ML text (both books freely available content), review materials on Stanford course syllabuses by Socher, Ng and Karpathy. Usual recs are Strang or Axler Linear Algebra, Ross or Tsitsiklis for Probability, Spivak Calculus, Boyd Convex Optimization, and other folks will recommend real analysis, diff eq's, topology, etc.

oddthink 10y ago

I'm sorry, that seems like the most backwards explanation of logistic regression I've ever seen. It uses the sigmoid function because it's logistic regression. The statistical model is a Bernoulli distribution. Write down the likelihood, maximize it, and you get the sigmoid.

llasram 10y ago
I think this is a good example of the problems with autodidactism. For all its flaws, structured education is what makes common knowledge common. When you study on your own, you don't know what you don't know, and there's no one there to point out your obvious-with-the-right-knowledge lapses.
- brudgers 10y ago
  To me it's an example of what Hacker News offers people interested in learning [the human, not machine type]. This very article shows how it is a place where there are no long term consequences equivalent to grades: just a wee bit of standard internet asshattery and those complaints about the youth of today that have been in the Western cannon since Socrates was drinking Hemlock.
krosaen 10y ago
Yes, it uses a sigmoid function because it's logistic regression and therefore you are using the inverse of the logistic function, the sigmoid, as the notebook explains. But I think it's worth running through that and exploring why it's useful to use a logistic function in the first place (maps linear combo to (-1, 1) range). It was useful to me anyways, perhaps obvious to others. Thanks for pointing out the relation to the Bernoulli distribution.
- ryanmonroe 10y ago
  The logistic function maps to (0,1). The hyperbolic tangent tanh(x)=2*logistic(2x)-1 maps to (-1,1).
brudgers 10y ago

mjw 10y ago

Their answer is pretty much 'because it's based on the log-odds', which to me is still only very mild motivation. There are other non-linearities which people use to map onto (0, 1), for example probit regression uses the Normal CDF. In fact you can use the CDF of any distribution supported on the whole real line, and the sigmoid is an example of this -- it's the CDF of a standard logistic distribution [1]. There's a nice interpretation for this using an extra latent variable: for probit regression, you take your linear predictor, add a standard normal noise term, and the response is determined by the sign of the result. For logistic regression, same thing except make it a standard logistic instead. This then extends nicely to ordinal regression too. [0] https://en.wikipedia.org/wiki/Probit_model https://en.wikipedia.org/wiki/Probit_model [1] https://en.wikipedia.org/wiki/Logistic_distribution https://en.wikipedia.org/wiki/Logistic_distribution

gbrown 10y ago
There are other nice properties. For example, because the logit link is canonical for the binomial GLM, inference about unknown parameters using it is based on sufficient statistics. It's certainly not the only option though, and not always the best fit.
- mjw 10y ago
  Ah yep, I forgot it's the canonical link. That's more of a small computational convenience though, right, at least when fitting a straightforward GLM -- it should be very cheap to fit regardless. I suppose the logistic having heavier tails than the normal is probably the main consideration in motivating one or the other as the better model for a given situation. Logistic being is heavier-tailed, is potentially more robust to outliers. Which in terms of binary data, means that it might be a better choice in cases where an unexpected outcome is possible even in the most clear-cut cases. Probit regression with its heavier normal tails, might be a better fit in cases where the response is expected to be pretty much deterministic in clear-cut cases, and where quite severe inferences can be drawn from unexpected outcomes in those cases. Sound fair?
jules 10y ago
Is there a natural justification for the logistic distribution though?

pflats 10y ago

Perhaps I was educated wrong, or perhaps I've forgotten almost everything from my math degrees, but I can't wrap my head around this blog post, let alone how it was upvoted so highly. Asking why the logistic/sigmoid function is used in logistic regression is like asking why the quadratic x^2 function is used in quadratic regression or why The Beatles are the band whose recording is on the White Album. A regression fits discrete data to a curve. A logistic regression fits discrete data to the logistic curve. Asking where these functions come from is interesting, but that's another question entirely. I had always looked at the logistic function as having an easy source, the ODE: y' = y(1-y) As for how that relates to probability (p' = p[1-p]) I leave as an exercise to the reader. >Note: the log of the odds function is often called "the logistic" function. I was taught that this (the inverse of the logistic function) is the logit function. My brain is admittedly a little fuzzier than it used to be these days due to medical issues, so if anyone can clarify for me, I'd appreciate it.

TDL 10y ago
"let alone how it was upvoted so highly. The reason I upvoted this post was with the hope that comments like this would be made. I'm still solidifying my grasp of ML/math topics so when these posts come up I always hope there are commenters with more experience/education in these topics to further enlighten me.
- argonaut 10y ago
  Yikes. Unfortunately this is why (IMO) I see a lot of mediocre ML articles upvoted very highly on HN that have either very little discussion, or generally poor quality discussion, especially around hot topics like neural networks. The discussion I've seen here is honestly the exception to the norm (probably due to the statistical nature of the topic). Personally this is one of the few areas where Reddit does better.
  - TDL 10y ago
    Any suggestions which subreddits I should be looking at?

tacos 10y ago

Agreed. This was painful to read until I clicked "About" and realized the author was on some sort of... self-exploration mission. The need to publish this stuff is debatable. I don't think promoting it -- without clear caveats -- is particularly helpful. Self-submitting it to HN borders on naive arrogance. Frankly you don't know what you don't know and the amount of cultural damage you can do in hopes of finding the one person who'll risk downvotes to say "nope, that's sorta wrong" doesn't seem like a great use of everyone's time. I'd prefer we reserve that for bigger, broader, riskier, innovative concepts. Not e^-x.

brudgers 10y ago
For me, one of the great things about Hacker News is that every person on the site is a new comer to some of the topics presented. Perhaps more importantly, for any technical topic there is a high probability that there is a user for whom the article is so basic as to be technically uninformative...I mean there is a reasonable probability that one or more of the people who use HN actually rode in Feynman's van at some point. To me, the risk to Hacker News is that comments inconsistent with HN's ethos of civility, dissuade people from the vulnerability that comes when one posts original work. And I'd prefer people put a bit more effort into writing thoughtful and reflective comments.
- tacos 10y ago
  Agreed. However I draw the line at obvious amateurs submitting their own articles. I fail to see why we need to celebrate every dope making a lounge chair out of packing crates. Coincidentally I've driven Feynman's van. In fact it's the quality of Feynman's highly accessible writing that has me so disappointed in half-assed articles such as this one. A young, knowledge-hungry, enthusiastic crowd such as HN deserves BETTER input, not junk like this. Surely we can retain the positivity while culling the crap.
  - skybrian 10y ago
    If there are better articles, how about submitting some and hopefully they'll be upvoted more.

hammock 10y ago

What regression do you use when modeling total views collected by a YouTube video over time?

gbrown 10y ago
Probably some sort of growth model.

Animats 10y ago

Sigmoid functions also come up in neural nets. But that's because you need something that's differentiable. If you just used a step function, backpropagation wouldn't work, because small changes in the tuning parameters wouldn't do anything. You can think of a sigmoid as a step function pushed through a low-pass filter. It would be interesting to try step functions run through low-pass filters with different cutoffs to see which work best for logistic regression. Anything that produces a monotonically increasing value ought to converge. The only question is how fast.

gcp 10y ago
>backpropagation wouldn't work, because small changes in the tuning parameters wouldn't do anything cough Rectified Linear Units cough
eggy 10y ago
I was going to chime in about neural nets too, since I thought that was what the article was going to mention instead of just defining it for logistic regression. It is used because it is continuous and differentiable as you say, and it keeps the bounds neat between 0 and 1, although most modern deep neural nets use an ReLU, or rectifier function of some sort. The softplus is an ReLU, that has the nice property of its derivative being the sigmoid function.
argonaut 10y ago
To expand on what gcp is saying, that's not true. Rectified linear units do better than sigmoids but are not differentiable (but are sub-differentiable), and ReLUs have the problem where small changes don't do anything (in the negative regime).

moultano 10y ago

There are much better motivations for log-odds and the logistic function than just ensuring the predicted values are bounded probabilities. If you want to do bayesian updating of your model upon seeing new data, and you want to express this as a linear function, a sum of log-odds values is the way to do it. The best way to understand this is to learn about "naive bayes" first. If you start with naive bayes, but replace the values that are based on observed frequencies with values that you can just optimize for your objective, you get logistic regression.