4 ms·

Perhaps I was educated wrong, or perhaps I've forgotten almost everything from my math degrees, but I can't wrap my head around this blog post, let alone how it

by pflats 10y ago

Perhaps I was educated wrong, or perhaps I've forgotten almost everything from my math degrees, but I can't wrap my head around this blog post, let alone how it was upvoted so highly. Asking why the logistic/sigmoid function is used in logistic regression is like asking why the quadratic x^2 function is used in quadratic regression or why The Beatles are the band whose recording is on the White Album. A regression fits discrete data to a curve. A logistic regression fits discrete data to the logistic curve. Asking where these functions come from is interesting, but that's another question entirely. I had always looked at the logistic function as having an easy source, the ODE: y' = y(1-y) As for how that relates to probability (p' = p[1-p]) I leave as an exercise to the reader. >Note: the log of the odds function is often called "the logistic" function. I was taught that this (the inverse of the logistic function) is the logit function. My brain is admittedly a little fuzzier than it used to be these days due to medical issues, so if anyone can clarify for me, I'd appreciate it.

TDL 10y ago
"let alone how it was upvoted so highly. The reason I upvoted this post was with the hope that comments like this would be made. I'm still solidifying my grasp of ML/math topics so when these posts come up I always hope there are commenters with more experience/education in these topics to further enlighten me.
- argonaut 10y ago
  Yikes. Unfortunately this is why (IMO) I see a lot of mediocre ML articles upvoted very highly on HN that have either very little discussion, or generally poor quality discussion, especially around hot topics like neural networks. The discussion I've seen here is honestly the exception to the norm (probably due to the statistical nature of the topic). Personally this is one of the few areas where Reddit does better.
  - TDL 10y ago
    Any suggestions which subreddits I should be looking at?
    - argonaut 10y ago
      /r/machinelearning is pretty high quality, very technical. You'll see a lot of well-known researchers posting there sometimes (whereas here you only see karpathy every once in a blue moon). I'm only a random guy who did ml research as an undergrad.
  - mindcrime 10y ago
    Unfortunately this is why (IMO) I see a lot of mediocre ML articles upvoted very highly on HN that have either very little discussion, or generally poor quality discussion, especially around hot topics like neural networks I think you're asking too much of upvotes. Since "upvote" and "save" are the same thing on HN, a lot of people upvote anything that meets the bar of "based on the title, it's plausible this is something I'd like to read eventually". That's really about all you can expect of upvotes.
    - argonaut 10y ago
      I realize that's what many people do, but I don't agree with it because it's an implicit signal of interest/worthiness, due to affecting rankings. It gives people on HN the wrong perception of what is popular/worth learning and risks leading autodidacts astray.
      - mindcrime 10y ago
        I actually agree and I would love to see HN adopt a separate "save" flag aside from "upvote" (like Reddit does), but... it is what is, for the moment anyway.
thashim 10y ago
There are good reasons other than `it models the data well' to use a logit. Of course, the perspective that much like least-squares, logit regression does best when your data is generated from a logistic GLM is true, but not the only story. In my opinion, the best justification for the logit is that it is a easily-optimizable member of the family of surrogate losses: http://fa.bianp.net/blog/2014/surrogate-loss-functions-in-machine-learning/ http://fa.bianp.net/blog/2014/surrogate-loss-functions-in-ma... The papers cited in the article pretty cleanly explain why the logit and other surrogate losses are the 'natural choice' for classification. This also explains why other non-generative models like the SVM perform similarly well to the logit in practice.
drostie 10y ago
Right, log(p/(1-p)) is the logit function; its inverse is the logistic function. Here's another motivation for why logits are practically useful: Bayes' Theorem! You've probably seen Bayes' theorem in the depressing form: Pr(C | E) = Pr(E | C) Pr(C) / Pr(E). Here E is evidence, C is a claim, and Pr() is the probability function mapping sets of possibilities to their probability. It says that the probability of a claim, given some evidence, turns out to be the probability of the evidence given the claim, times the prior probability of the claim, divided by the prior probability of the evidence. If you've gotten a little further with it, you may have seen that (if you work out a lot of mathematics) you can tack on a | O to all of these probabilities, where the O stands for old-evidence about the world, so that you can make this expression recursive: Pr(C | E O) = Pr(E | C O) Pr(C | O) / Pr(E | O). (In case it's not totally clear, when I juxtapose sets I intend to mean their set-intersection, which is usually analogized to a sort of "set product". So `E O` above means "both E and O": given both the old evidence O and the new evidence E.) And if you've gotten a bit further you may have seen that in practice we often will use the fact that Pr(E | O) = Pr(E | C O) Pr(C | O) + Pr(E | !C O) Pr(!C | O), where ! is of course set-complement or the prefix "not-". This allows us to write: Pr(E | C O) Pr(C | O) Pr(C | E O) = ---------------------------------- Pr(E | C O) Pr(C | O) + Pr(E | !C O) [1 − Pr(C | O)] Same theorem, but a lot more complicated because it's a lot more real-world! But we can actually rewrite this last expression using the odds ratio, Odds(A) = Pr(A) / [1 - Pr(A)] in a much simpler form as: Pr(E | C O) Odds(C | E O) = --------------- * Odds(C | O) Pr(E | !C O) The recursive formula therefore looks like a chain of multiplications of factors due to newer and newer evidence. On the assumption that a bunch of pieces of evidence are independent, each factor decouples from the others and just becomes some factor W_i. The logit expression looks simply like: log(Odds(C | E1 E2 ... En)) = log(Odds(C)) + log(W1) + log(W2) + ... + log(Wn). The logistic regression is therefore assuming a linear approximation: each continuous factor F_i that you're studying corresponds to some independent piece of evidence E_i which contributes some factor W_i = log(Pr(E_i | C) / Pr(E_i | !C)), and we're assuming that this logarithm-of-a-conditional-probability-ratio can then be roughly approximated as a_i + b_i F_i. We collect together log(Pr(C)) + Sum_i a_i together as some leading coefficient A and then we get our fit as N+1 variables (A, {b_i}).

by pflats 10y ago

TDL 10y ago

"let alone how it was upvoted so highly. The reason I upvoted this post was with the hope that comments like this would be made. I'm still solidifying my grasp of ML/math topics so when these posts come up I always hope there are commenters with more experience/education in these topics to further enlighten me.

argonaut 10y ago
Yikes. Unfortunately this is why (IMO) I see a lot of mediocre ML articles upvoted very highly on HN that have either very little discussion, or generally poor quality discussion, especially around hot topics like neural networks. The discussion I've seen here is honestly the exception to the norm (probably due to the statistical nature of the topic). Personally this is one of the few areas where Reddit does better.
- TDL 10y ago
  Any suggestions which subreddits I should be looking at?
  - argonaut 10y ago
    /r/machinelearning is pretty high quality, very technical. You'll see a lot of well-known researchers posting there sometimes (whereas here you only see karpathy every once in a blue moon). I'm only a random guy who did ml research as an undergrad.
- mindcrime 10y ago
  Unfortunately this is why (IMO) I see a lot of mediocre ML articles upvoted very highly on HN that have either very little discussion, or generally poor quality discussion, especially around hot topics like neural networks I think you're asking too much of upvotes. Since "upvote" and "save" are the same thing on HN, a lot of people upvote anything that meets the bar of "based on the title, it's plausible this is something I'd like to read eventually". That's really about all you can expect of upvotes.
  - argonaut 10y ago
    I realize that's what many people do, but I don't agree with it because it's an implicit signal of interest/worthiness, due to affecting rankings. It gives people on HN the wrong perception of what is popular/worth learning and risks leading autodidacts astray.
    - mindcrime 10y ago
      I actually agree and I would love to see HN adopt a separate "save" flag aside from "upvote" (like Reddit does), but... it is what is, for the moment anyway.

thashim 10y ago

There are good reasons other than `it models the data well' to use a logit. Of course, the perspective that much like least-squares, logit regression does best when your data is generated from a logistic GLM is true, but not the only story. In my opinion, the best justification for the logit is that it is a easily-optimizable member of the family of surrogate losses: http://fa.bianp.net/blog/2014/surrogate-loss-functions-in-machine-learning/ http://fa.bianp.net/blog/2014/surrogate-loss-functions-in-ma... The papers cited in the article pretty cleanly explain why the logit and other surrogate losses are the 'natural choice' for classification. This also explains why other non-generative models like the SVM perform similarly well to the logit in practice.

drostie 10y ago

Right, log(p/(1-p)) is the logit function; its inverse is the logistic function. Here's another motivation for why logits are practically useful: Bayes' Theorem! You've probably seen Bayes' theorem in the depressing form: Pr(C | E) = Pr(E | C) Pr(C) / Pr(E). Here E is evidence, C is a claim, and Pr() is the probability function mapping sets of possibilities to their probability. It says that the probability of a claim, given some evidence, turns out to be the probability of the evidence given the claim, times the prior probability of the claim, divided by the prior probability of the evidence. If you've gotten a little further with it, you may have seen that (if you work out a lot of mathematics) you can tack on a | O to all of these probabilities, where the O stands for old-evidence about the world, so that you can make this expression recursive: Pr(C | E O) = Pr(E | C O) Pr(C | O) / Pr(E | O). (In case it's not totally clear, when I juxtapose sets I intend to mean their set-intersection, which is usually analogized to a sort of "set product". So `E O` above means "both E and O": given both the old evidence O and the new evidence E.) And if you've gotten a bit further you may have seen that in practice we often will use the fact that Pr(E | O) = Pr(E | C O) Pr(C | O) + Pr(E | !C O) Pr(!C | O), where ! is of course set-complement or the prefix "not-". This allows us to write: Pr(E | C O) Pr(C | O) Pr(C | E O) = ---------------------------------- Pr(E | C O) Pr(C | O) + Pr(E | !C O) [1 − Pr(C | O)] Same theorem, but a lot more complicated because it's a lot more real-world! But we can actually rewrite this last expression using the odds ratio, Odds(A) = Pr(A) / [1 - Pr(A)] in a much simpler form as: Pr(E | C O) Odds(C | E O) = --------------- * Odds(C | O) Pr(E | !C O) The recursive formula therefore looks like a chain of multiplications of factors due to newer and newer evidence. On the assumption that a bunch of pieces of evidence are independent, each factor decouples from the others and just becomes some factor W_i. The logit expression looks simply like: log(Odds(C | E1 E2 ... En)) = log(Odds(C)) + log(W1) + log(W2) + ... + log(Wn). The logistic regression is therefore assuming a linear approximation: each continuous factor F_i that you're studying corresponds to some independent piece of evidence E_i which contributes some factor W_i = log(Pr(E_i | C) / Pr(E_i | !C)), and we're assuming that this logarithm-of-a-conditional-probability-ratio can then be roughly approximated as a_i + b_i F_i. We collect together log(Pr(C)) + Sum_i a_i together as some leading coefficient A and then we get our fit as N+1 variables (A, {b_i}).