3 ms·
An explanation on what `logit` is, how it is obtained, would have been nice. Is it the count of occurrences of the word following "the mouse ate the"?
by qwertox 4y ago
An explanation on what `logit` is, how it is obtained, would have been nice.
Is it the count of occurrences of the word following "the mouse ate the"?
- drexlspivey 4y agoLogits are the raw predictions of the model that you then need to normalize to turn into probabilities (that sum to 1). You can do that is with the softmax function.
- spywaregorilla 4y agoMachine learning models don't do well with math that has hard bounds at 1 and 0 (probability). So we use the log of the odds ratio instead. This function goes from negative infinity to positive infinity. And is mappable back to 0 and 1. So as you get closer to infinity log odds, you get closer to 100% certainty that your answer is A. It's just ln(p/(1-p)) under the hood. https://en.wikipedia.org/wiki/Logit https://en.wikipedia.org/wiki/Logit
- abecedarius 4y agoYeah -- the choice of temperature corresponds to the base of the logarithm. When you think of probabilities in the log-odds representation, a bayesian update means adding an increment. https://arbital.com/p/bayes_log_odds/ https://arbital.com/p/bayes_log_odds/
- albystein 4y agothis is not what language models typically output as logits though. perhaps the term now is being misused and it’s causing a lot of confusion
- albystein 4y agoBasically, logits are the raw outputs of the last linear layer of a neural network just before the softmax(for multi-class classification tasks) or the sigmoid(for binary classification tasks) is applied to the outputs.