4 ms·
The comment in parenthesis mentions "they're not derived from a probability space" [1]. I don't know about probability spaces nor softmax to know what part of a
by rhdunn 6mo ago
The comment in parenthesis mentions "they're not derived from a probability space" [1]. I don't know about probability spaces nor softmax to know what part of a probability space this is missing compared to other probability distributions, nor how other probability distributions satisfy probability spaces.
[1] https://en.wikipedia.org/wiki/Probability_space https://en.wikipedia.org/wiki/Probability_space
- ghkbrew 6mo agoSounds like they're saying that since the distribution doesn't come from measuring or calculating the probability of something, it has the form of a probability distribution but isn't really one. Like saying 5 feet is a height that a person can have, but since I just made up that number it's not actually a person's height.
- mojosmojo 6mo agoiirc, there is a bunch of formal machinery you need to define probability distributions for situations such as infinite outcomes (eg what is the probability that a random real number between 0 and 10 is less than 3?)
- jmalicki 6mo agoThe soft max is the probability of the next token being whatever in the training data conditioned on the inputs. The author just doesn't know that apparently and thinks it was an arbitrary choice. The author's essay on the sigmoid similarly lacks the deep understanding that it comes from somewhere and isn't an arbitrary choice.
- canjobear 6mo agoThe softmax, after the network has been trained, yields an estimate of the probability in the training data, but it is not that probability itself.
- jmalicki 6mo agoWhich models are not trained with the log softmax as the loss function?
- canjobear 6mo agoSoftmax isn't a loss function. It is used to transform model outputs into positive numbers that sum to 1, so that they can be interpreted as probabilities, and then those numbers are passed into (typically) the cross entropy loss function. I think you mean, which models are trained using some function other than softmax to transform the model outputs. There are a number of alternatives to softmax, such as the ones described here https://www.emergentmind.com/topics/sparsemax https://www.emergentmind.com/topics/sparsemax
- jmalicki 6mo agoThe cross entropy loss function is softmax. They are one and the same.
- canjobear 6mo agoThey’re not. Cross entropy loss is E[-log q] where q is a probability. You could convert the model outputs x into probabilities using some other function like q = 1/Z x^2, and compute cross entropy loss just fine.
- jmalicki 6mo agoBehold the softmax: https://docs.pytorch.org/docs/2.11/generated/torch.nn.CrossEntropyLoss.html https://docs.pytorch.org/docs/2.11/generated/torch.nn.CrossE...
- canjobear 6mo agoBehold the actual definition of cross entropy: https://en.wikipedia.org/wiki/Cross-entropy https://en.wikipedia.org/wiki/Cross-entropy It's true that the PyTorch API conflates cross entropy and softmax, but they are separate concepts.