3 ms·
They’re not. Cross entropy loss is E[-log q] where q is a probability. You could convert the model outputs x into probabilities using some other function like q
by canjobear 6mo ago
They’re not. Cross entropy loss is E[-log q] where q is a probability. You could convert the model outputs x into probabilities using some other function like q = 1/Z x^2, and compute cross entropy loss just fine.
- jmalicki 6mo agoBehold the softmax: https://docs.pytorch.org/docs/2.11/generated/torch.nn.CrossEntropyLoss.html https://docs.pytorch.org/docs/2.11/generated/torch.nn.CrossE...
- canjobear 6mo agoBehold the actual definition of cross entropy: https://en.wikipedia.org/wiki/Cross-entropy https://en.wikipedia.org/wiki/Cross-entropy It's true that the PyTorch API conflates cross entropy and softmax, but they are separate concepts.