4 ms·
The fact that LLMs generate text which is better than random is not something the authors need to spell out. Reducing the cross entropy between predicted and ta
by janalsncm 3y ago
The fact that LLMs generate text which is better than random is not something the authors need to spell out. Reducing the cross entropy between predicted and target distributions is the entire purpose of training.
Put another way: a flat probability distribution from an untrained network is equivalent to the random decoding you’re skeptical an LLM is better than. That’s not a criticism the authors’ peers would likely put forward.
- YeGoblynQueenne 3y ago>> Reducing the cross entropy between predicted and target distributions is the entire purpose of training. Nope. Test distribution. Not target distribution.
- janalsncm 3y agoI’m not sure why you felt the need to correct me. Target is the correct terminology. As a sanity check I even looked up the PyTorch doc which explicitly uses “target” in their example. https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html https://pytorch.org/docs/stable/generated/torch.nn.CrossEntr...
- YeGoblynQueenne 3y agoBecause the target distribution is not known, whatever PyTorch docs say. The test distribution is known and nobody knows what's its relation to the target distribution except a hope and a prayer.