3 ms·
Is it actually a next-word predictor? I thought the training loss is against a set of words, not just one.
by uh_uh 4y ago
Is it actually a next-word predictor? I thought the training loss is against a set of words, not just one.
- skybrian 4y agoI'm not sure what distinction you're getting at, but transformers do use "fill in the missing word" training and text generation chooses the next word (token actually) one at a time. Once it chooses a word, it doesn't go back.