3 ms·
It's a next-word predictor like a Markov chain, but a Markov chain couldn't do all the things ChatGPT does. ChatGPT has learned lots of syntax-level patterns pr
by skybrian 4y ago
It's a next-word predictor like a Markov chain, but a Markov chain couldn't do all the things ChatGPT does. ChatGPT has learned lots of syntax-level patterns pretty well.
- uh_uh 4y agoIs it actually a next-word predictor? I thought the training loss is against a set of words, not just one.
- skybrian 4y agoI'm not sure what distinction you're getting at, but transformers do use "fill in the missing word" training and text generation chooses the next word (token actually) one at a time. Once it chooses a word, it doesn't go back.