4 ms·
Agreed. If LLMs did predict only a single word ahead, they would repeat themselves, if not in the next sentence then at least in the next paragraph. LLMs cert
by randcraw 3y ago
Agreed. If LLMs did predict only a single word ahead, they would repeat themselves, if not in the next sentence then at least in the next paragraph. LLMs certainly retain memory of what they’ve already said. They are most certainly NOT Markov-1 models, despite the popularization of that oversimplification.
- simonw 3y agoThey really do compute just a single token at a time, but the input to that decision is the number of tokens in the context window - so that's usually around 8,000 tokens (approximately 6000 words). You can experiment with smaller LLMs on your own devices to get a better feeling for how that works.
- uoaei 3y agoI don't think anyone's saying they're Markov chains with context length 1. In all the critiques I've read no one's even come close to articulating that. But they are Markov chains with context length N. They're approximating (implicit) Markov transition matrices through a fancier method of computation but they still are just generating one token at a time.
- torginus 3y agoI thought it was common knowledge that LLMs have a context of X tokens (8192 in the case of GPT4) and use those as input to predict the next token probabilities. Still repetition is a problem, that's why they introduce a penalty for generating the exact same token, so when picking the actual token to output from a list of probabilities, they penalize the ones already in the context so its less likely to repeat itself.