2 ms·
Isn't any LLM mathematically a Markov chain, such that the current state includes the context of the last (finite) n tokens?
by falcor84 4y ago
Isn't any LLM mathematically a Markov chain, such that the current state includes the context of the last (finite) n tokens?
- seabass-labrax 4y agoThe distinctive aspect of the 'transformer' family of neural networks is that they incorporate 'attention', which is a model that identifies which parts of the input are critical to its meaning. You're correct in that modern transformer models are essentially Markov chains, but the function that derives the probabilities (of which the attention heads are part of) is imbued with 'understanding' of related concepts during the training process. In contrast, a traditional Markov chain text generator might have a probability function that only takes into account the frequency of the n-grams in the training data, and so produces superficially coherent (but mostly meaningless) output.
- skybrian 4y agoThis is like saying all computers are state machines because they have finite amounts of memory and disk space. It's sort of true, but it's not a useful mathematical model. With a Markov chain, you're assuming a state machine where each state has independent probabilities on outgoing edges. As the number of states gets larger, you have fewer training samples for each state. When n gets large enough, nearly all states have zero training samples; they've never been seen before. How do you estimate probabilities? Better to just say it's a stateless function of the input.