3 ms·
1. An LLM is absolutely a Markov chain. That is not meant to dismiss that they are a remarkable feat of generating and compressing the representation of really
by asdlfaglkj 1y ago
1. An LLM is absolutely a Markov chain. That is not meant to dismiss that they are a remarkable feat of generating and compressing the representation of really cool Markov chains.
2. Because they are Markov chains, their skill ceiling is bounded by whatever the skill ceiling of Markov chains happens to be. What LLMs demonstrate is that the skill ceiling of Markov chains is higher than previously understood.
3. If the skill ceiling of Markov chains is high enough, then one could take over some or all of someone's job.
- drdeca 1y agoI think there’s an equivocation being accidentally made between n-gram models, and Markov processes, because “Markov chain” is used to mean both things. N-gram models are not useful in many of the ways LLMs are. N-gram models are very limited. On the other hand, basically any process can be considered a Markov process if you make the state include enough. So, calling both the very-limited n-gram models and the nearly-unlimited Markov processes, by the same name “Markov chain” is just, super confusing.