5 ms·
> LLMs are text completion engines and can't really do math It blows my mind that there are people on HN that don't know this. LLMs are basically Markov Chain
by Sohcahtoa82 2y ago
> LLMs are text completion engines and can't really do math
It blows my mind that there are people on HN that don't know this.
LLMs are basically Markov Chains on a massive dose of steroids. It looks at the last $context_window tokens and decides what the next token should be. They just work differently in that a neural network creates more "fuzzy" matching than a Markov Chain, which is purely statistical.
- visarga 2y agoThat is RNNs. LLMs (usually transformers) are non-markovian.
- Jensson 2y ago> That is RNNs. LLMs (usually transformers) are non-markovian. How? They have a hard cut off context, and randomly generates next state from that. That is the definition of a markov chain. It is a markov chain with a pretty large state at every point, but still a markov chain.