5 ms·
> The claim that LLMs are nothing more than fancy expensive markov chains is a dangerous oversimplification or, in other words, wrong. But they are fancy expen
by Jensson 4y ago
> The claim that LLMs are nothing more than fancy expensive markov chains is a dangerous oversimplification or, in other words, wrong.
But they are fancy expensive Markov chains. The learning here is that fancy expensive Markov chains can do a lot, not that the statement is wrong.
- sebzim4500 4y agoThey are fancy markov chains in the same way that your computer is a finite state machine. It is technically true, but it is an extremely unhelpful way of thinking about them.
- taf2 4y agoUntil it is helpful when you need to control the computer to process a sequence of data in a finite way similar to say a regular expression… #thisisfun
- taeric 4y agoUnhelpful is very different from wrong. Just as simplified does not mean reduced. It can align and help understanding, even if incomplete.
- culi 4y agoCorrect is very different from helpful Just as munged does not mean increased. It can contextualize understanding, even if incomplete
- inimino 4y agoTell that to Alan Turing.
- sebzim4500 4y agoAlan Turing wouldn't think of a modern computer as a finite state machine either, he would think of it as a very good approximation to a turing machine.
- inimino 4y agoWhich is a finite state machine attached to an infinite tape.
- famouswaffles 4y agoThey really aren't lol
- Jensson 4y agoLook up the definition for Markov chain.
- akprasad 4y agoA Markov chain satisfies the Markov property [1] in which the probability of an event is conditioned only on the previous event. Regardless of whether an LLM event is a character or a word, an LLM certainly does not follow a Markov process. [1]: https://en.wikipedia.org/wiki/Markov_property https://en.wikipedia.org/wiki/Markov_property
- inimino 4y agoThe state space of a Markov process must be fixed but is otherwise unspecified, which the LLM satisfies, so you are technically incorrect.
- kelipso 4y agoIf you consider the entire input for the language model as the state, and consider the output to be the input concatenated with the next token, then it's a markov chain. But that's only if you don't use BEAM search, which LLMs do use. If you use BEAM search with beam=4 (typical for LLMs), for example, then it does a tree search where it keeps track of the top 4 highest probability outputs, then I guess it's not a markov chain process anymore but it just uses a markov chain.
- bluejay2387 4y ago"All models are wrong, some are useful". You could also look at them as a highly compressed PGM representing every possible conversation you could have in every language the model knows that has been trimmed to only the highly probable paths through training. In that view word prediction is really navigating nodes and this would explain their ability to enforce global constraints as well as they do :) There isn't going to be one "correct" way to view them and different perspectives might offer different insights.
- rcarmo 4y agoNot quite, I didn’t go into the details, but the analogy is flawed in a few ways: https://taoofmac.com/space/blog/2023/03/18/0140 https://taoofmac.com/space/blog/2023/03/18/0140