> GPT-4 absolutely isn’t a Markov chain I mean it is technically a high order Markov chain, or at least every published GPT-N is so far. For example, GPT-3 is

> GPT-4 absolutely isn’t a Markov chain I mean it is technically a high order Markov chain, or at least every published GPT-N is so far. For example, GPT-3 is a 2048-order Markov chain over tokens. There may be some confusion because of the difference between the technical definition of Markov chain vs. how low-order Markov chains are usually implemented. Maybe you are used to Markov chains that have explicit transition matrices stored in memory and they are trained by counting the number of times a token appears after every prefix. That does give a Markov chain. But technically Markov chains aren't required to have their transition matrix be explicitly stored in memory, and they aren't required to be trained by simple counting (max likelihood transitions). They can have implicit transition matrices and be trained by gradient descent or whatever and still technically be Markov chains. I saw Karpathy weigh in on this and he said it's a Markov chain in the same way that a computer is a Markov chain. I guess his point is that if you have a high enough order Markov chain, the intuitions and connotations of Markov chains become less useful, and maybe 2048 order is high enough that it crosses that threshold.