3 ms·
>Is GPT simply a Markov chain? Yes, but much more efficiently stored and looked up than just a massive table. The way its very large space is encoded in a smal
by lamp987 4y ago
>Is GPT simply a Markov chain?
Yes, but much more efficiently stored and looked up than just a massive table. The way its very large space is encoded in a small amount of memory is what makes GPT so interesting.
- jonny_eh 4y agoA neural net instead of a table?
- nuclearnice3 4y agoA markov chain maps states to states. You move from word "I" to word "will" or "can" or "want" The transition probabilities could be stored in a huge table. You have a word and the probability of each other word. Draw a number and hop. [1,2] Alternatively you could ask a neural net for the next state. Pass it the current token or the last 4000 token. [1] https://towardsdatascience.com/text-generation-with-markov-chains-an-introduction-to-using-markovify-742e6680dc33 https://towardsdatascience.com/text-generation-with-markov-c... [2] https://crowintelligence.org/2021/05/27/a-simple-markov-chain-text-generator-in-python/ https://crowintelligence.org/2021/05/27/a-simple-markov-chai...
- oezi 4y agoGPT is not architecturally implemented as a Markov chain. But given enough time one could model it as one.