4 ms·
I guess if you squint, it kinda is, in the sense that it generates one token at a time.
by TechBro8615 4y ago
I guess if you squint, it kinda is, in the sense that it generates one token at a time.
- nerdponx 4y agoIt's not a Markov chain because by definition a Markov chain only looks at the previous word. ChatGPT looks at a long sequence of previous words. But the general idea is still broadly the same.
- xwdv 4y agoHence why you have to squint.
- Accujack 4y agoActually, it looks at all the meanings of the tokens within its window.
- PartiallyTyped 4y agoThat's not correct. In a Markov chain, the current state is a sufficient characteristic of the future. For all intents and purposes you can create a state with sufficiently long history to look at a long sequence of words.
- nerdponx 4y agoAlso fair, but then the "current" state would also be a long window/sequence. Maybe that interpretation is valid if you look at the activations inside the network, but I wouldn't know about that.
- PartiallyTyped 4y agoYes, the state for both is a long window / sequence. Under this view, for the transformer we do not need to compute anything for the previous tokens as due to the causal nature of the model, the tokens at [0, ... N-1] are oblivious to the token N. For token N we can use the previous computations since they do not change.
- theGnuMe 4y agoIt is in the sense that it is Markov chain with an 8k token memory and the "MCMC step" is the DNN.