3 ms·
It's not a Markov chain because by definition a Markov chain only looks at the previous word. ChatGPT looks at a long sequence of previous words. But the genera
by nerdponx 4y ago
It's not a Markov chain because by definition a Markov chain only looks at the previous word. ChatGPT looks at a long sequence of previous words. But the general idea is still broadly the same.
- xwdv 4y agoHence why you have to squint.
- Accujack 4y agoActually, it looks at all the meanings of the tokens within its window.
- PartiallyTyped 4y agoThat's not correct. In a Markov chain, the current state is a sufficient characteristic of the future. For all intents and purposes you can create a state with sufficiently long history to look at a long sequence of words.
- nerdponx 4y agoAlso fair, but then the "current" state would also be a long window/sequence. Maybe that interpretation is valid if you look at the activations inside the network, but I wouldn't know about that.
- PartiallyTyped 4y agoYes, the state for both is a long window / sequence. Under this view, for the transformer we do not need to compute anything for the previous tokens as due to the causal nature of the model, the tokens at [0, ... N-1] are oblivious to the token N. For token N we can use the previous computations since they do not change.