4 ms·
That's not correct. In a Markov chain, the current state is a sufficient characteristic of the future. For all intents and purposes you can create a state with
by PartiallyTyped 4y ago
That's not correct. In a Markov chain, the current state is a sufficient characteristic of the future. For all intents and purposes you can create a state with sufficiently long history to look at a long sequence of words.
- nerdponx 4y agoAlso fair, but then the "current" state would also be a long window/sequence. Maybe that interpretation is valid if you look at the activations inside the network, but I wouldn't know about that.
- PartiallyTyped 4y agoYes, the state for both is a long window / sequence. Under this view, for the transformer we do not need to compute anything for the previous tokens as due to the causal nature of the model, the tokens at [0, ... N-1] are oblivious to the token N. For token N we can use the previous computations since they do not change.