3 ms·

Also fair, but then the "current" state would also be a long window/sequence. Maybe that interpretation is valid if you look at the activations inside the netwo

Also fair, but then the "current" state would also be a long window/sequence. Maybe that interpretation is valid if you look at the activations inside the network, but I wouldn't know about that.

PartiallyTyped 4y ago
Yes, the state for both is a long window / sequence. Under this view, for the transformer we do not need to compute anything for the previous tokens as due to the causal nature of the model, the tokens at [0, ... N-1] are oblivious to the token N. For token N we can use the previous computations since they do not change.