An attention layer is a kind of meta neural network whose weights depend on input. If you think of each token as a neuron and the attention matrix as defining t

An attention layer is a kind of meta neural network whose weights depend on input. If you think of each token as a neuron and the attention matrix as defining the weights, then extra tokens simply mean a wider network.