> RWKV combines the best features of RNNs and transformers. During training, we use the transformer type formulation of the architecture, which allows massive p

> RWKV combines the best features of RNNs and transformers. During training, we use the transformer type formulation of the architecture, which allows massive parallelization (with a sort of attention which scales linearly with the number of tokens). For inference, we use an equivalent formulation which works like an RNN with a state. This allows us to get the best of both worlds. > So we basically have a model which trains like a transformer, except that long context length is not expensive. And during inference, we need substantially less memory and can implicitly handle “infinite” context length (though in practice, the model might have a hard time generalizing to much longer context lengths than it saw during training). The tl:dr Linear Attention avoids the quadratic token scaling and is equivalent to specific instances of RNNs.