3 ms·
> We propose a novel model architecture, Receptance Weighted Key Value (RWKV), that combines the efficient parallelizable training of Transformers with the effi
by RC_ITR 3y ago
> We propose a novel model architecture, Receptance Weighted Key Value (RWKV), that combines the efficient parallelizable training of Transformers with the efficient inference of RNNs.
Just to be clear to everyone: this is “use attention to train parameters, use recurrence for inference”
It’s a very cool idea and I hope we get more interesting approaches to inference, but attention is here to stay for training.
- matteogrella 3y agoAs one of the authors, I'd like to clarify: the equations of the RWKV model enable computational parallelization, provided that the sequence is predetermined. This parallelization occurs during both the training and inference stages, specifically during the prompt reading process (consider it an "encoding"), right before the generation (or decoding phase).
- RC_ITR 3y agoHow can something recurrent be parallelized? > the equations of the RWKV model enable computational parallelization, provided that the sequence is predetermined. And sure, this is the core concept of self attention, no?