3 ms·

> though in practice, the model might have a hard time generalizing to much longer context lengths than it saw during training What's the benefit of RWKV then?

by ipsum2 4y ago

> though in practice, the model might have a hard time generalizing to much longer context lengths than it saw during training What's the benefit of RWKV then? The results are practically identical to transformers.

sebzim4500 4y ago
It is way cheaper to run inference on commodity hardware, because you don't have to keep track of the activations on all the previous tokens (only the 3 previous ones IIRC).
PartiallyTyped 4y ago
> RWKV combines the best features of RNNs and transformers. During training, we use the transformer type formulation of the architecture, which allows massive parallelization (with a sort of attention which scales linearly with the number of tokens). For inference, we use an equivalent formulation which works like an RNN with a state. This allows us to get the best of both worlds. > So we basically have a model which trains like a transformer, except that long context length is not expensive. And during inference, we need substantially less memory and can implicitly handle “infinite” context length (though in practice, the model might have a hard time generalizing to much longer context lengths than it saw during training). The tl:dr Linear Attention avoids the quadratic token scaling and is equivalent to specific instances of RNNs.
famouswaffles 4y ago
You can easily fine-tune for more context and indeed he does. There are at least 8192 ctx versions out now (started with 1024) and he plans to go all the way to at least 16k
vagabund 4y ago
- from O(N²) to O(N) complexity during training wrt context length - only need the hidden state at position t to compute t+1, i.e. run inference locally on edge devices - fine-tunable to longer context lengths than seen in pre-training If it continues to scale as well as it has so far it's pretty huge.

3 ms·

> though in practice, the model might have a hard time generalizing to much longer context lengths than it saw during training What's the benefit of RWKV then?

by ipsum2 4y ago

sebzim4500 4y ago
It is way cheaper to run inference on commodity hardware, because you don't have to keep track of the activations on all the previous tokens (only the 3 previous ones IIRC).
PartiallyTyped 4y ago
> RWKV combines the best features of RNNs and transformers. During training, we use the transformer type formulation of the architecture, which allows massive parallelization (with a sort of attention which scales linearly with the number of tokens). For inference, we use an equivalent formulation which works like an RNN with a state. This allows us to get the best of both worlds. > So we basically have a model which trains like a transformer, except that long context length is not expensive. And during inference, we need substantially less memory and can implicitly handle “infinite” context length (though in practice, the model might have a hard time generalizing to much longer context lengths than it saw during training). The tl:dr Linear Attention avoids the quadratic token scaling and is equivalent to specific instances of RNNs.
famouswaffles 4y ago
You can easily fine-tune for more context and indeed he does. There are at least 8192 ctx versions out now (started with 1024) and he plans to go all the way to at least 16k
vagabund 4y ago
- from O(N²) to O(N) complexity during training wrt context length - only need the hidden state at position t to compute t+1, i.e. run inference locally on edge devices - fine-tunable to longer context lengths than seen in pre-training If it continues to scale as well as it has so far it's pretty huge.