3 ms·
Yes, like RWKV and Mamba this is a new generation of models that are more like big RNNs than pure transformers we have now
by VHRanger 3y ago
Yes, like RWKV and Mamba this is a new generation of models that are more like big RNNs than pure transformers we have now
- boywitharupee 3y agoand is Griffin a state space model?
- imjonse 3y agoNo, it's a combination of RNN and Transformer.
- michwilinski 3y agoI mean, SSMs are in fact under the hood RNNs
- VHRanger 3y agoAt the end of the day, either you carry around a hidden state, or you have a fixed window for autoregression. You can call hidden states "RNN-like" and autoregressive windows "transformer-like", but apart from those two core paradigms I don't know of other ways to do sequence modelling. Mamba/RWKV/Griffin are somewhere between those two extremes.
- stri8ed 3y agoIsn't that how previous models were, before the attention is all you need paper?