Recurrent models like RWKV should theoretically allow for unbounded context size. The problem is training them, which requires looking at a lot of long contexts

Recurrent models like RWKV should theoretically allow for unbounded context size. The problem is training them, which requires looking at a lot of long contexts and which isn't well supported by the RWKV "trains like a transformer, runs like an RNN" model.