3 ms·
> RWKV The current versions of RWKV slowly go insane when exposed to sequences that are too long, because the state slowly diverges over time as you increase p
by LoganDark 3y ago
> RWKV
The current versions of RWKV slowly go insane when exposed to sequences that are too long, because the state slowly diverges over time as you increase past the context length of the training session. They are experimenting with ways to avoid this though: https://github.com/Blealtan/RWKV-LM-LoRA/tree/dev-infctx https://github.com/Blealtan/RWKV-LM-LoRA/tree/dev-infctx
- lostmsu 3y agoCan you share more details about the divergence?