Thank you. The evidence I've seen suggests RWKV-LM does not work well with long sequences: https://news.ycombinator.com/item?id=35375616 https://news.ycombinat

Thank you. The evidence I've seen suggests RWKV-LM does not work well with long sequences: https://news.ycombinator.com/item?id=35375616 https://news.ycombinator.com/item?id=35375616 So far, every method I've come across that works well with long sequences does it by explicitly modeling (approximating) all possible pairwise relationships between tokens. In the case of attention, that means modeling n×n, or n², pairwise relationships. In the case of that dynamic routing method, it means modeling n×m pairwise relationships, where n and m are the number of tokens in the input and output sequences. I suspect that the key reason why that routing method works for long sentences is because m can be much smaller than n: Routing 1M tokens to 100 tokens requires ~10,000x less compute than applying attention over 1M tokens for each of 1M tokens. If the work described by the OP succeeds in bringing computational cost down from O(n²) to O(n log n) while still explicitly modeling (approximating) all n² pairwise relationships, that would be a huge win.