4 ms·
Too early to tell, but this looks like it could both simplify implementation and speed-up execution of linear RNNs like RWKV, Mamba, retnet, etc.
by cs702 3y ago
Too early to tell, but this looks like it could both simplify implementation and speed-up execution of linear RNNs like RWKV, Mamba, retnet, etc.
- jimsimmons 3y agoThey already use closed form summations. That's their whole point
- cs702 3y agoYes, but this uses only two plan-vanilla parallel scans, shown as a* and b* in the pdf. In the sample code, that's two calls to PyTorch's built-in API. That looks very efficient to me, almost like it shouldn't be possible. But I tested the sample code[a] and it works. Not an expert on this, but I don't think I've ever seen that before. Have you? [a] https://github.com/glassroom/heinsen_sequence https://github.com/glassroom/heinsen_sequence
- jimsimmons 3y agoThere's a GitHub issue on the repo that explores this!
- cs702 3y agoThank you. It seems only testing will answer the question.