3 ms·
Thank you. The OP is from the authors of Hyena, and I agree it's unclear if it will scale as well as self-attention in transformers. The paper on RNNs looks ver
by cs702 4y ago
Thank you. The OP is from the authors of Hyena, and I agree it's unclear if it will scale as well as self-attention in transformers. The paper on RNNs looks very interesting. I like the fact that it keeps things simple and familiar. I've added it to my reading list.
- smaddox 4y agoYeah, just saw that. Sorry for the noise.
- cs702 4y agoNot a big deal. The paper on RNNs is exactly the sort of thing I was hoping someone would bring up here. Thank you!
- ilaksh 4y agoHow does that one compare to RWKV?
- smaddox 4y agoRWKV is random chance on the PathX problem. Both of those models are > 95% on it, if I remember correctly. I don't think Hyena can be run in RNN mode, though. But it _might_ scale better than the LRU model (second paper).