3 ms·
Other approaches for long sequence lengths: - "Hyena Hierarchy: Towards Larger Convolutional Language Models" https://arxiv.org/pdf/2302.10866.pdf https://arxi
by smaddox 4y ago
Other approaches for long sequence lengths:
- "Hyena Hierarchy: Towards Larger Convolutional Language Models" https://arxiv.org/pdf/2302.10866.pdf https://arxiv.org/pdf/2302.10866.pdf
- "Resurrecting Recurrent Neural Networks for Long Sequences" https://arxiv.org/pdf/2303.06349.pdf https://arxiv.org/pdf/2303.06349.pdf
It's unclear (at least to me) if any of these scale as well as transformers at this point.
- cs702 4y agoThank you. The OP is from the authors of Hyena, and I agree it's unclear if it will scale as well as self-attention in transformers. The paper on RNNs looks very interesting. I like the fact that it keeps things simple and familiar. I've added it to my reading list.
- smaddox 4y agoYeah, just saw that. Sorry for the noise.
- cs702 4y agoNot a big deal. The paper on RNNs is exactly the sort of thing I was hoping someone would bring up here. Thank you!
- ilaksh 4y agoHow does that one compare to RWKV?
- smaddox 4y agoRWKV is random chance on the PathX problem. Both of those models are > 95% on it, if I remember correctly. I don't think Hyena can be run in RNN mode, though. But it _might_ scale better than the LRU model (second paper).