Rumor is RedPajama is going to have upwards of 60k token context by using Hyena: https://arxiv.org/abs/2302.10866 https://arxiv.org/abs/2302.10866 But it's jus

Rumor is RedPajama is going to have upwards of 60k token context by using Hyena: https://arxiv.org/abs/2302.10866 https://arxiv.org/abs/2302.10866 But it's just a rumor. We'll see.