4 ms·
> The context length for these models is 4096 tokens. !!! And I was excited that llama gave us 2048!!
by andy_xor_andrew 4y ago
> The context length for these models is 4096 tokens.
!!! And I was excited that llama gave us 2048!!
- MacsHeadroom 4y agoRumor is RedPajama is going to have upwards of 60k token context by using Hyena: https://arxiv.org/abs/2302.10866 https://arxiv.org/abs/2302.10866 But it's just a rumor. We'll see.