3 ms·
Interesting. I was skeptical about some of their claims regarding longer context, since it's been my experience that these models just get lost after enough of
by phillipcarter 3y ago
Interesting. I was skeptical about some of their claims regarding longer context, since it's been my experience that these models just get lost after enough of it.
- msp26 3y agoYeah, degraded performance on long contexts has been observed in plenty of other models [https://arxiv.org/abs/2307.03172 https://arxiv.org/abs/2307.03172] so I was cautious too. Unfortunately I don't have access to 4-32k. I would have liked to test that out too.