3 ms·
Can someone with a better understanding than I have comment about the relationship between these results and this paper: https://arxiv.org/abs/2109.09115 https:
by skepticATX 4y ago
Can someone with a better understanding than I have comment about the relationship between these results and this paper: https://arxiv.org/abs/2109.09115 https://arxiv.org/abs/2109.09115, which seems to demonstrate that a longer context length has diminishing returns?
- rsfern 4y agoI’m not deeply familiar with all these papers, but two things stand out to me The model architectures are different, and in the very latest paper they scale these not-transformer models to sequence length of 64k, where the paper you linked only considers up to 8k