There is already another breakthrough on top of RoPE that promises even better scaling to larger contexts (based on NTK-awareness) - see my submission here: ht

There is already another breakthrough on top of RoPE that promises even better scaling to larger contexts (based on NTK-awareness) - see my submission here: https://news.ycombinator.com/item?id=36533816 https://news.ycombinator.com/item?id=36533816 The lmsys team is already looking into it, expect further progress soon. It's really amazing to follow this progress in realtime, even if the pace can sometimes feel exhausting!