There are techniques to extend the context window via fine tuning. The authors claim this method was used to extend Llama 2 to 128k: https://github.com/jquesne

There are techniques to extend the context window via fine tuning. The authors claim this method was used to extend Llama 2 to 128k: https://github.com/jquesnelle/yarn https://github.com/jquesnelle/yarn