You can easily fine-tune for more context and indeed he does. There are at least 8192 ctx versions out now (started with 1024) and he plans to go all the way to

You can easily fine-tune for more context and indeed he does. There are at least 8192 ctx versions out now (started with 1024) and he plans to go all the way to at least 16k