If you've got 20 tokens of query at the start and then 200 tokens of text data that it's querying, it seems really impressive that it's able to work out (via in

If you've got 20 tokens of query at the start and then 200 tokens of text data that it's querying, it seems really impressive that it's able to work out (via instruct tuning) to answer the query rather than continue the text data. A continuation of the text data is the actual most likely next token. I don't know about the super large contexts but you can also just make the text data clearly delimited instead of putting the query at the end, so that "predict the next token" isn't fighting the instruction-following training