Yeah, exactly. Chat models ingest the entire conversation every time they are queried, so the context (and the training data) is the whole Q+A. Hence sometime

Yeah, exactly. Chat models ingest the entire conversation every time they are queried, so the context (and the training data) is the whole Q+A. Hence sometimes the llm thinks a particular answer is "done" without emitting the proper end token, so they keep trying to complete the context... Which is another question a user would ask from its training dataset. You can force this behavior in llama chat finetunes by forcing the model to generate infinitely. It will keep on generating questions + answers.