It's just down to small context window. Once the context window is big enough to fit entire examples in the training data, then it should be trivially solvable

It's just down to small context window. Once the context window is big enough to fit entire examples in the training data, then it should be trivially solvable to train a model to do it. There are various finetuned models out there for conversations or story-telling, though they're quite small in terms of parameter count at the moment, but I don't see it as being fundamentally impossible.