I don't buy this. I don't think it's reasonable to set up the task such that the "fact" can be distinguished from the background without knowing the question.

I don't buy this. I don't think it's reasonable to set up the task such that the "fact" can be distinguished from the background without knowing the question. For example, if your fact is always "[name] went to the [room]", and question is always "where is [name]?" the background text should be other sentences of the same form about different names. If your background text is instead drawn from a different distribution (as in this paper), then the model can simply cheat by identifying the fact (which constitutes only a few bits of information) and ignoring everything else. That isn't scaling to 1M tokens - the model should be expected to answer questions about any text within its context window.