Because the model doesn't contain facts at any point - only words. The size of the LLMs is an easy way to currently demonstrate this - even if you took only th

Because the model doesn't contain facts at any point - only words. The size of the LLMs is an easy way to currently demonstrate this - even if you took only the factual statements from all its training material and compressed them, they'd be larger than the model. The model isn't magic thus it can't contain all those facts. The only way it can write a fact is if those words are simply the most likely completions and happen to be right. This means you'd be essentially training it randomly by selecting factual answers. You wouldn't be reinforcing that it gave you a correct fact, just whatever the sentence structure was that you judged to be factual. I think what would happen is that it would start to write very careful statements which would be more likely to be technically correct merely by not being wrong. For example, if you trained by asking questions like what year president George Washington was born it would quickly learn to stop guessing a year because that's got a low probability of being right and those statements would get trained out. It'd probably write something like "Before 1760" because that statement has a much higher likelihood of being right even if it's a less useful answer.