LLMs simply answer the question: given this corpus of text you've read so far, what's the most probable next word? If half of the training dataset says the next

LLMs simply answer the question: given this corpus of text you've read so far, what's the most probable next word? If half of the training dataset says the next word in similar conditions is A, and the other half says it's B, then LLMs will be "uncertain" whether it's A or B, but LLMs will be oblivious to the fact that both A and B are wrong, because most of the training dataset was LLM-generated slop. The current stage of extracting the essense of reason from LLMs feels a lot like attempts to extract gold from iron in the medieval ages.