It tracks with the way LLMs work. From what I understand, during training, the LLMs build a correlation between words, a process after which their weight networ

It tracks with the way LLMs work. From what I understand, during training, the LLMs build a correlation between words, a process after which their weight network may be configured in a way that superficially resemble actual reality. Then for whatever input this LLM takes in, it uses its "reasoning" based on this configuration to spit out a response. A bigger LLM has more parameters to model the world and may have a better performance. But I guess even 1 trillion parameters is still not enough to model the actual reality with enough depth. So if their internal understanding of the world is inherently flawed, you can't expect them to be able to spot their own faults. To be fair, this is the same for human. I am sure we all have that experience trying to tell someone where they are wrong and had a hard time. But with a human, we can update our internal world model on the fly. An LLM at runtime can't do that. It needs to be retrained.