I wouldn't say always. The way the "confidently wrong" answers make sense to me are in the context of asking people trick questions that seem to have an obvious

I wouldn't say always. The way the "confidently wrong" answers make sense to me are in the context of asking people trick questions that seem to have an obvious answer but actually require more deliberate thought to get right. What happens in that case is you are engaging the part of the brain that has simply memorized a fact or developed a quick but not always accurate heuristic which it then applies to the question that seems to meet the criteria and it results in a confidently wrong answer given by a real meat and bones human. LLMs right now are essentially a big collection of facts and heuristics with the ability to find the often right association between input and the applicable output. Everyone is working on building the other thing that does the careful step by step reasoning and error checking, the so called System 2, and if brains can tell us anything about the implementation of such a system it is that it's made of the same stuff that System 1 is but arranged and controlled in such a way that it produces more reliable answers. *edit: I should specify that when I say LLMs are a collection of facts and heuristics I mean they are a collection of those things encoded as language which itself has been encoded as vectors of floats which in turn have modified the weights of the network to produce yet another encoding. I don't mean that the facts and heuristics are stored in a lookup table or as procedures.