I've been using confabulate privately for a while about this phenomenon, because if you know anything about dementia (my grandmother had it - sitting in the car

I've been using confabulate privately for a while about this phenomenon, because if you know anything about dementia (my grandmother had it - sitting in the car with her you got used to the "covers" feeling like chatbot answer dodges to some extent) or split brain experiments, then it's quite a remarkable phenomenon that does feel a lot more like what an LLM is doing. Hallucinate generally implies experiencing sensory inputs which aren't there, but having a rational reaction to them - in the popular psyche though, people who hallucinate are still "crazy" - unfairly so, because you can function normally with hallucinations under quite a number of circumstances (i.e. I remember someone saying that they realized that if they saw people without faces, it was fine because they weren't really there - apparently quite common, since face recognition is a different part of the brain). Basically: reacting to invalid sensory inputs implies rational behavior, but to non-real inputs. Confabulation on the other hand is different because it's closer to "trying to rationalize an autonomous behavior without knowledge of the sensory inputs causing it" (maybe, it's a bizarre phenomenon). In split-brain patients in lab settings, it manifests as being unable to answer a question about why you're having a physiological response to imagery which is being shown only to the right-side of the brain, whereas language is processed on the left - but rather then be confused, people will apparently make something up that they're unaware is actually a "lie" (in quotes because, well, they're not lying - as far as we know there's no intent or even knowledge that is a lie). Where this leaves us with LLMs I don't really know, because it feels like an imperfect descriptor: except perhaps with the context that LLMs are fairly limited networks, and so to some extent the whole "being wrong" phenonmenon is in fact just a failure of the attention mechanism to be able to draw the right data together.