I would at least start to be convinced that this is NOT the case if I ever saw it respond with something like "I actually don't know the answer to that query" o

I would at least start to be convinced that this is NOT the case if I ever saw it respond with something like "I actually don't know the answer to that query" or "as far as I'm aware, there is no way to do the thing you asked". These are responses that would have shown up innumerable times in it's training data and make perfect sense as "the most logical next set of tokens", and yet it will never say them. Instead it will hallucinate something that sounds nearly indistinguishable from fact, but turns out to be a total fabrication. If all its doing is returning the next most logical set of tokens, and the training data it was based on included a non-trivial number of examples where one party in the conversation didn't have a clear answer, then there's no reason GPT-4 should be so averse to simply saying "yeah, I dunno bro". The only logical reason I can see is that it's "learned" that it's more likely to receive the positive feedback signal when it makes up convincing bullshit, than if it states that it doesn't have an answer. EDIT: To be clear, I mean it telling me it doesn't know something BEFORE hallucinating something incorrect and being caught out on it by me. It will admit that it lied, AFTER being caught, but it will never (in my experience) state that it doesn't have an answer for something upfront, and will instead default to hallucinating. Also - even when it does admit to lying, it will often then correct itself with an equally convincing, but often just as untrue "correction" to its original lie. Honestly, anyone who wants to learn how to gaslight people just needs to spend a decent amount of time around GPT-4.