The thing to understand about ChatGPT is that at its core it isn't a question-answering machine, it's a text-continuation machine which by pure coincidence ofte

The thing to understand about ChatGPT is that at its core it isn't a question-answering machine, it's a text-continuation machine which by pure coincidence often continues a question with its answer. And it's learning primarily from data from people posting stuff on the internet (and only very rarely seeing actual conversations) so it has learned that "human language" pretty much never involves people saying that they don't know stuff, but does involve a LOT of assertive speculating and convinced 'preaching' even when saying untrue things. Like, imagine seeing a human-written a blog post for some click-bait site titled "Venus Williams eighth grand slam victory" or "The 10 secret endings for Inside" - ChatGPT will write pretty much the same thing as that human would. That blog post won't say "ah, I have no idea about these endings, I haven't seen them in my games", it would confidently repeat various rumors and if the author would invent some new fan theories, then they would often be stated in an assertive manner as fact. Furthermore, when doing its final reinforcement learning fine-tuning on top of the core GPT "3.5" model, it almost certainly was asked about some stuff which was simply more recent than the data used in training the core model - I mean, it can't know what happened last month - but it would be penalized if it would assert that things are the exact same as they were when the training data ended and reject things the user explicitly states that contradict its (inevitably outdated) knowledge. For all it knows, Williams has won 8 grand slams, simply the eighth one happened in the year after ChatGPT "stopped reading newspapers", so if a discussion mentions the eighth victory it has to assume that it is real, otherwise it would fail in so many other scenarios - e.g. refusing to acknowledge that some country has elected a different government or that a war has started.