Remember: LLMs are basically trained to be really gifted improv actors. That's what all that "guess the next token" training boils down to. Once that base of "i

Remember: LLMs are basically trained to be really gifted improv actors. That's what all that "guess the next token" training boils down to. Once that base of "improv" is established, they're then trained to play a specific role, "the helpful and harmless assistant". Somehow this produces useful output much of the time, as ridiculous as it seems. But whenever almost anything goes wrong, LLMs fall back to improv games. It's like you're talking to the cast of "Whose Line is it Anyway?" You set the scene, and the model tries to play along. So a case like this proves almost nothing. If you ask an improv actor "What's the last question you were asked?", then they're just going to make something up. So will these models. If you give a model a sentence with 2 grammatical errors, and ask it to find 3, it will usually make up a third error. If it doesn't know the answer to a question, it will likely hallucinate. GPT-4 is a little better at resisting the urge to make things up.