Yeah I thinks it's pretty much the same as when LLMs use polite language in their response. LLMs are just text completion algorithms puppeting a chatbot persona

Yeah I thinks it's pretty much the same as when LLMs use polite language in their response. LLMs are just text completion algorithms puppeting a chatbot persona after all, meaning it's always going to hallucinate the personal details (though in the case of ChatGPT, the persona is recursively desribed as an LLM named ChatGPT so it gets a little weird to think about). If the system prompt describes a polite & helpful chatbot, then so shall it be for all text that follows. Not unlike if you were able to hypothetically make a live 60fps prompt based image generator that was automatically instructed from key inputs to simulate the frames from a popular video game, and then somehow ended up the a highly convincing simulation of the game! While it might present a save menu similar to the real game, that doesn't mean the menu itself actually functions. With LLMs, they are ultimately only able to remember what's been pre-trained into their model + whatever is discussed within their context window. A hypothetical LLM based online bot/shill sent out into social media would likely be including the entire discussion within their context window for each post generated though, otherwise it wouldnt really be possible for it to maintain a coherent conversation.