I think "What you want to hear" is in the same spirit as an undergraduate attempting to regurgitate half-baked knowledge into an answer that will get him the gr

I think "What you want to hear" is in the same spirit as an undergraduate attempting to regurgitate half-baked knowledge into an answer that will get him the grade. Maybe a better rephrasing would be "ChatGPT has been trained to give answers to questions in a clear, confident manner, regardless of the content" From the announcement post: https://openai.com/blog/chatgpt https://openai.com/blog/chatgpt > We trained this model using Reinforcement Learning from Human Feedback (RLHF), using the same methods as InstructGPT, but with slight differences in the data collection setup. We trained an initial model using supervised fine-tuning: human AI trainers provided conversations in which they played both sides—the user and an AI assistant. So I agree with OP; it's been trained to give answers that sound plausible but not necessarily correct. It's even mentioned in the "Limitations" section at the bottom of the blog post. > ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers. Fixing this issue is challenging, as: (1) during RL training, there’s currently no source of truth; (2) training the model to be more cautious causes it to decline questions that it can answer correctly; and (3) supervised training misleads the model because the ideal answer depends on what the model knows, rather than what the human demonstrator knows.