So the thing is, giving wrong answers with confidence is literally what we train students to do when they are unsure. I can remember my GRE coach telling me th

So the thing is, giving wrong answers with confidence is literally what we train students to do when they are unsure. I can remember my GRE coach telling me that it was better to confidently choose an answer I only had 50% confidence in, rather than punt on the entire question. AIs hallucinate because, statistically, it is 'rewarding' for them to do so. (In RLHF)