My guess is that ChatGPT does it because RLHF rewards strongly stated opinions, because that's what humans prefer. It's a kind of "sycophantic behavior" that r

My guess is that ChatGPT does it because RLHF rewards strongly stated opinions, because that's what humans prefer. It's a kind of "sycophantic behavior" that researchers have observed in these models (https://arxiv.org/abs/2310.13548 https://arxiv.org/abs/2310.13548)