There is still a significant problem with sycophancy in the way they're training these models. I have a horrible feeling it might be a re-run of the "engagement

There is still a significant problem with sycophancy in the way they're training these models. I have a horrible feeling it might be a re-run of the "engagement vs healthy behaviour" we saw with social media, i.e. profit before ethics. Or best-case a side-effect of over-reliance on RLHF training. The behaviours that get people to use ChatGPT more aren't necessarily the ones that are best for truthful, helpful responses. TL;DR - people tend to click the "thumbs up" icon on the A/B tests on the more sycophantic replies. This results in a feedback loop.