I've actually had ChatGPT admit it was wrong by simply asking a question ("how is X achieved with what you described for Y"). It responded with "Oh, it's a grea

I've actually had ChatGPT admit it was wrong by simply asking a question ("how is X achieved with what you described for Y"). It responded with "Oh, it's a great question which highlights how I was wrong: this is what really happens...": but still, it couldn't get there without me understanding the underlying truth (it was about key exchange in a particular protocol that I knew little about, but I know about secure messaging in general), and it would easily confuse less experienced engineers with fully confident sounding explanation. For things I don't understand deeply, I can only look if it sounds plausible and realistic, but I can't have full trust. The "language" it uses when it's wrong is still just an extension of the token-completion it does (because that's what text contains in many of the online discussions etc).