For better or worse in the current age of the internet prose is a good first pass filter for quality. Someone arguing incoherently is seen as less believable.

For better or worse in the current age of the internet prose is a good first pass filter for quality. Someone arguing incoherently is seen as less believable. Unfortunately the prose for these chat models doesn't change based on how certain it is of the facts. So you can't tell based on how it is talking whether it is true or not. Certainly people online speak well while lying either intentionally or unintentionally but usually well intentioned people will coach things they aren't as certain about helping to paint a more accurate picture. I haven't taken a deep dive on the latest models but historically most AI haven't worried about "facts" as much as associating speech patterns. It knows how to talk about facts because other people have done so in the past kind of thing. This means you need to patch in arbitrary rules to reintroduce some semblance of truth to the outputs which isn't an easy task. False training is a whole different area IMO. Especially when there is a difference between responding to a particular user and responding to everyone based on new information.