My guess is that a low-probability token corresponds as much with creativity (it's saying something in an unusual way) as truth, since it's at the word level. W

My guess is that a low-probability token corresponds as much with creativity (it's saying something in an unusual way) as truth, since it's at the word level. Words aren't normally true or false, sentences are. Do LLM's represent their confidence in what they're saying anywhere? I don't think anyone knows. Another mystery. But another problem is that confidence is a character attribute, not a writer attribute. If you ask an LLM to imitate Richard Feynman it's going to write pretty confidently about physics, despite not knowing as much as him about physics. This is the equivalent of giving your RPG character high intelligence on the character sheet. Doesn't make you smart! To make an LLM express confidence consistent with its actual knowledge, it would need to have good self-knowledge and actually use it. So far this doesn't happen automatically. Instead, OpenAI uses reinforcement learning based on what the people at OpenAI think the LLM can do. So that's why it sometimes refuses to answer for some kinds of questions. That has the most effect on the default character, the "helpful AI assistant". Any other characters you ask for will likely have poorer self-knowledge. I've read that, mysteriously, more training does make bigger LLM's better calibrated, but the reinforcement learning makes it worse again.