3 ms·

>> If an LLM was trained to always say "I don't know", that'd be a useless LLM, so they're trained to NOT say that, even when they don't actually know. Do LLMs

>> If an LLM was trained to always say "I don't know", that'd be a useless LLM, so they're trained to NOT say that, even when they don't actually know. Do LLMs "know" that they don't know? Real understanding includes acknowledging what you don't know. Can the predictive text generation of LLMs recognize that its training set did not include data?

famouswaffles 3y ago
There's quite a lot of indication that the computation can distinguishing hallucinations. It just has no incentive to communicate this. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 https://arxiv.org/abs/2207.05221