4 ms·
It was just a contrived example to illustrate low variance in the response distribution doesn't necessarily indicate accuracy. Just indicating that "hallucinat
by program_whiz 2y ago
It was just a contrived example to illustrate low variance in the response distribution doesn't necessarily indicate accuracy. Just indicating that "hallucination" is a different axis from "generates different responses" though they might not be totally orthogonal.
A better example might be that the model overtrained on AWS cloud formation API 2 and when v3 comes out produces low entropy answers that are wrong for v3 but right for v2 (due to training bias), but the answers are low variance (e.g. "bucket" instead of the new "bucket_name" key).
Another example based on a quick test I did on GPT4:
In a single phrase, what is Paris?
Paris is the city of Light.
Paris is the capital of France.
Paris is the romantic capital of the world renowned for its art, fashion, and culture.
- svnt 2y agoIt seems though that you may be conflating wrong with hallucination. Hallucination is one class of wrong, while your example, that of reliance on an outdated training set, produces answers that are wrong but probably shouldn’t be considered a hallucination. This paper specifically addresses hallucinations. Hallucinations are an error that occurs based on providing an answer with high uncertainty, not wrong or outdated training data.
- svnt 2y agoI came back to this because I felt like it would be good to address the Paris question. I don’t know the technical term for it, if there is one, but essentially by prompting with “In a single phrase” you are over-constraining the answer space, and so again you’ve moved into a high uncertainty regime. Consider if you prompted a tool to render Paris in a single image. Eiffel Tower and Arc? Paris street with bakery? Overhead maps view? With what included and what omitted?