4 ms·

> On the other hand, "What is Taylor Swift known for?" the answers "standup comedy", "comedian", and "comedy actress" are semantically similar but represent hal

by svnt 2y ago

> On the other hand, "What is Taylor Swift known for?" the answers "standup comedy", "comedian", and "comedy actress" are semantically similar but represent hallucinations. Without knowing the distribution characteristics (e.g multivariate moments and estimates) we couldn't say for certain these are correct merely by their proximity in vector space. It depends on the fact that a high uncertainty answer by definition is less probable. That means if you ask multiple times you will not get the same unlikely answer, such as that Taylor swift is a comedian, you will instead get several semantically different answers. Maybe you’re saying the same thing, but if so I’m missing the problem. If your training data tells you that Taylor Swift is known as a comedian, then hallucinations are not your problem.

sigmoid10 2y ago
This. For a model to consistently output that Taylor Swift is a comedian or something similarly wrong at reasonable temperature settings, there must be a problem in the training data. That doesn't mean that "Taylor Swift is a comedian" needs to be in the training data, it it can simply mean that "Taylor Swift" doesn't appear at all. Then "singer" and "comedian" (and tons of other options) will likely appear at similar probabilities during generation. Blaming human semantics for LLMs is generally a bad idea, since we only use human semantics to qualitatively explain how the models abstract ideas. In practice you simply don't know how the model relates words.
program_whiz 2y ago
It was just a contrived example to illustrate low variance in the response distribution doesn't necessarily indicate accuracy. Just indicating that "hallucination" is a different axis from "generates different responses" though they might not be totally orthogonal. A better example might be that the model overtrained on AWS cloud formation API 2 and when v3 comes out produces low entropy answers that are wrong for v3 but right for v2 (due to training bias), but the answers are low variance (e.g. "bucket" instead of the new "bucket_name" key). Another example based on a quick test I did on GPT4: In a single phrase, what is Paris? Paris is the city of Light. Paris is the capital of France. Paris is the romantic capital of the world renowned for its art, fashion, and culture.
- svnt 2y ago
  It seems though that you may be conflating wrong with hallucination. Hallucination is one class of wrong, while your example, that of reliance on an outdated training set, produces answers that are wrong but probably shouldn’t be considered a hallucination. This paper specifically addresses hallucinations. Hallucinations are an error that occurs based on providing an answer with high uncertainty, not wrong or outdated training data.
- svnt 2y ago
  I came back to this because I felt like it would be good to address the Paris question. I don’t know the technical term for it, if there is one, but essentially by prompting with “In a single phrase” you are over-constraining the answer space, and so again you’ve moved into a high uncertainty regime. Consider if you prompted a tool to render Paris in a single image. Eiffel Tower and Arc? Paris street with bakery? Overhead maps view? With what included and what omitted?

by svnt 2y ago

sigmoid10 2y ago

This. For a model to consistently output that Taylor Swift is a comedian or something similarly wrong at reasonable temperature settings, there must be a problem in the training data. That doesn't mean that "Taylor Swift is a comedian" needs to be in the training data, it it can simply mean that "Taylor Swift" doesn't appear at all. Then "singer" and "comedian" (and tons of other options) will likely appear at similar probabilities during generation. Blaming human semantics for LLMs is generally a bad idea, since we only use human semantics to qualitatively explain how the models abstract ideas. In practice you simply don't know how the model relates words.

program_whiz 2y ago

It was just a contrived example to illustrate low variance in the response distribution doesn't necessarily indicate accuracy. Just indicating that "hallucination" is a different axis from "generates different responses" though they might not be totally orthogonal. A better example might be that the model overtrained on AWS cloud formation API 2 and when v3 comes out produces low entropy answers that are wrong for v3 but right for v2 (due to training bias), but the answers are low variance (e.g. "bucket" instead of the new "bucket_name" key). Another example based on a quick test I did on GPT4: In a single phrase, what is Paris? Paris is the city of Light. Paris is the capital of France. Paris is the romantic capital of the world renowned for its art, fashion, and culture.

svnt 2y ago
It seems though that you may be conflating wrong with hallucination. Hallucination is one class of wrong, while your example, that of reliance on an outdated training set, produces answers that are wrong but probably shouldn’t be considered a hallucination. This paper specifically addresses hallucinations. Hallucinations are an error that occurs based on providing an answer with high uncertainty, not wrong or outdated training data.
svnt 2y ago
I came back to this because I felt like it would be good to address the Paris question. I don’t know the technical term for it, if there is one, but essentially by prompting with “In a single phrase” you are over-constraining the answer space, and so again you’ve moved into a high uncertainty regime. Consider if you prompted a tool to render Paris in a single image. Eiffel Tower and Arc? Paris street with bakery? Overhead maps view? With what included and what omitted?