3 ms·

I don't think discretizing the results solves the problem, we don't know whether the distribution is accurate without apriori knowledge. See my real GTP4 outpu

I don't think discretizing the results solves the problem, we don't know whether the distribution is accurate without apriori knowledge. See my real GTP4 output about Paris. Are the words "city of light" "center of culture" and "capital of France" confabulations? Without apriori knowledge is it more or less confabulatory than "city of roses", "site of religious significance", "capital of Korea"? If it simply output "Capital of Rome" 3 times, would that indicate its probably not a confabulation? You can discretize the concepts but that only serves to reduce the granularity of comparisons, and does solve the underlying problem I originally described.

eutropia 2y ago
The paper's method is trying to more accurately identify answers that are wrong and arbitrary (i.e., subject to random seed variance / temperature) - that's their definition of "confabulation". > If it simply output "Capital of Rome" 3 times, would that indicate its probably not a confabulation? Correct, it would tend to indicate that. Whether or not that is a true output is a different question, but it would tend to indicate that the "Capital of Rome" is something that comes up consistently regardless of variations in random seed. It's not a solution for hallucinations writ large, and it doesn't introduce an oracle of ground-truth accuracy on factoids. It just reweights possible answers by their 'semantic entropy'. They did something else cool in the paper too (unrelated to whatever problem you want to solve with respect to apriori knowledge) They made a method for decomposing originally generated answers into "factoids" and fact checking them using this semantic entropy concept: - Generate an output to a question e.g. "who is tom cruise?" - for each sentence fragment that represents a factoid "known for the movie top gun", make a question jeopardy style: "what fighter pilot movie is tom cruise known for?" - for each question, generate multiple answers and select one with low semantic entropy - compare that answer with the original factoid pretty cool overall, but still doesn't get us any closer to LLMs with a sense for whether something is fact-ish / a sense of truth.

eutropia 2y ago

The paper's method is trying to more accurately identify answers that are wrong and arbitrary (i.e., subject to random seed variance / temperature) - that's their definition of "confabulation". > If it simply output "Capital of Rome" 3 times, would that indicate its probably not a confabulation? Correct, it would tend to indicate that. Whether or not that is a true output is a different question, but it would tend to indicate that the "Capital of Rome" is something that comes up consistently regardless of variations in random seed. It's not a solution for hallucinations writ large, and it doesn't introduce an oracle of ground-truth accuracy on factoids. It just reweights possible answers by their 'semantic entropy'. They did something else cool in the paper too (unrelated to whatever problem you want to solve with respect to apriori knowledge) They made a method for decomposing originally generated answers into "factoids" and fact checking them using this semantic entropy concept: - Generate an output to a question e.g. "who is tom cruise?" - for each sentence fragment that represents a factoid "known for the movie top gun", make a question jeopardy style: "what fighter pilot movie is tom cruise known for?" - for each question, generate multiple answers and select one with low semantic entropy - compare that answer with the original factoid pretty cool overall, but still doesn't get us any closer to LLMs with a sense for whether something is fact-ish / a sense of truth.