4 ms·

the method described by this paper does not > draw[ing] several samples and evaluating the entropy and/or disagreement between those draws the method from the

the method described by this paper does not > draw[ing] several samples and evaluating the entropy and/or disagreement between those draws the method from the paper (as I understand it): - samples multiple answers, (e.g. "music:0.8, musician:0.9, concert:0.7, actress:0.5, superbowl:0.6") - groups them by semantic similarity and gives them an id ([music, musician, concert] -> MUSIC, [actress] -> ACTING, [superbowl] -> SPORTS), note that they just use an integer or something for the id - sums the probability of those grouped answers and normalizes: (MUSIC:2.4, ACTING:0.5, SPORTS:0.6 -> MUSIC:0.686, SPORTS:0.171, ACTING:0.143) They also go to pains in the paper to clearly define what they are trying to prevent, which is confabulations. > We focus on a subset of hallucinations which we call ‘confabulations’ for which LLMs fluently make claims that are both wrong and arbitrary—by which we mean that the answer is sensitive to irrelevant details such as random seed. Common misconceptions will still be strongly represented in the dataset. What this method does is it penalizes semantically isolated answers (answers dissimilar to other possible answers) with mediocre likelihood. Now technically, this paper only compares the effectiveness of "detecting" the confabulation to other methods - it doesn't offer an improved sampling method which utilizes that detection. And of course, if it were used as part of a generation technique it is subject to the extreme penalty of 10xing the number of model generations required. link to the code: https://github.com/jlko/semantic_uncertainty https://github.com/jlko/semantic_uncertainty

eigenspace 2y ago
Right, but the problem pointed out here is that if you compare that to the answers for Tom Cruise, you’d get a bunch of disparate answers that under this method would seem to indicate that it was confabulating, when in reality, Tom Cruise is just known for a lot of different things.
program_whiz 2y ago
I don't think discretizing the results solves the problem, we don't know whether the distribution is accurate without apriori knowledge. See my real GTP4 output about Paris. Are the words "city of light" "center of culture" and "capital of France" confabulations? Without apriori knowledge is it more or less confabulatory than "city of roses", "site of religious significance", "capital of Korea"? If it simply output "Capital of Rome" 3 times, would that indicate its probably not a confabulation? You can discretize the concepts but that only serves to reduce the granularity of comparisons, and does solve the underlying problem I originally described.
- eutropia 2y ago
  The paper's method is trying to more accurately identify answers that are wrong and arbitrary (i.e., subject to random seed variance / temperature) - that's their definition of "confabulation". > If it simply output "Capital of Rome" 3 times, would that indicate its probably not a confabulation? Correct, it would tend to indicate that. Whether or not that is a true output is a different question, but it would tend to indicate that the "Capital of Rome" is something that comes up consistently regardless of variations in random seed. It's not a solution for hallucinations writ large, and it doesn't introduce an oracle of ground-truth accuracy on factoids. It just reweights possible answers by their 'semantic entropy'. They did something else cool in the paper too (unrelated to whatever problem you want to solve with respect to apriori knowledge) They made a method for decomposing originally generated answers into "factoids" and fact checking them using this semantic entropy concept: - Generate an output to a question e.g. "who is tom cruise?" - for each sentence fragment that represents a factoid "known for the movie top gun", make a question jeopardy style: "what fighter pilot movie is tom cruise known for?" - for each question, generate multiple answers and select one with low semantic entropy - compare that answer with the original factoid pretty cool overall, but still doesn't get us any closer to LLMs with a sense for whether something is fact-ish / a sense of truth.

eigenspace 2y ago

Right, but the problem pointed out here is that if you compare that to the answers for Tom Cruise, you’d get a bunch of disparate answers that under this method would seem to indicate that it was confabulating, when in reality, Tom Cruise is just known for a lot of different things.

program_whiz 2y ago

I don't think discretizing the results solves the problem, we don't know whether the distribution is accurate without apriori knowledge. See my real GTP4 output about Paris. Are the words "city of light" "center of culture" and "capital of France" confabulations? Without apriori knowledge is it more or less confabulatory than "city of roses", "site of religious significance", "capital of Korea"? If it simply output "Capital of Rome" 3 times, would that indicate its probably not a confabulation? You can discretize the concepts but that only serves to reduce the granularity of comparisons, and does solve the underlying problem I originally described.

eutropia 2y ago
The paper's method is trying to more accurately identify answers that are wrong and arbitrary (i.e., subject to random seed variance / temperature) - that's their definition of "confabulation". > If it simply output "Capital of Rome" 3 times, would that indicate its probably not a confabulation? Correct, it would tend to indicate that. Whether or not that is a true output is a different question, but it would tend to indicate that the "Capital of Rome" is something that comes up consistently regardless of variations in random seed. It's not a solution for hallucinations writ large, and it doesn't introduce an oracle of ground-truth accuracy on factoids. It just reweights possible answers by their 'semantic entropy'. They did something else cool in the paper too (unrelated to whatever problem you want to solve with respect to apriori knowledge) They made a method for decomposing originally generated answers into "factoids" and fact checking them using this semantic entropy concept: - Generate an output to a question e.g. "who is tom cruise?" - for each sentence fragment that represents a factoid "known for the movie top gun", make a question jeopardy style: "what fighter pilot movie is tom cruise known for?" - for each question, generate multiple answers and select one with low semantic entropy - compare that answer with the original factoid pretty cool overall, but still doesn't get us any closer to LLMs with a sense for whether something is fact-ish / a sense of truth.