3 ms·

So, I can understand how their semantic entropy (which seems to require a LLM trained to detect semantic equivalence) might be better at catching hallucinations

So, I can understand how their semantic entropy (which seems to require a LLM trained to detect semantic equivalence) might be better at catching hallucinations. However, I don't see how semantic equivalence directly tackles the problem of hallucinations. Currently, I naively suspect it is just a heuristic for catching hallucinations. Furthermore, the requirement of a second LLM trained at detecting semantic equivalence to catch these events seems like an unnecessary complication. If I had a dataset of semantic equivalence to train a second LLM, I would directly incorporate this into the training process of my primary LLM

bravura 2y ago
I haven’t really grokked this work yet well enough to critique it, but to answer your question: Yes you could incorporate a semantic equivalence dataset into your training but: 1) when you have a bunch of ‘clear-cut’ functions (“achieve good AUC on semantics”) and you mix them to compensate for the weaknesses of a complicated model with an unknown perceptual objective, things are still kinda weird. You don’t know if you’re mixing them well, to start, and you also don’t know if they introduce unpredictable consequences or hazards or biases in the learning. 2) on a kinda narrowly defined task like: “can you determine semantic equivalence”, you can build a good model with less risk of unknown unknowns (than when there are myriad unpredictable interactions with other goal scoring measures) 3) if you can apply that model in a relatively clear cut way, you also have fewer unknowns unknowns. Thus, carving a path to a particular reasonable heuristic using two slightly biased estimators can be MUCH safer and more general than mixing that data into a preexisting unholy brew and expecting its contribution to be predictable.
jampekka 2y ago
Catching "hallucinations" is quite useful for many applications. I'm doing some research in mitigating effects of factual errors in LLM generated answers for public agencies, where giving a factually wrong answer may be illegal. If those could be detected (with sufficient accuracy), the system could simply decline to give an answer and ask the user to contact the agency. Training the models not to give wrong answers (or giving them less) in the first place would of course be even better. Unnecessary complications come also from the use of pre-trained commercial black-box LLMs through APIs, which is (sadly) the way LLMs are used in applications in vast majority of times. These could perhaps be fine tuned through the APIs too, but it tends to be rather fiddly and limited and very expensive to do for large synthetic datasets like would be used here. P.S. I found it quite difficult to figure out from the article how the "semantic entropy" (actually multiple different entropies) is concretely computed. If somebody is interested in this, it's a lot easier to figure out from the code: https://github.com/jlko/semantic_uncertainty/blob/master/semantic_uncertainty/uncertainty/uncertainty_measures/semantic_entropy.py https://github.com/jlko/semantic_uncertainty/blob/master/sem...

bravura 2y ago

I haven’t really grokked this work yet well enough to critique it, but to answer your question: Yes you could incorporate a semantic equivalence dataset into your training but: 1) when you have a bunch of ‘clear-cut’ functions (“achieve good AUC on semantics”) and you mix them to compensate for the weaknesses of a complicated model with an unknown perceptual objective, things are still kinda weird. You don’t know if you’re mixing them well, to start, and you also don’t know if they introduce unpredictable consequences or hazards or biases in the learning. 2) on a kinda narrowly defined task like: “can you determine semantic equivalence”, you can build a good model with less risk of unknown unknowns (than when there are myriad unpredictable interactions with other goal scoring measures) 3) if you can apply that model in a relatively clear cut way, you also have fewer unknowns unknowns. Thus, carving a path to a particular reasonable heuristic using two slightly biased estimators can be MUCH safer and more general than mixing that data into a preexisting unholy brew and expecting its contribution to be predictable.

jampekka 2y ago

Catching "hallucinations" is quite useful for many applications. I'm doing some research in mitigating effects of factual errors in LLM generated answers for public agencies, where giving a factually wrong answer may be illegal. If those could be detected (with sufficient accuracy), the system could simply decline to give an answer and ask the user to contact the agency. Training the models not to give wrong answers (or giving them less) in the first place would of course be even better. Unnecessary complications come also from the use of pre-trained commercial black-box LLMs through APIs, which is (sadly) the way LLMs are used in applications in vast majority of times. These could perhaps be fine tuned through the APIs too, but it tends to be rather fiddly and limited and very expensive to do for large synthetic datasets like would be used here. P.S. I found it quite difficult to figure out from the article how the "semantic entropy" (actually multiple different entropies) is concretely computed. If somebody is interested in this, it's a lot easier to figure out from the code: https://github.com/jlko/semantic_uncertainty/blob/master/semantic_uncertainty/uncertainty/uncertainty_measures/semantic_entropy.py https://github.com/jlko/semantic_uncertainty/blob/master/sem...