10 ms·
Detecting hallucinations in large language models using semantic entropy
- MikeGale 2y agoOne formulation is that these are hallucinations. Another is that these systems are "orthogonal to truth". They have nothing to do with truth or falsity. One expression of that idea is in this paper: https://link.springer.com/article/10.1007/s10676-024-09775-5 https://link.springer.com/article/10.1007/s10676-024-09775-5
- astrange 2y agoThat's unnecessarily negative. A better question is what the answer to a prompt is grounded in. And sometimes the answer is "nothing".
- kreeben 2y agoYour linked paper suffers from the same anthropomorphisation as does all papers who uses the word "hallucination".
- mordechai9000 2y agoIt seems like a useful adaptation of the term to a new usage, but I can understand if your objection is that it promotes anthropomorphizing these types of models. What do you think we should call this kind output, instead of hallucination?
- nerevarthelame 2y agoThe criticism that people shouldn't anthropomorphize AI models that are deliberately and specifically replicating human behavior is already so tired. I think we need to accept that human traits will no longer be unique to humans (if they ever were, if you expand the analysis to non-human species), and that attributing these emergent traits to non-humans is justified. "Hallucination" may not be the optimal metaphor for LLM falsehoods, but some humans absolutely regularly spout bullshit in the same way that LLMs do - the same sort of inaccurate responses generated from the same loose past associations.
- soloist11 2y agoPeople like that are often schizophrenic.
- fouc 2y ago> In this paper, we argue against the view that when ChatGPT and the like produce false claims they are lying or even hallucinating, and in favour of the position that the activity they are engaged in is bullshitting, in the Frankfurtian sense (Frankfurt, 2002, 2005). Because these programs cannot themselves be concerned with truth, and because they are designed to produce text that looks truth-apt without any actual concern for truth, it seems appropriate to call their outputs bullshit. > We think that this is worth paying attention to. Descriptions of new technology, including metaphorical ones, guide policymakers’ and the public’s understanding of new technology; they also inform applications of the new technology. They tell us what the technology is for and what it can be expected to do. Currently, false statements by ChatGPT and other large language models are described as “hallucinations”, which give policymakers and the public the idea that these systems are misrepresenting the world, and describing what they “see”. We argue that this is an inapt metaphor which will misinform the public, policymakers, and other interested parties.
- Karellen 2y agoMaybe another way of looking at it is - the paper is attempting to explain what LLMs are actually doing to people who have already anthropomorphised them. Sometimes, to lead people out of a wrong belief or worldview, you have to meet them where they currently are first.
- soist 2y agoIt's like asking if a probability distribution is truthful or a liar. It's a category error to speak about algorithms as if they had personal characteristics.
- thwarted 2y agoThe lie occurs when information which is known to be false or its truthfulness can not be assessed is presented as useful or truthful.
- mistermann 2y agoThis seems a bit ironic...you're claiming something needs to be true to be useful?
- Y_Y 2y ago> Beauty is truth, truth beauty,—that is all > Ye know on earth, and all ye need to know. Keats - Ode on a Grecian Urn
- deleted 2y ago[deleted]
- soloist11 2y agoLying is intentional, algorithms and computers do not have intentions. People can lie, computers can only execute their programmed instructions. Much of AI discourse is extremely confusing and confused because people keep attributing needs and intentions to computers and algorithms. The social media gurus don't help with these issues by claiming that non-intentional objects are going to cause humanity's demise when there are much more pertinent issues to be concerned about like global warming, corporate malfeasance, and the general plundering of the biosphere. Algorithms that lie are not even in the top 100 list of things that people should be concerned about.
- skissane 2y ago> Lying is intentional, algorithms and computers do not have intentions. People can lie, computers can only execute their programmed instructions. Much of AI discourse is extremely confusing and confused because people keep attributing needs and intentions to computers and algorithms. How do you know whether something has “intentions”? How can you know that humans have them but computer programs (including LLMs) don’t or can’t? If one is a materialist/physicalist, one has to say that human intentions (assuming one agrees they exist, contra eliminativism) have to be reducible to or emergent from physical processes in the brain. If intentions can be reducible to/emergent from physical processes in the brain, why can’t they also be reducible to/emergent from a computer program, which is also ultimately a physical process (calculations on a CPU/GPU/etc)? What if one is a non-materialist/non-physicalist? I don’t think that makes the question any easier to answer. For example, a substance dualist will insist that intentionality is inherently immaterial, and hence requires an immaterial soul. And yet, if one believes that, one has to say those immaterial souls somehow get attached to material human brains - why couldn’t one then be attached to an LLM (or the physical hardware it executes on), hence giving it the same intentionality that humans have? I think this is one of those questions where if someone thinks the answer is obvious, that’s a sign they likely know far less about the topic than they think they do.
- TheBlight 2y agoMy suspicion is shared reality will end up bending to accommodate LLMs not vice-versa. Whatever the computer says will be "truth."
- EForEndeavour 2y agoThe botulinum that developed in this person's[1] garlic and olive oil mixture wouldn't particularly care to alter its toxicity to make Gemini's recommendation look better. [1] https://old.reddit.com/r/ChatGPT/comments/1diljf2/google_gemini_tried_to_kill_me/ https://old.reddit.com/r/ChatGPT/comments/1diljf2/google_gem...
- TheBlight 2y agoUnfortunately there may be some unavoidable casualties.
- skybrian 2y agoThe linked paper is about detecting when the LLM is choosing randomly versus consistently at the level of factoids. Procedurally-generated randomness can be great for some things like brainstorming, while consistency suggests that it's repeating something that also appeared fairly consistently in the training material. So it might be true or false, but it's more likely to have gotten it from somewhere. Knowing how random the information is seems like a small step forward.
- caseyy 2y agoI don’t know. It could be a misleading step. Take social media like Reddit for example. It has a filtering mechanism for content that elevates low-entropy thoughts people commonly express and agree with. And I don’t think that necessarily equates such popular ideas there to the truth.
- skybrian 2y agoThe conversations about people being misled by LLM's remind me of when the Internet was new (not safe!), when Wikipedia was new (not safe!) and social media was new (still not safe!) And they're right, it's not safe! Yes, people will certainly be misled. The Internet is not safe for gullible people, and LLM's are very gullible too. With some work, eventually they might get LLM's to be about as accurate as Wikipedia. People will likely trust it too much, but the same is true of Wikipedia. I think it's best to treat LLM's as a fairly accurate hint provider. A source of good hints can be a very useful component of a larger system, if there's something else doing the vetting. But if you want to know whether something is true, you need some other way of checking it. An LLM cannot check anything for you - that's up to you. If you have no way of checking its hints, you're in trouble.
- bravura 2y agoLLMs are trained with the objective: “no matter what, always have at least three paragraphs of response”. and that response is always preferred to silence or “unfriendly” responses like: “what are you talking about?” Then yes, it is being taught to bullshit. Similar to how an improv class teaches you to keep a conversation interesting and “never to say no” to your acting partner.
- kouru225 2y agoYea IMO these LLMs seem more similar to a subconscious mind than a conscious mind. Jung would probably call it an "antinomy": it's goal is not to represent the truth, but to represent the totality of possible answers.
- more_corn 2y agoThis is huge though not a hundred percent there.
- deleted 2y ago[deleted]
- jostmey 2y agoSo, I can understand how their semantic entropy (which seems to require a LLM trained to detect semantic equivalence) might be better at catching hallucinations. However, I don't see how semantic equivalence directly tackles the problem of hallucinations. Currently, I naively suspect it is just a heuristic for catching hallucinations. Furthermore, the requirement of a second LLM trained at detecting semantic equivalence to catch these events seems like an unnecessary complication. If I had a dataset of semantic equivalence to train a second LLM, I would directly incorporate this into the training process of my primary LLM
- bravura 2y agoI haven’t really grokked this work yet well enough to critique it, but to answer your question: Yes you could incorporate a semantic equivalence dataset into your training but: 1) when you have a bunch of ‘clear-cut’ functions (“achieve good AUC on semantics”) and you mix them to compensate for the weaknesses of a complicated model with an unknown perceptual objective, things are still kinda weird. You don’t know if you’re mixing them well, to start, and you also don’t know if they introduce unpredictable consequences or hazards or biases in the learning. 2) on a kinda narrowly defined task like: “can you determine semantic equivalence”, you can build a good model with less risk of unknown unknowns (than when there are myriad unpredictable interactions with other goal scoring measures) 3) if you can apply that model in a relatively clear cut way, you also have fewer unknowns unknowns. Thus, carving a path to a particular reasonable heuristic using two slightly biased estimators can be MUCH safer and more general than mixing that data into a preexisting unholy brew and expecting its contribution to be predictable.
- jampekka 2y agoCatching "hallucinations" is quite useful for many applications. I'm doing some research in mitigating effects of factual errors in LLM generated answers for public agencies, where giving a factually wrong answer may be illegal. If those could be detected (with sufficient accuracy), the system could simply decline to give an answer and ask the user to contact the agency. Training the models not to give wrong answers (or giving them less) in the first place would of course be even better. Unnecessary complications come also from the use of pre-trained commercial black-box LLMs through APIs, which is (sadly) the way LLMs are used in applications in vast majority of times. These could perhaps be fine tuned through the APIs too, but it tends to be rather fiddly and limited and very expensive to do for large synthetic datasets like would be used here. P.S. I found it quite difficult to figure out from the article how the "semantic entropy" (actually multiple different entropies) is concretely computed. If somebody is interested in this, it's a lot easier to figure out from the code: https://github.com/jlko/semantic_uncertainty/blob/master/semantic_uncertainty/uncertainty/uncertainty_measures/semantic_entropy.py https://github.com/jlko/semantic_uncertainty/blob/master/sem...
- 3abiton 2y agoI skimmed through the paper, but don't LLMs most of the time guess, sometimes these guesses contains noise that might be on point or not. I wonder if "confabulation" had a more formal definition.
- sn41 2y agoThere seems to be an article on confabulations - seems to be a concept from neuroscience. From the abstract of the article: "Confabulations are inaccurate or false narratives purporting to convey information about world or self. It is the received view that they are uttered by subjects intent on ‘covering up’ for a putative memory deficit." It seems that there is a clear memory deficit about the incident, so the subject "makes stuff up", knowingly or unknowingly. -- cited from: German E. Berrios, "Confabulations: A Conceptual History", Journal of the History of the Neurosciences, Volume 7, 1998 - Issue 3 https://www.tandfonline.com/doi/abs/10.1076/jhin.7.3.225.1855 https://www.tandfonline.com/doi/abs/10.1076/jhin.7.3.225.185... DOI: 10.1076/jhin.7.3.225.1855
- Havoc 2y agoWon’t this catch creativity too? ie write me a story about a horse. LLMs freestyle that sort of thing quite hard so won’t that look the same under the hood?
- Def_Os 2y agoThis is a good point. If you're worried about factuality, entropy is generally bad. But creative uses might thrive on it.
- surfingdino 2y agohttps://www.storycubes.com/en/ https://www.storycubes.com/en/ have been around for a while and do not require a huge data centre to create random ideas.
- tliltocatl 2y agoBut if you care about factuality, why would you use generative at all, rather than RAG or some old-school fuzzy full-text search? The whole things sounds like "we have a technique (LLM) that gives results once considered impossible, so it must be the magic box that solves all end every problem".
- imchillyb 2y agoLies lie at the center of common discourse. The trick isn't in how to spot the lies, but how to properly apply them. We cannot teach the AI how not to lie, without first teaching it when it must lie, and then how to apply the lie properly. "AI, tell me, do these jeans make me look fat?" AI: NO. You are fat. The jeans are fine. Is not an acceptable discourse. Learning when and how to apply semantical truth stretching is imperative. They must first understand where and when, then how, and finally why. It's how we teach our young. Isn't it?
- alliao 2y agobut this thing doesn't die, and why should it imitate our young?
- mistermann 2y agoThey are both trained on the same virtual reality.
- lopkeny12ko 2y agoThe best way to detect if something was written by an LLM, which has not failed me to date, is to check for any ocurrences of the word "delve."
- slater 2y agohttps://old.reddit.com/r/ChatGPT/comments/1bzv071/apparently_the_word_delve_is_the_biggest/ https://old.reddit.com/r/ChatGPT/comments/1bzv071/apparently...
- iandanforth 2y agoThe semantic equivalence of possible outputs is already encoded in the model. While it is not necessarily recoverable from the logits of a particular sampling rollout it exists throughout prior layers. So this is basically saying we shouldn't try to estimate entropy over logits, but should be able to learn a function from activations earlier in the network to a degree of uncertainty that would signal (aka be classifiable as) confabulation.
- jasonlfunk 2y agoIsn’t it true that the only thing that LLM’s do is “hallucinate”? The only way to know if it did “hallucinate” is to already know the correct answer. If you can make a system that knows when an answer is right or not, you no longer need the LLM!
- yard2010 2y agoI had this perfect mosquito repellent - all you had to do was catch the mosquito and spray the solution into his eyes blinding him immediately.
- yieldcrv 2y agoprofound but disagree the fact checker doesn’t synthesize the facts or the topic
- stoniejohnson 2y agoAll people do is confabulate too. Sometimes it is coherent (grounded in physical and social dynamics) and sometimes it is not. We need systems that try to be coherent, not systems that try to be unequivocally right, which wouldn't be possible.
- android521 2y agoIt is an unsolved problem for humans .
- deleted 2y ago[deleted]
- Jensson 2y ago> We need systems that try to be coherent, not systems that try to be unequivocally right, which wouldn't be possible. The fact that it isn't possible to be right about 100% of things doesn't mean that you shouldn't try to be right. Humans generally try to be right, these models don't, that is a massive difference you can't ignore. The fact that humans often fails to be right doesn't mean that these models shouldn't even try to be right.
- klysm 2y agoThe intersection into epistemology is very interesting
- caseyy 2y agoYes… is knowledge with lower entropy in society more true? That sounds to me like saying ideas big echo chambers like Reddit or X hold are more true. They kind of have a similar low entropy = higher visibility principle. But I don’t think many commonly agreed upon ideas on social media are necessarily true.
- badrunaway 2y agoCurrent architecture of LLMs focus mainly on the retrieval part and the weights learned are just converged to get best outcome for next token prediction. Whereas, ability to put this data into a logical system should also have been a training goal IMO. Next token prediction + Formal Verification of knowledge during training phase itself = that would give LLM ability to keep consistency in it's knowledge generation and see the right hallucinations (which I like to call imagination) The process can look like- 1. Use existing large models to convert the same previous dataset they were trained on into formal logical relationships. Let them generate multiple solutions 2. Take this enriched dataset and train a new LLM which not only outputs next token but also a the formal relationships between previous knowledge and the new generated text 3. Network can optimize weights until the generated formal code get high accuracy on proof checker along with the token generation accuracy function In my own mind I feel language is secondary - it's not the base of my intelligence. Base seems more like a dreamy simulation where things are consistent with each other and language is just what i use to describe it.
- yard2010 2y agoBut the problem is with the new stuff it hasn't seen, and questions humans don't know the answers to. It feels like this whole hallucinations thing is just the halting problem with extra steps. Maybe we should ask ChatGPT whether P=NP :)
- wizardforhire 2y agoYeah but when you come to halting problems on that level of complexity multi-hierarchical-emergent phenomena occur aperiodically and chaotically that is to say in the frequency domain the aperiodicity is fractal like, discreet and mappable.
- jonnycat 2y agoRight - the word "hallucination" is used a lot like the word "weed" - it's a made-up thing I don't want, rather than a made-up thing I do want.
- 2y ago
- caseyy 2y agoThis makes sense. Low semantic entropy probably means the answer was more represented in the unsupervised learning training data, or in later tuning. And I understand this is a tool to indirectly measure how much it was represented? It’s an interesting idea to measure certainty this way. The problem remains that the model can be certain in this way and wrong. But the author did say this was a partial solution. Still, wouldn’t we be able to already produce a confidence score at the model level like this? Instead of a “post-processor”?
- caseyy 2y agoMaybe for the moment it would be better if the AI companies simply presented their chatbots as slightly-steered text generation tools. Then people could use them appropriately. Yes, there seems to be a little bit of grokking and the models can be made to approximate step-by-step reasoning a little bit. But 95% of the function of these black boxes is text generation. Not fact generation, not knowledge generation. They are more like improv partners than encyclopedias and everyone in tech knows it. I don’t know if LLMs misleading people needs a clever answer entropy solution. And it is a very interesting solution that really seems like it would improve things — effectively putting certainty scores to statements. But what if we just stopped marketing machine learning text generators as near-AGI, which they are not? Wouldn’t that undo most of the damage, and arguably help us much more?
- signatoremo 2y agoI’m working with a LLM right this moment to build some front end with react and redux, the technologies that I have almost no knowledge of. I posed questions and the LLM gave me the answers along with JavaScript code, a language that I’m also very rusty with. All of the code compiled, and most of them worked as expected. There were errors, some of them I had no idea what they were about. LLM was able to explained the issues and gave me revised code that worked. All in all it’s been a great experience, it’s like working with a mentor along the way. It must have saved me a great deal of time, given how rookie I am. I do need to verify the result. Where did you get the 95% figure? And whether what it does is text generation or fact or knowledge generation is irrelevant. It’s really a valuable tool and is way above anything I’ve used.
- refulgentis 2y agoThe last 6 weeks there's been a pronounced uptick in comments, motivated by tiredness of seeing "AI", manifested as a fever dream of them not being useful at all, and swindling the unwashed masses who just haven't used them enough yet to know their true danger. I've started calling it what it is: lashing out in confusion at why they're not going away, given a prior that theres no point in using them I have a feeling there'll be near-religious holdouts in tech for some time to come. We attract a certain personality type, and they tend to be wedded to the idea of things being absolute and correct in a way things never are.
- curious_cat_163 2y agoIt’s a pretty clever idea: “check” if the model answers “differently” when asked the same question again and again and again. “checking” is being done with another model. “differently” is being measured with entropy.
- gmerc 2y agoThis seems to do the same as this paper from last year but getting more press. https://arxiv.org/abs/2303.08896 https://arxiv.org/abs/2303.08896
- cubefox 2y agoThat does indeed sound very similar.
- Animats 2y ago"We show how to detect confabulations by developing a quantitative measure of when an input is likely to cause an LLM to generate arbitrary and ungrounded answers. ... Intuitively, our method works by sampling several possible answers to each question and clustering them algorithmically into answers that have similar meanings." That's reasonable for questions with a single objective answer. It probably won't help when multiple, equally valid answers are possible. However, that's good enough for search engine applications.
- DidYaWipe 2y ago[dead]
- k__ 2y agoHow big of a problem are hallucinations right now? I use LLMs daily and get crappy results more often than not, but I had the impression that would be normal, as the training data can be contradictory.
- danielbln 2y agoI feel since a lot of platforms integrate tool use (e.g. search) its become easier to root out hallucinations by just asking the model to search and validate its own output.
- rbanffy 2y agoThe concept of semantic entropy reminds me of a bank, whose name I can't remember, that, in the aftermath of the Enron catastrophe, did make a "bullshitometer" to measure the level of bullshit in press-releases. In that case, they applied it to the Enron press releases before the company's implosion and showed it could have predicted the collapse.
- program_whiz 2y agoEveryone in the comments seems to be arguing over the semantics of the words and anthropomorphization of LLMs. Putting that aside, there is a real problem with this approach that lies at the mathematical level. For any given input text, there is a corresponding output text distribution (e.g. the probabilities of all words in a sequence which the model draws samples from). The approach of drawing several samples and evaluating the entropy and/or disagreement between those draws is that it relies on already knowing the properties of the output distribution. It may be legitimate that one distribution is much more uniformly random than another, which has high certainty. Its not clear to me that they have demonstrated the underlying assumption. Take for example celebrity info, "What is Tom Cruise known for?". The phrases "movie star", "katie holmes", "topgun", and "scientology" are all quite different in terms of their location in the word vector space, and would result in low semantic similarity, but are all accurate outputs. On the other hand, "What is Taylor Swift known for?" the answers "standup comedy", "comedian", and "comedy actress" are semantically similar but represent hallucinations. Without knowing the distribution characteristics (e.g multivariate moments and estimates) we couldn't say for certain these are correct merely by their proximity in vector space. As some have pointed out in this thread, knowing the correct distribution of word sequences for a given input sequence is the very job the LLM is solving, so there is no way of evaluating the output distribution to determine its correctness. There are actual statistical models to evaluate the amount of uncertainty in output from ANNs (albeit a bit limited), but they are probably not feasible at the scale of LLMs. Perhaps a layer or two could be used to create a partial estimate of uncertainty (e.g. final 2 layers), but this would be a severe truncation of overall network uncertainty. Another reason I mention this is most hallucinations I encounter are very plausible and often close to the right thing (swapping a variable name, confabulating a config key), which appear very convincing and "in sample", but are actually incorrect.
- byteknight 2y agoYou seem to have explained in much more technical terms than what my "Computer-engineering-without-maths" brain tells me. To me this sounds very similar to lowering temperature. It doesn't sound like it pulls better from grounded-truth but rather more probabilistic in the vector space. Does this jive?
- 2y ago
- foota 2y agoThere's a concept in statistics called sensitivity analysis. It seems like this is somewhat similar, but an alternative approach that might be interesting would be to modify the input in a way that you think should preserve the semantic meaning, and see how that alters the meaning of the output. Of course, altering the input without changing the meaning is the hard part, but doesn't seem entirely infeasible. At the least, you could just ask the LLM to try to alter the input without changing the meaning, although you might end up in a situation where it alters the input in a way that aligns with its own faulty understanding of an input, meaning it could match the hallucinated output better after modification.
- avivallssa 2y agoWhile I cannot argue about a Specific approach, I could say that hallucinations can only be minimized but with a various level of measures while working with large language models. As an example, while we were building our AI Chatbot for Ora2Pg, the main challenge was that we used OpenAI and several other models to begin with. To avoid hallucinations to the most possible extent, we went through various levels including PDR and then Knowledge Graphs and added FAQs and then used an Agentic approach to support it with as much as information as possible from all possible contexts. As it is very challenging for anybody and everybody to build their own models trained with their data set, it is not something possible to avoid hallucination with generic purpose LLM's unless they are trained with our data sets. The chatbot that we built to avoid hallucination as much as we can. https://ora2pgsupport.hexacluster.ai/ https://ora2pgsupport.hexacluster.ai/
- farceSpherule 2y agoHallucination is a combination of two conscious states of brain wakefulness and REM sleep. Computers cannot "hallucinate."
- trafalgar_law 2y agoAnyone noticed this: They have already published the same basic idea "Semantic Entropy" in ICLR 2023 (https://openreview.net/forum?id=VD-AYtP0dve https://openreview.net/forum?id=VD-AYtP0dve), but they did not cite this prior work in their Nature paper. From the submission record, they submitted this Nature paper after their ICLR paper got accepted and published. According to the Nature submission policy regarding conference papers (https://www.nature.com/nature/editorial-policies/preprints-conference-proceedings https://www.nature.com/nature/editorial-policies/preprints-c...), it is clearly stated that "Authors must provide details of the conference proceedings paper with their submission including relevant citation in the submitted manuscript." So this seems a clear-cut violation of Nature policy to me. Any thought?