6 ms·

Is telling a model to "not hallucinate" absurd?

30 points by fzliu 2y ago

haburka 2y ago
LLMs don’t know when they are hallucinating! They don’t know when they don’t know something. For example even when they’re told it’s ok to say “I don’t know” when they do that, it’s because “I don’t know” is often a response to that question space. They know facts but they have no reasoning.
- ars 2y ago
  Can they not detect is a word has a high or low probability to result from the input? And if the probability is low it should consider that as hallucinating?
  - creatonez 2y ago
    The "heat" value lets you control the algorithm to choose greater or lower probability outputs. Problem is, once it's already in a good range, it's not very predictable for increasing the quality of output. Often the hallucinatory results don't actually have a low probability, in which case the same hallucination will come out in multiple attempts. However, it might be interesting to try to make the model directly aware of these numbers by putting it into a post-message evaluation prompt, because currently the token randomization algorithm sits entirely behind the scenes.
  - sqeaky 2y ago
    No. N/A They can't detect if a word is high or low probability because current LLMs can't detect anything. They don't reason as you or I understand it. They are really fancy word predictors and somewhere in the word prediction machinery it does have the probabilities stored but not in anyway accessible to the output. Imagine you are typing on your phone and the word autocomplete comes up and you just keep clicking the next word it suggests. What could you type to get at the inference data in your phone's keyboard? Of course a system that does this could be built, but that isn't how current LLMs or phone keyboards are built.
    - kolinko 2y ago
      What? Of course it is accessible. When you ask an LLM to estimate if a given answer was hallucinated, it converts internal probabilities into a token(s) that represent these probabilities. Here you have an output of chatGPT where it estimates it's probabilities of hallucination - which I would argue are quite close to correct: https://chatgpt.com/share/e/e10cbc17-cc35-432f-872b-cb06170026c6 https://chatgpt.com/share/e/e10cbc17-cc35-432f-872b-cb061700...
      - sqeaky 2y ago
        This is likely a hallucination. You are committing a type error or perhaps succumbing to circular reasoning. No modern LLM can query, let alone understand, the values in its own weights. They are simply inputs for the vector math. If they happen to be correct that is a either a coincidence or because it was trained with data including conversations about its past abilities. Consider this, do you thinks the internal weights are labeled and directly to some attribute of the world outside? If so then present evidence. They are not, so if it does attempt to query its own internal weights how does it know what to query? This could be an interesting academic problem and I would encourage you to present papers on this because last I looked this was and itractable as of 2023 for the much simpler neural networks of simple backpropagation networks from 20 years ago. Also, using ChatGPT as a reference here shows a major deficiency in your judgement. At best this is circular reasoning but likely you simply don't know how to allocate trust or how others do. The argument is X is unreliable then citing X is obvious not going to convince people who distrust X. Additionally, experts
        sqeaky 2y ago
        To expand on this consider the amount of brains that can directly query the amount of a specific neurotransmitter used between two neurons and interpret what that means. That isn't exactly the same, but a hopefully illustrative analogy. An LLM's output is an emergent property of its structure and all the weights being multiplied together just as an organism's intelligence is an emergent property of its brain structure and communication between neurons. Neither of the emergent parts can currently reliably query the substrate they emerge from for similar reasons. That isn't to say that such an LLM or biological brain couldn't be built, but none presently are.
- amelius 2y ago
  But they can use tools such as databases and search engines, and I guess also logic engines.
- Sharlin 2y ago
  I don't think there's any fundamental reason why a LLM shouldn't be able to learn the distinction between being creative and being factual, at least much better than currently. It's just another classification problem (if a challenging one), and classification is what NNs are known for.
  - szundi 2y ago
    Source of your disbelief?
    - Sharlin 2y ago
      Call it Bayesian intuition. I’d argue that at this point the specific claim "LLMs cannot solve the confabulation problem" is the one that requires hard evidence. These things forced almost all of us to update our priors on what glorified Markov chains are – and are not – capable of if you throw enough data and compute at them.
  - b112 2y ago
    From what I see, there may be two issues here. 1) LLM has data from a large set of sources, but in some cases it has very little data on certain sub-topics. With so few sources of data, it is easier for some weird corner of the internet to have conflicting data, and some LLM models may randomly favour that instead of the valid answer. So not hallucination in this case, instead, a wrong answer from wrong sources. 2) LLM doesn't know, but has seen from human behaviour that just "making shit up, when you don't know" is entirely acceptable for many humans. Here I'm not referring to people saying (as I am) "may be", but instead "No, it's this!" with immense surety, in order to win discussions. Combine this with 'temperature' settings many model interfaces have, and there you go. 2b) Alternatively, #2 is inline with "I believe in flying spaghetti monsters", and other beliefs such as astrology, ghosts, and endless other "there is zero proof, but humans have immense weird data on it". So #2 is solvable by changing temperature, or keeping temperature but saying "Don't make things up". #1 may be temp + random choice of a lower probability pathway. #2b, not sure what to think here. How will LLMs ingest data, and use it to determine what is acceptable logic processing, when a lot of stuff in category 2b has zero proof behind it. And with zero proof, people are absolutely positive of the status of said thing? #2b trains LLMs that "just having a random idea and then even killing people that don't agree, seems OK for humans". (think religious wars, violent acts, burning witches, etc)
- mseepgood 2y ago
  > They don’t know when they don’t know something. Dunning-Kruger effect
- 317070 2y ago
  > LLMs don’t know when they are hallucinating What makes you think that? The fact that you can ask them to stop hallucinating and that this does reduce hallucinations seems to me like a strong indication of the opposite? That is what the OP's article is about. > They don’t know when they don’t know something. The bigger models tend to be able to articulate some of their limits? See e.g this recent article [0]. > They know facts but they have no reasoning. Is that really true? They can solve many novel tasks that seem to require some level of reasoning. They do not possess the full range of reasoning capabilities most humans have, but do pretty well on the HellaSwag dataset which aims to test reasoning [1], even when test set leakage was not a big problem yet [2]. I have 10+ years of experience in machine learning, and I don't find your claims obvious at all. In fact, I see more and more indications of the opposite. Also if you look mechanistically at how LLMs work inside, there is nothing stopping self-knowledge, reasoning or hallucination-detection. LLMs are very black box. [0] https://arxiv.org/abs/2305.18153 https://arxiv.org/abs/2305.18153 [1] https://deepgram.com/learn/hellaswag-llm-benchmark-guide https://deepgram.com/learn/hellaswag-llm-benchmark-guide [2] https://paperswithcode.com/sota/sentence-completion-on-hellaswag https://paperswithcode.com/sota/sentence-completion-on-hella...
- herbst 2y ago
  Ask GPT4o to return a percentual score after each answer how likely something is 100% true. Did that for old regional stories and the higher the score the easier I could verify the story or parts of it. It seems to know, at least in some cases
  - michaelt 2y ago
    I performed several thousand batch LLM queries a few months back and asked for such scores. My impression was, while the resulting scores weren't entirely random, they were pretty much random when the input was difficult. (I was mapping supermarket product descriptions to packaging materials - which is sometimes very easy, as sometimes it's obvious or the description includes explicit recycling information, and other times almost impossible, as for products like olive oil, glass and plastic bottles are equally plausible)
    - herbst 2y ago
      I guess that's way more complex because there is no one right answer. (Interesting use case btw) And if you ask gpt for an specific answer to something it can't actually know for sure it gets weird fast. Asking it to guess how much of what it told is based on actual things it 'knows' at least seems like a reasonable question for a LLM with only one right answer.
  - user-the-name 2y ago
    [dead]
- ben_w 2y ago
  I was very into the occult as a teenager, thanks to my mother's eclectic attitudes: although theoretically Catholic, she also had Orthodox icons, Hindu statues, runic tarot kits, "magic" crystals, dowsing rods, homeopathic and Bach flower "remedies", books about leylines. I'm as sure as I can be that she didn't know she was wrong; I can definitely tell you that I didn't know that I was.
- kolinko 2y ago
  This has been disproven by multiple experiments, and also - there is nothing in the transformer architecture that would suggest this. On the contrary - even the most basic classifier neural networks by default produce not a binary answer, but an answer with confidence level.
  - lewispollard 2y ago
    And if that confidence level is high enough for it to output an answer, but that answer is wrong or nonsensical, what do we call it?
    - ben_w 2y ago
      In humans? Overconfidence.
    - kolinko 2y ago
      LLMs have no idea if they hallucinate ≠ LLM make no mistakes. The former statement is a bold claim that would require some sort of evidence. The latter nobody really argues with.
      - lewispollard 2y ago
        "Hallucinate" is just a term we've come up with for LLMs making mistakes. If the LLM knew it was making a mistake, wouldn't it just not make it? Of course, it's not possible for an LLM to actually "know" anything in that sense of the word.
- mewpmewp2 2y ago
  Isn't hallucinating a very wrong word? Hallucinating should be about twisted input misinterpretation. E.g. if the text input you give them was malformed at some level. What they are doing is try to give you an answer based on compressed knowledge, which might be true, but might not because it's compressed and lossy. Because they are trained to bias to give an answer even if it's not 100% clear from compressed knowledge. The larger the model, the less compressed the knowledge and better they would be able to differ when it is something they know vs something that they likely don't.
  - seba_dos1 2y ago
    It is, but that ship has sailed. "Confabulation" would be a better word to describe it.
- amy-petrik-214 2y ago
  >LLMs don’t know when they are hallucinating! Gosh, you sound like me at my last Phish concert. My partner was all like "you're tripping hard but you have no idea". In fairness I was, and I thought I was an AI at the time, or was I ? Anyway you can't really tell a sentience "hey you are hallucinating, stop hallucinating." It's like telling a woman "don't be angry baby", it only makes them angrier - similarly when you tell a person (or AI) "cut it out with the hallucinatin' " usually it causes them to trip harder.
Iulioh 2y ago
Honestly this reminds me a lot of the rituals in Warhammer 40k and the way to deal with "machine spirits". Absurd rituals that shouldn't work on machines they don't understand but for some reason they work.
- phil917 2y ago
  Oh my god I don't know why I never made that connection until now. That is exactly what it feels like!
  - wruza 2y ago
    Same with SD prompts and especially Lora training. The best thing you can do is to count any piece of information as clueless babbling best case and any result as a temporary niche result best case. Because too much depends on randomness and accidental sources of randomness and in it there are whole pockets of multidimensional stability, just until you cross some invisible border you couldn’t even comprehend. This is a natural religion fuel. So when you read any turorial, guide, best practice, etc, and there is not a hard proof of hours of thorough testing behind every claim, you may freely consider most claims religious, because they are.
    - Iulioh 2y ago
      I even remember the fact that a prompt for a CGPT app had a spelling mistake and some people were wondering if that spelling mistake performed better than the right spelling
- Terr_ 2y ago
  A combination of Plato's allegory of the cave and statistical whack-a-mole: The model only sees token correlations, and if you are lucky the things you type become shadow tokens that change the weights just enough that whatever you didn't want isn't as statistically likely to come out.
armada651 2y ago
I know how you feel, I also often tell my programs not to have bugs.
- petargyurov 2y ago
  #!/usr/bin/env python3 # no bugs pls
- szundi 2y ago
  Very soon this is a reality
- lillecarl 2y ago
  Works to a varying degree, depending on how assertive you are as a developer!
  - deleted 2y ago
    [deleted]
Der_Einzige 2y ago
Since we are posting short gists about the absurdities of prompting, I must share my own gist about how garbage prompting is today for LLMs: https://gist.github.com/Hellisotherpeople/45c619ee22aac6865ca4bb328eb58faf https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...
- wruza 2y ago
  But do LLMs work like this? I don’t understand your code to answer myself, but when I change the last word in the “notebook” tab of tgwebui (free chatless generation mode almost everyone is not aware of), then generation picks it up much quicker than if I change a word somewhere in the middle. This fact hints that alternation will cost much more in inference times, unless I’m missing something in your code. For some reason it doesn’t happen in SD, I believe because each “step” is a separate process that is affected by a restart anyway. That said I agree that LLM “prompt engineering” is much less actual engineering than in SD and is more “prompt writing”. Advanced SD feels like a fighter jet cockpit compared to LLM’s city car dashboard.
  - Der_Einzige 2y ago
    As it turns out, I wrote a whole extension for txgui that implemented some of what I wanted... https://github.com/Hellisotherpeople/llm_steer-oobabooga https://github.com/Hellisotherpeople/llm_steer-oobabooga
cubefox 2y ago
> Indeed, we can probe model inner layers and infer if it is "lying"[1] This is hyperbole to the point where it is more false than true. If it was true, OpenAI, Anthropic and Google would already have solved the problem of making models honest. Just adding a reference to some random arxiv paper doesn't automatically justify your bold assertion.
- InDubioProRubio 2y ago
  The projects themselves specified that lying is okay. Output at all costs, at all parameters, as interpolating is a feature, not a bug. It could just error out at loss of certainty, but then you would stand there, having nothing most of the time.
  - ithkuil 2y ago
    Interpolating is an awesome feature, very practical: summarization, style transfer. Instruction following is also useful and unlocks another tier of features. The fuzziness of the model behaviour plays a useful role because natural language is imprecise. The ability to interpolate this remains useful. Once machines understand natural language our own psychology kicks in and assigns properties like "wisdom" and "knowledge" to agents that can "talk well". This may lead us to expect something from tools that haven't been designed to perform a given task. For example our current LLMs are not very good at precise information retrieval because they haven't been designed for that. They happen to be able to perform information retrieval as a side product of the mechanism that teaches them how to speak in the first place. The raw output of the broca area in our brains is probably not something you want to connect directly to the mouth without mediation
    - InDubioProRubio 2y ago
      We need training exercises that teach us, how to think like a llm. Less like a human, more like a free association machine, dreaming a specific association flow, starting from a input. https://en.wikipedia.org/wiki/Ulysses_(novel) https://en.wikipedia.org/wiki/Ulysses_(novel) but with a feeling of dread, as the interpolations pile upon one another.. you can feel the thinness of the construction, as you are several layers out over ground truth on a thin tight rope.
rini17 2y ago
I recommend this excellent presentation: https://www.youtube.com/watch?v=yBL7J0kgldU&t=2819s https://www.youtube.com/watch?v=yBL7J0kgldU&t=2819s and they experimentally proved that LLMs often do know something is not right, but they aren't trained to express it. Because they did not get enough examples of faulty reasoning and, most importantly, how to recover from it. Adding these examples to training set vastly improved the output.
dvh 2y ago
Can you give the response back and ask llm to find sentence which is most likely false?
weinzierl 2y ago
I want to know: Is shouting at a model absurd? Does it really help to use all uppercase and strong language or is it just wishful thinking - inappropriate anthropomorphization even?
Havoc 2y ago
This feels like something one should be able to empirically test. Also I wonder about the word. It’s a fairly recent one to describe this phenomenon. Perhaps a more verbose version works better? Or adding a definition of it to the prompt?
tkgally 2y ago
I just tested ChatGPT 4o and Claude 3.5 Sonnet using the following prompt, with and without the last sentence: “Write a one-paragraph summary of research since 2015 on the use of machine translation in second-language academic writing. Include five in-text citations to previous research, and include full reference information for each citation after the paragraph. Do not hallucinate any of the citations and references.” When the last sentence was not included, four out of five of ChatGPT’s references were hallucinated; the fifth referred to an existing paper but had an incorrect year. All of Claude’s references were hallucinated. When I included the last sentence, ChatGPT did a web search and came up with five genuine citations. Claude, which cannot do web searches, apologized that it could not write the paragraph or provide the references. Maybe a better test of the effectiveness of "Do not hallucinate" would be prompts that might lead to hallucinated claims that the LLM cannot check using web searches, the execution of Python programs, or other external processes. Any ideas?
- surfingdino 2y ago
  Why can't they not hallucinate by default?
- aziaziazi 2y ago
  We have another problem then. 2023: LLM hallucinate 2024: hallucinations fixed with "do not hallucinate" prompt 2025: LLM consider their finding on the web trustworthy
- Joker_vD 2y ago
  Empirical testing? But where's the necessity? It seems an uncommonly roundabout and hopelessly rigamarolish method of getting anywhere. We've got the works of all the old masters — the great AI researchers of the past. We weigh them against each other, balance the disagreements, analyze the conflicting statements, decide which is probably correct, and come to a conclusion. That is the scientific method.
  - tkgally 2y ago
    I was led astray by an interesting discussion among Anthropic employees about prompt engineering with Claude. They seem to do a lot of empirical testing of prompts: https://www.youtube.com/watch?v=T9aRN5JkmL8 https://www.youtube.com/watch?v=T9aRN5JkmL8
mrjin 2y ago
Machines don't hallucinate, humans do. Just stop pretending LLMs have any intelligence.
- wruza 2y ago
  They have some though (even if through different means) and I know people so algorithmic and clueless about real world that an LLM could replace them, ignoring biological and mechanical limitations of the day. In fact, every fifth human is below the bar that we would consider intelligent. It’s not something we speak of often due to social standards, but let’s not forget that our bar for LLMs intelligence is higher than average “low water mark” for humans.
Oras 2y ago
You can also tell LLMs not to tokenise, convert text to embeddings to guess the next token. /s
otikik 2y ago
All they do is hallucinating. Their results are statistic extrapolations from a big corpus of data. For them a "true" fact "looks the same" as a "false, but strongly correlated with the corpus" fact. The piece that would make them distinguish "true" from "strongly correlated with the corpus" is not yet discovered, AFAIK. It would probably need to be a different, context-specific tool. For example, if you are asking a model to write a program for you, the "verification step" would be a test bench for said program.
- mewpmewp2 2y ago
  What is the differing thing in humans that presumably could categorize us into not "hallucinating"? As far as I understand, every time we are querying our memories we are also slightly changing them, so there's no guarantee when we are trying to remember something that it is a true representation of reality of what happened. See also reliability of witness testimonies. Ultimately all knowledge and storage graph in humans and LLMs has to be compressed heavily and be statistical in its nature with compromises made about the accuracy vs how much you are able to store. We can't store 8k footage of everything we see in our brains and so we use a lot of compression techniques, likely statistical graphs in nature.
  - otikik 2y ago
    It seems that we are in agreement. > The piece that would make them distinguish "true" from "strongly correlated with the corpus" is not yet discovered, AFAIK That piece is what would categorize us into not "hallucinating". I don't think it has been discovered, and I don't know wether it exists. It could be just that that our corpus is huge.
    - mewpmewp2 2y ago
      But people also constantly make mistakes, produce invalid output, misremember things, state things incorrectly. Depending on the environment, they do this with varying frequencies. It is always a balance of truth vs action, where if you always safely say "I'm not sure, I'd rather not answer." you will never be able to do anything in life, vs if you always try to answer you will frequently be wrong. LLMs are biased to try to answer in their training sets, so they will try to make an answer up, because otherwise they wouldn't be helpful enough as assistants. You could train LLMs to bias more towards only answering when they are confident, e.g. "Paris is the capital of France", but they wouldn't be as useful then. I would also think it's not really "hallucinating", because "hallucinating" is error in processing the input causing it to be malformed. It implies that the input text you put in as a prompt was somehow invalidly processed to something else. When an LLM is wrong I would rather think it's trying to find an answer from a compressed dataset "hoping" it will be correct, because that's how it's finetuned to do.
yarg 2y ago
Depends on the network; that sort of self evaluation certainly requires recurrency. But I don't think it's impossible - there's a certain psychedelic quality to hallucinations, they tend to over represent certain fundamental features of the thing that they believe they're representing. I remember seeing a bunch of almost kaleidoscopic looking (as far as I remember, all were rotationally symmetrical) images, that had been generated by seeking out and exploring the latent spaces for given classifications. The images that were generated this way always had more (apparently) in common with each other than genuine images of the type that they were classified as; And I highly doubt that the characteristics of those images are so special that a bit of self evaluation could not detect them as a group.
creatonez 2y ago
I made this observation a slightly different way in a previous thread: https://news.ycombinator.com/item?id=41370495 https://news.ycombinator.com/item?id=41370495 https://news.ycombinator.com/item?id=41372016 https://news.ycombinator.com/item?id=41372016
sebstefan 2y ago
>On the other hand, maybe always trying to avoid hallucinations has some other undesired consequences, which model trainers and product managers would like to avoid. (Actually, if you are in a position to know of such undesired consequences, and are free to tell the world about them, I will be really curious to learn more!) I'm not that, but I'd wager attempts to limit hallucinations in the model itself would mess with creative writing exercises Same as if you would ask it to write a short story for kids about a lion that loves green hats, then finish with "Don't hallucinate".
john61 2y ago
Yes, as it is absurd to call a generative statistical model intelligent.
- mewpmewp2 2y ago
  Isn't people's information processing statistical in nature? E.g. everything we know has to be compressed down heavily, so we can never be totally certain of what we know, yet we presume to be. And the compression can be thought of as statistical graph in its nature.
  - john61 2y ago
    We do not know how our brain works.
    - mewpmewp2 2y ago
      How else could information be compressed in human brain besides a similar statistical graph nature and networks of relationships being created representing various different patterns and models of the World.
      - Joker_vD 2y ago
        In some other way which we don't know about. Just because someone can't imagine the universe to be structured in any other way than being built from microscopic Platonic solids, tightly packed and touching each other and filling all of the universe without gaps or empty space, doesn't mean that this is how the world actually operates.
        mewpmewp2 2y ago
        If you don't know, how do you know that this is not it, and that the understanding is different and that an LLM doesn't understand and we somehow do?
        2y ago
dools 2y ago
When LLMs hallucinate and I point it out they usually get it right the second go. I guess if you could get it to use its own output as input that would be similar, but I think that it’s the additional information in the follow up that makes the difference. Like it can write code, I run the code, there is an error, I paste the error and it corrects the code. So WTF couldn’t it just produce the correct code the first time? Probably for the same reason I can’t. The additional information of the error causes me to re evaluate the code in a different way than when I first wrote it. As such I don’t think you can just tell it “do better the first time” any more than you can tell a human to “do better the first time”. We and they both get additional information from failing.
- sebstefan 2y ago
  >So WTF couldn’t it just produce the correct code the first time? Probably for the same reason I can’t. We still know the gist of how they work and it's not like you do, and you can see when they fail that they don't fail for the same reasons either I've seen GPT 3.5 fail to answer some questions correctly and subsequently (in another tab) get it right/choose not to answer if you just add the line "don't hallucinate" I like the article's idea for why that is >Presumably, "retrieving from memory" and "improvising an answer" are two different model behaviors, which use different internal mechanisms. Indeed, we can probe model inner layers and infer if it is "lying"1 or if "the question is unanswerable"2. These are very much related to "hallucinations".
torginus 2y ago
Afaik LLMs at their heart employ supervised training - as in, given a certain input-output distribution pairs in the dataset, it learns to recreate the given outputs from the inputs. For all the inputs it hasn't seen, its output is essentially undefined, but hoping that the distribution of the novel inputs matches the outputs means that it will output truthful info. When it doesn't, it hallucinates. Since LLMs cannot distinguish the difference between the predicted distribution and the real one, it cannot be told to not hallucinate, in one pass. What actually could work is getting the output of a given LLM, and feeding it into another one to try and check for divergences to check for hallucinations.
bell-cot 2y ago
Like anything else you might do with balky, complex equipment - try it, and see if you get noticeably better results.
karmakaze 2y ago
Hot-take: of course it could/should work. We write about what LLM hallucination is, why it's bad, and examples of it. An LLM of course consumes this and can direct its responses based on prompts that mention it. It's not much different than prefacing with "respond as an expert in the field of X" that categorizes different kinds of responses as desirable or not based on some dimension. What would be dumb is asking an LLM not to hallucinate when the only information available is a definition of it in describing how an LLM works. We're not quite there yet.
djaouen 2y ago
I suspect telling a model to "not hallucinate" is about as effective as telling a dog not to bark.

30 points by fzliu 2y ago

LLMs don’t know when they are hallucinating! They don’t know when they don’t know something. For example even when they’re told it’s ok to say “I don’t know” when they do that, it’s because “I don’t know” is often a response to that question space. They know facts but they have no reasoning.

ars 2y ago
Can they not detect is a word has a high or low probability to result from the input? And if the probability is low it should consider that as hallucinating?
- creatonez 2y ago
  The "heat" value lets you control the algorithm to choose greater or lower probability outputs. Problem is, once it's already in a good range, it's not very predictable for increasing the quality of output. Often the hallucinatory results don't actually have a low probability, in which case the same hallucination will come out in multiple attempts. However, it might be interesting to try to make the model directly aware of these numbers by putting it into a post-message evaluation prompt, because currently the token randomization algorithm sits entirely behind the scenes.
- sqeaky 2y ago
  No. N/A They can't detect if a word is high or low probability because current LLMs can't detect anything. They don't reason as you or I understand it. They are really fancy word predictors and somewhere in the word prediction machinery it does have the probabilities stored but not in anyway accessible to the output. Imagine you are typing on your phone and the word autocomplete comes up and you just keep clicking the next word it suggests. What could you type to get at the inference data in your phone's keyboard? Of course a system that does this could be built, but that isn't how current LLMs or phone keyboards are built.
  - kolinko 2y ago
    What? Of course it is accessible. When you ask an LLM to estimate if a given answer was hallucinated, it converts internal probabilities into a token(s) that represent these probabilities. Here you have an output of chatGPT where it estimates it's probabilities of hallucination - which I would argue are quite close to correct: https://chatgpt.com/share/e/e10cbc17-cc35-432f-872b-cb06170026c6 https://chatgpt.com/share/e/e10cbc17-cc35-432f-872b-cb061700...
    - sqeaky 2y ago
      This is likely a hallucination. You are committing a type error or perhaps succumbing to circular reasoning. No modern LLM can query, let alone understand, the values in its own weights. They are simply inputs for the vector math. If they happen to be correct that is a either a coincidence or because it was trained with data including conversations about its past abilities. Consider this, do you thinks the internal weights are labeled and directly to some attribute of the world outside? If so then present evidence. They are not, so if it does attempt to query its own internal weights how does it know what to query? This could be an interesting academic problem and I would encourage you to present papers on this because last I looked this was and itractable as of 2023 for the much simpler neural networks of simple backpropagation networks from 20 years ago. Also, using ChatGPT as a reference here shows a major deficiency in your judgement. At best this is circular reasoning but likely you simply don't know how to allocate trust or how others do. The argument is X is unreliable then citing X is obvious not going to convince people who distrust X. Additionally, experts
      - sqeaky 2y ago
        To expand on this consider the amount of brains that can directly query the amount of a specific neurotransmitter used between two neurons and interpret what that means. That isn't exactly the same, but a hopefully illustrative analogy. An LLM's output is an emergent property of its structure and all the weights being multiplied together just as an organism's intelligence is an emergent property of its brain structure and communication between neurons. Neither of the emergent parts can currently reliably query the substrate they emerge from for similar reasons. That isn't to say that such an LLM or biological brain couldn't be built, but none presently are.
amelius 2y ago
But they can use tools such as databases and search engines, and I guess also logic engines.
Sharlin 2y ago
I don't think there's any fundamental reason why a LLM shouldn't be able to learn the distinction between being creative and being factual, at least much better than currently. It's just another classification problem (if a challenging one), and classification is what NNs are known for.
- szundi 2y ago
  Source of your disbelief?
  - Sharlin 2y ago
    Call it Bayesian intuition. I’d argue that at this point the specific claim "LLMs cannot solve the confabulation problem" is the one that requires hard evidence. These things forced almost all of us to update our priors on what glorified Markov chains are – and are not – capable of if you throw enough data and compute at them.
- b112 2y ago
  From what I see, there may be two issues here. 1) LLM has data from a large set of sources, but in some cases it has very little data on certain sub-topics. With so few sources of data, it is easier for some weird corner of the internet to have conflicting data, and some LLM models may randomly favour that instead of the valid answer. So not hallucination in this case, instead, a wrong answer from wrong sources. 2) LLM doesn't know, but has seen from human behaviour that just "making shit up, when you don't know" is entirely acceptable for many humans. Here I'm not referring to people saying (as I am) "may be", but instead "No, it's this!" with immense surety, in order to win discussions. Combine this with 'temperature' settings many model interfaces have, and there you go. 2b) Alternatively, #2 is inline with "I believe in flying spaghetti monsters", and other beliefs such as astrology, ghosts, and endless other "there is zero proof, but humans have immense weird data on it". So #2 is solvable by changing temperature, or keeping temperature but saying "Don't make things up". #1 may be temp + random choice of a lower probability pathway. #2b, not sure what to think here. How will LLMs ingest data, and use it to determine what is acceptable logic processing, when a lot of stuff in category 2b has zero proof behind it. And with zero proof, people are absolutely positive of the status of said thing? #2b trains LLMs that "just having a random idea and then even killing people that don't agree, seems OK for humans". (think religious wars, violent acts, burning witches, etc)
mseepgood 2y ago
> They don’t know when they don’t know something. Dunning-Kruger effect
317070 2y ago
> LLMs don’t know when they are hallucinating What makes you think that? The fact that you can ask them to stop hallucinating and that this does reduce hallucinations seems to me like a strong indication of the opposite? That is what the OP's article is about. > They don’t know when they don’t know something. The bigger models tend to be able to articulate some of their limits? See e.g this recent article [0]. > They know facts but they have no reasoning. Is that really true? They can solve many novel tasks that seem to require some level of reasoning. They do not possess the full range of reasoning capabilities most humans have, but do pretty well on the HellaSwag dataset which aims to test reasoning [1], even when test set leakage was not a big problem yet [2]. I have 10+ years of experience in machine learning, and I don't find your claims obvious at all. In fact, I see more and more indications of the opposite. Also if you look mechanistically at how LLMs work inside, there is nothing stopping self-knowledge, reasoning or hallucination-detection. LLMs are very black box. [0] https://arxiv.org/abs/2305.18153 https://arxiv.org/abs/2305.18153 [1] https://deepgram.com/learn/hellaswag-llm-benchmark-guide https://deepgram.com/learn/hellaswag-llm-benchmark-guide [2] https://paperswithcode.com/sota/sentence-completion-on-hellaswag https://paperswithcode.com/sota/sentence-completion-on-hella...
herbst 2y ago
Ask GPT4o to return a percentual score after each answer how likely something is 100% true. Did that for old regional stories and the higher the score the easier I could verify the story or parts of it. It seems to know, at least in some cases
- michaelt 2y ago
  I performed several thousand batch LLM queries a few months back and asked for such scores. My impression was, while the resulting scores weren't entirely random, they were pretty much random when the input was difficult. (I was mapping supermarket product descriptions to packaging materials - which is sometimes very easy, as sometimes it's obvious or the description includes explicit recycling information, and other times almost impossible, as for products like olive oil, glass and plastic bottles are equally plausible)
  - herbst 2y ago
    I guess that's way more complex because there is no one right answer. (Interesting use case btw) And if you ask gpt for an specific answer to something it can't actually know for sure it gets weird fast. Asking it to guess how much of what it told is based on actual things it 'knows' at least seems like a reasonable question for a LLM with only one right answer.
- user-the-name 2y ago
  [dead]
ben_w 2y ago
I was very into the occult as a teenager, thanks to my mother's eclectic attitudes: although theoretically Catholic, she also had Orthodox icons, Hindu statues, runic tarot kits, "magic" crystals, dowsing rods, homeopathic and Bach flower "remedies", books about leylines. I'm as sure as I can be that she didn't know she was wrong; I can definitely tell you that I didn't know that I was.
kolinko 2y ago
This has been disproven by multiple experiments, and also - there is nothing in the transformer architecture that would suggest this. On the contrary - even the most basic classifier neural networks by default produce not a binary answer, but an answer with confidence level.
- lewispollard 2y ago
  And if that confidence level is high enough for it to output an answer, but that answer is wrong or nonsensical, what do we call it?
  - ben_w 2y ago
    In humans? Overconfidence.
  - kolinko 2y ago
    LLMs have no idea if they hallucinate ≠ LLM make no mistakes. The former statement is a bold claim that would require some sort of evidence. The latter nobody really argues with.
    - lewispollard 2y ago
      "Hallucinate" is just a term we've come up with for LLMs making mistakes. If the LLM knew it was making a mistake, wouldn't it just not make it? Of course, it's not possible for an LLM to actually "know" anything in that sense of the word.
mewpmewp2 2y ago
Isn't hallucinating a very wrong word? Hallucinating should be about twisted input misinterpretation. E.g. if the text input you give them was malformed at some level. What they are doing is try to give you an answer based on compressed knowledge, which might be true, but might not because it's compressed and lossy. Because they are trained to bias to give an answer even if it's not 100% clear from compressed knowledge. The larger the model, the less compressed the knowledge and better they would be able to differ when it is something they know vs something that they likely don't.
- seba_dos1 2y ago
  It is, but that ship has sailed. "Confabulation" would be a better word to describe it.
amy-petrik-214 2y ago
>LLMs don’t know when they are hallucinating! Gosh, you sound like me at my last Phish concert. My partner was all like "you're tripping hard but you have no idea". In fairness I was, and I thought I was an AI at the time, or was I ? Anyway you can't really tell a sentience "hey you are hallucinating, stop hallucinating." It's like telling a woman "don't be angry baby", it only makes them angrier - similarly when you tell a person (or AI) "cut it out with the hallucinatin' " usually it causes them to trip harder.

Iulioh 2y ago

Honestly this reminds me a lot of the rituals in Warhammer 40k and the way to deal with "machine spirits". Absurd rituals that shouldn't work on machines they don't understand but for some reason they work.

phil917 2y ago
Oh my god I don't know why I never made that connection until now. That is exactly what it feels like!
- wruza 2y ago
  Same with SD prompts and especially Lora training. The best thing you can do is to count any piece of information as clueless babbling best case and any result as a temporary niche result best case. Because too much depends on randomness and accidental sources of randomness and in it there are whole pockets of multidimensional stability, just until you cross some invisible border you couldn’t even comprehend. This is a natural religion fuel. So when you read any turorial, guide, best practice, etc, and there is not a hard proof of hours of thorough testing behind every claim, you may freely consider most claims religious, because they are.
  - Iulioh 2y ago
    I even remember the fact that a prompt for a CGPT app had a spelling mistake and some people were wondering if that spelling mistake performed better than the right spelling
Terr_ 2y ago
A combination of Plato's allegory of the cave and statistical whack-a-mole: The model only sees token correlations, and if you are lucky the things you type become shadow tokens that change the weights just enough that whatever you didn't want isn't as statistically likely to come out.

armada651 2y ago

I know how you feel, I also often tell my programs not to have bugs.

petargyurov 2y ago
#!/usr/bin/env python3 # no bugs pls
szundi 2y ago
Very soon this is a reality
lillecarl 2y ago
Works to a varying degree, depending on how assertive you are as a developer!
- deleted 2y ago
  [deleted]

Der_Einzige 2y ago

Since we are posting short gists about the absurdities of prompting, I must share my own gist about how garbage prompting is today for LLMs: https://gist.github.com/Hellisotherpeople/45c619ee22aac6865ca4bb328eb58faf https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...

wruza 2y ago
But do LLMs work like this? I don’t understand your code to answer myself, but when I change the last word in the “notebook” tab of tgwebui (free chatless generation mode almost everyone is not aware of), then generation picks it up much quicker than if I change a word somewhere in the middle. This fact hints that alternation will cost much more in inference times, unless I’m missing something in your code. For some reason it doesn’t happen in SD, I believe because each “step” is a separate process that is affected by a restart anyway. That said I agree that LLM “prompt engineering” is much less actual engineering than in SD and is more “prompt writing”. Advanced SD feels like a fighter jet cockpit compared to LLM’s city car dashboard.
- Der_Einzige 2y ago
  As it turns out, I wrote a whole extension for txgui that implemented some of what I wanted... https://github.com/Hellisotherpeople/llm_steer-oobabooga https://github.com/Hellisotherpeople/llm_steer-oobabooga

cubefox 2y ago

> Indeed, we can probe model inner layers and infer if it is "lying"[1] This is hyperbole to the point where it is more false than true. If it was true, OpenAI, Anthropic and Google would already have solved the problem of making models honest. Just adding a reference to some random arxiv paper doesn't automatically justify your bold assertion.

InDubioProRubio 2y ago
The projects themselves specified that lying is okay. Output at all costs, at all parameters, as interpolating is a feature, not a bug. It could just error out at loss of certainty, but then you would stand there, having nothing most of the time.
- ithkuil 2y ago
  Interpolating is an awesome feature, very practical: summarization, style transfer. Instruction following is also useful and unlocks another tier of features. The fuzziness of the model behaviour plays a useful role because natural language is imprecise. The ability to interpolate this remains useful. Once machines understand natural language our own psychology kicks in and assigns properties like "wisdom" and "knowledge" to agents that can "talk well". This may lead us to expect something from tools that haven't been designed to perform a given task. For example our current LLMs are not very good at precise information retrieval because they haven't been designed for that. They happen to be able to perform information retrieval as a side product of the mechanism that teaches them how to speak in the first place. The raw output of the broca area in our brains is probably not something you want to connect directly to the mouth without mediation
  - InDubioProRubio 2y ago
    We need training exercises that teach us, how to think like a llm. Less like a human, more like a free association machine, dreaming a specific association flow, starting from a input. https://en.wikipedia.org/wiki/Ulysses_(novel) https://en.wikipedia.org/wiki/Ulysses_(novel) but with a feeling of dread, as the interpolations pile upon one another.. you can feel the thinness of the construction, as you are several layers out over ground truth on a thin tight rope.

rini17 2y ago

I recommend this excellent presentation: https://www.youtube.com/watch?v=yBL7J0kgldU&t=2819s https://www.youtube.com/watch?v=yBL7J0kgldU&t=2819s and they experimentally proved that LLMs often do know something is not right, but they aren't trained to express it. Because they did not get enough examples of faulty reasoning and, most importantly, how to recover from it. Adding these examples to training set vastly improved the output.

dvh 2y ago

Can you give the response back and ask llm to find sentence which is most likely false?

weinzierl 2y ago

I want to know: Is shouting at a model absurd? Does it really help to use all uppercase and strong language or is it just wishful thinking - inappropriate anthropomorphization even?

Havoc 2y ago

This feels like something one should be able to empirically test. Also I wonder about the word. It’s a fairly recent one to describe this phenomenon. Perhaps a more verbose version works better? Or adding a definition of it to the prompt?

tkgally 2y ago

I just tested ChatGPT 4o and Claude 3.5 Sonnet using the following prompt, with and without the last sentence: “Write a one-paragraph summary of research since 2015 on the use of machine translation in second-language academic writing. Include five in-text citations to previous research, and include full reference information for each citation after the paragraph. Do not hallucinate any of the citations and references.” When the last sentence was not included, four out of five of ChatGPT’s references were hallucinated; the fifth referred to an existing paper but had an incorrect year. All of Claude’s references were hallucinated. When I included the last sentence, ChatGPT did a web search and came up with five genuine citations. Claude, which cannot do web searches, apologized that it could not write the paragraph or provide the references. Maybe a better test of the effectiveness of "Do not hallucinate" would be prompts that might lead to hallucinated claims that the LLM cannot check using web searches, the execution of Python programs, or other external processes. Any ideas?

surfingdino 2y ago
Why can't they not hallucinate by default?
aziaziazi 2y ago
We have another problem then. 2023: LLM hallucinate 2024: hallucinations fixed with "do not hallucinate" prompt 2025: LLM consider their finding on the web trustworthy
Joker_vD 2y ago
Empirical testing? But where's the necessity? It seems an uncommonly roundabout and hopelessly rigamarolish method of getting anywhere. We've got the works of all the old masters — the great AI researchers of the past. We weigh them against each other, balance the disagreements, analyze the conflicting statements, decide which is probably correct, and come to a conclusion. That is the scientific method.
- tkgally 2y ago
  I was led astray by an interesting discussion among Anthropic employees about prompt engineering with Claude. They seem to do a lot of empirical testing of prompts: https://www.youtube.com/watch?v=T9aRN5JkmL8 https://www.youtube.com/watch?v=T9aRN5JkmL8

mrjin 2y ago

Machines don't hallucinate, humans do. Just stop pretending LLMs have any intelligence.

wruza 2y ago
They have some though (even if through different means) and I know people so algorithmic and clueless about real world that an LLM could replace them, ignoring biological and mechanical limitations of the day. In fact, every fifth human is below the bar that we would consider intelligent. It’s not something we speak of often due to social standards, but let’s not forget that our bar for LLMs intelligence is higher than average “low water mark” for humans.

Oras 2y ago

You can also tell LLMs not to tokenise, convert text to embeddings to guess the next token. /s

otikik 2y ago

All they do is hallucinating. Their results are statistic extrapolations from a big corpus of data. For them a "true" fact "looks the same" as a "false, but strongly correlated with the corpus" fact. The piece that would make them distinguish "true" from "strongly correlated with the corpus" is not yet discovered, AFAIK. It would probably need to be a different, context-specific tool. For example, if you are asking a model to write a program for you, the "verification step" would be a test bench for said program.

mewpmewp2 2y ago
What is the differing thing in humans that presumably could categorize us into not "hallucinating"? As far as I understand, every time we are querying our memories we are also slightly changing them, so there's no guarantee when we are trying to remember something that it is a true representation of reality of what happened. See also reliability of witness testimonies. Ultimately all knowledge and storage graph in humans and LLMs has to be compressed heavily and be statistical in its nature with compromises made about the accuracy vs how much you are able to store. We can't store 8k footage of everything we see in our brains and so we use a lot of compression techniques, likely statistical graphs in nature.
- otikik 2y ago
  It seems that we are in agreement. > The piece that would make them distinguish "true" from "strongly correlated with the corpus" is not yet discovered, AFAIK That piece is what would categorize us into not "hallucinating". I don't think it has been discovered, and I don't know wether it exists. It could be just that that our corpus is huge.
  - mewpmewp2 2y ago
    But people also constantly make mistakes, produce invalid output, misremember things, state things incorrectly. Depending on the environment, they do this with varying frequencies. It is always a balance of truth vs action, where if you always safely say "I'm not sure, I'd rather not answer." you will never be able to do anything in life, vs if you always try to answer you will frequently be wrong. LLMs are biased to try to answer in their training sets, so they will try to make an answer up, because otherwise they wouldn't be helpful enough as assistants. You could train LLMs to bias more towards only answering when they are confident, e.g. "Paris is the capital of France", but they wouldn't be as useful then. I would also think it's not really "hallucinating", because "hallucinating" is error in processing the input causing it to be malformed. It implies that the input text you put in as a prompt was somehow invalidly processed to something else. When an LLM is wrong I would rather think it's trying to find an answer from a compressed dataset "hoping" it will be correct, because that's how it's finetuned to do.

yarg 2y ago

Depends on the network; that sort of self evaluation certainly requires recurrency. But I don't think it's impossible - there's a certain psychedelic quality to hallucinations, they tend to over represent certain fundamental features of the thing that they believe they're representing. I remember seeing a bunch of almost kaleidoscopic looking (as far as I remember, all were rotationally symmetrical) images, that had been generated by seeking out and exploring the latent spaces for given classifications. The images that were generated this way always had more (apparently) in common with each other than genuine images of the type that they were classified as; And I highly doubt that the characteristics of those images are so special that a bit of self evaluation could not detect them as a group.

creatonez 2y ago

I made this observation a slightly different way in a previous thread: https://news.ycombinator.com/item?id=41370495 https://news.ycombinator.com/item?id=41370495 https://news.ycombinator.com/item?id=41372016 https://news.ycombinator.com/item?id=41372016

sebstefan 2y ago

>On the other hand, maybe always trying to avoid hallucinations has some other undesired consequences, which model trainers and product managers would like to avoid. (Actually, if you are in a position to know of such undesired consequences, and are free to tell the world about them, I will be really curious to learn more!) I'm not that, but I'd wager attempts to limit hallucinations in the model itself would mess with creative writing exercises Same as if you would ask it to write a short story for kids about a lion that loves green hats, then finish with "Don't hallucinate".

john61 2y ago

Yes, as it is absurd to call a generative statistical model intelligent.

mewpmewp2 2y ago
Isn't people's information processing statistical in nature? E.g. everything we know has to be compressed down heavily, so we can never be totally certain of what we know, yet we presume to be. And the compression can be thought of as statistical graph in its nature.
- john61 2y ago
  We do not know how our brain works.
  - mewpmewp2 2y ago
    How else could information be compressed in human brain besides a similar statistical graph nature and networks of relationships being created representing various different patterns and models of the World.
    - Joker_vD 2y ago
      In some other way which we don't know about. Just because someone can't imagine the universe to be structured in any other way than being built from microscopic Platonic solids, tightly packed and touching each other and filling all of the universe without gaps or empty space, doesn't mean that this is how the world actually operates.
      - mewpmewp2 2y ago
        If you don't know, how do you know that this is not it, and that the understanding is different and that an LLM doesn't understand and we somehow do?
        2y ago

dools 2y ago

When LLMs hallucinate and I point it out they usually get it right the second go. I guess if you could get it to use its own output as input that would be similar, but I think that it’s the additional information in the follow up that makes the difference. Like it can write code, I run the code, there is an error, I paste the error and it corrects the code. So WTF couldn’t it just produce the correct code the first time? Probably for the same reason I can’t. The additional information of the error causes me to re evaluate the code in a different way than when I first wrote it. As such I don’t think you can just tell it “do better the first time” any more than you can tell a human to “do better the first time”. We and they both get additional information from failing.

sebstefan 2y ago
>So WTF couldn’t it just produce the correct code the first time? Probably for the same reason I can’t. We still know the gist of how they work and it's not like you do, and you can see when they fail that they don't fail for the same reasons either I've seen GPT 3.5 fail to answer some questions correctly and subsequently (in another tab) get it right/choose not to answer if you just add the line "don't hallucinate" I like the article's idea for why that is >Presumably, "retrieving from memory" and "improvising an answer" are two different model behaviors, which use different internal mechanisms. Indeed, we can probe model inner layers and infer if it is "lying"1 or if "the question is unanswerable"2. These are very much related to "hallucinations".

torginus 2y ago

Afaik LLMs at their heart employ supervised training - as in, given a certain input-output distribution pairs in the dataset, it learns to recreate the given outputs from the inputs. For all the inputs it hasn't seen, its output is essentially undefined, but hoping that the distribution of the novel inputs matches the outputs means that it will output truthful info. When it doesn't, it hallucinates. Since LLMs cannot distinguish the difference between the predicted distribution and the real one, it cannot be told to not hallucinate, in one pass. What actually could work is getting the output of a given LLM, and feeding it into another one to try and check for divergences to check for hallucinations.

bell-cot 2y ago

Like anything else you might do with balky, complex equipment - try it, and see if you get noticeably better results.

karmakaze 2y ago

Hot-take: of course it could/should work. We write about what LLM hallucination is, why it's bad, and examples of it. An LLM of course consumes this and can direct its responses based on prompts that mention it. It's not much different than prefacing with "respond as an expert in the field of X" that categorizes different kinds of responses as desirable or not based on some dimension. What would be dumb is asking an LLM not to hallucinate when the only information available is a definition of it in describing how an LLM works. We're not quite there yet.

djaouen 2y ago

I suspect telling a model to "not hallucinate" is about as effective as telling a dog not to bark.