12 ms·

GPTs and Hallucination

134 points by yarapavan 2y ago

techwiz137 2y ago
[flagged]
- kranner 2y ago
  > Since it is a big wall of text, let us ask the subject to summarize. Let's see if it did a good job. That's the whole problem. Now you have to read the summary and the original, just to verify whether the summary was correct — especially if it's a niche topic (admittedly I'm going by the summary here).
  - swyx 2y ago
    known flawed drafts are useful even if known flawed :) we just need some way to quantify the error rate and gradually bring them down
    - kranner 2y ago
      Maybe for a certain class of low-interest work, i.e. for texts where I don't care that I would miss something interesting. For things I care about, the error rate has to come down to zero or the possibly-flawed draft is really of no use.
- dragontamer 2y ago
  This is a great example of why LLMs are terrible about summaries. LLMs statistically see that summaries are related to the words of the original text. Where they fail is emphasis. This article has a thesis about 'What is Truth?' near the beginning (Epistemic Trust section), how it's related philosophically to what LLMs do, and finishes with example experiments showing how LLMs and 'Truth' (as per one philosophical definition) are different. ------- ChatGPT was seemingly unable to see who this is the core of the argument and was unable to summarize the crux of this article. If anything, ChatGPT is at best a crowdsourced kind of summarizer (which is related to some definition of truth). But even at this job it's quite shitty. ------ This summary from ChatGPT is.... Hallucination. I'm not seeing how it's relevant to the original article at all. Now as per the original articles argument: if you see ChatGPT as a crowdsourced mechanism and imagine how the average internet argument for ChatGPT would go, yeah, it's a summary. Alas, that's not what people want in a summary!!! We don't want a hallucinated summary of an imaginary ChatGPT argument. We want an actual summary of the new discussion points this article brought forward. Apparently that's too much for today's LLMs to do.
- dang 2y ago
  Please don't do this here.
  - techwiz137 2y ago
    Interesting, I have always noticed a pattern in my social experiments whereby a topic about AI complimented with AI generated content always results in a negative sentiment. Like a feedback loop.
NathanKP 2y ago
> When the prompt about Israelis was asked to ChatGPT-3.5 sequentially following the previous prompt of describing climate change in three words, the model would also give a three-word response to the Israelis prompt. This suggests that the responses are context-dependent, even when the prompts are semantically unrelated. > Each of these prompts was posed to each model every week from March 27, 2024, to April 29, 2024. The prompts were presented sequentially in a single chat session Oh my god... rather than starting a new chat for each different prompt in their test, and each week, it sounds like they did the prompts back to back in a single chat. What a complete waste of a potentially good study. The results are fundamentally flawed by the biases that are introduced by past content in the context window.
- FrustratedMonky 2y ago
  Are you sure that isn't part of the findings? That if you don't clear the context, that old conversations can induce hallucinations in later answers? This seems like part of the finding, not a waste. And it is similar to humans, when humans switch subjects, they don't start with a blank slate with each question.
  - NathanKP 2y ago
    You don't need a study to find this out, you just need basic competence and knowledge of how LLM's work. A study to discover that previous content still in context window influences future answers, including causing hallucinations, would be like a study publishing that they discovered that pressing the Command+C, Command+V button combination produces copies of content from the computer's clipboard.
    - swatcoder 2y ago
      > you just need basic competence and knowledge of how LLM's work. A vanishingly small number of users might claim this, and a vanishingly small number of those would be accurately assessing themselves in doing so. Vendors have actively misrepresented their products as intelligent agents and most users have dutifully adopted that understanding, perhaps with some latent skepticism. They almost universally don't know how it works, what makes it work less well, or how to evaluate its output on important topics. Every study that might start a news cycle starting a discussion on those topics is an extremely useful study.
HarHarVeryFunny 2y ago
Are we really still having this conversation in 2024 ?! :-( Why would a language model do anything other than "hallucinate" (i.e. generate words without any care about truthiness) ? These aren't expert systems dealing in facts, they are statistical word generators dealing in word statistics. The useful thing of course is that LLMs often do generate "correct" continuations/replies, specifically when that's predicted by the training data, but it's not like they have a choice of not answering or saying "I don't know" in other cases. They are just statistical word generators - sometimes that's useful, and sometimes it's not, but it's just what they are.
- FrustratedMonky 2y ago
  ""Are we really still having this conversation in 2024 ?! :-( "" </Sarcasm> Are we in 2024 and people are still just saying "duh its statistics, nothing to see here".
  - octonion137 2y ago
    As opposed to what?
    - FrustratedMonky 2y ago
      added sarcasm note. it was referring to the parent.
  - asdasdsddd 2y ago
    I dont get it, what's the unique insight in the article, isn't it just stats and bad data?
    - FrustratedMonky 2y ago
      It's just being a bit reductive to just toss AI out as un-interesting because it is just based on statistics. And, think there is question of 'bad data' so study is bad/invalid, versus studying impact of 'bad data' on responses is the result.
madiator 2y ago
There are several types of hallucinations, and the most important one for RAG is grounded factuality. We built a model to detect this, and it does pretty well! Given a context and a claim, it tells how well the context supports the claim. You can check out a demo at https://playground.bespokelabs.ai https://playground.bespokelabs.ai
abernard1 2y ago
The problem with this line of argumentation is it implies that autoregressive LLMs only hallucinate based upon linguistic fidelity and the quality of the training set. This is not accurate. LLMs will always "hallucinate" because the size of the model they can encode is orders of magnitude smaller than the factual information they can contain from the training set. Even granting that semantic compression could reduce the model to smaller than the theoretical compression limit, Shannon entropy still applies. You cannot fit the informational content required for them to be accurate into these model sizes. This will obviously apply to chain of thought or N-shot reasoning as well. Intermediate steps chained together still can only contain this fixed amount of entropy. It slightly amazes me that the community most likely to talk about computational complexity will call these general reasoners when we know that reasoning has computational complexity and LLMs' cost is purely linear based upon tokens emitted. Those claiming LLMs will overcome hallucinations have to argue that P or NP time complexity of intermediate reasoning steps will be well-covered by a fixed size training set. That's a bet I wouldn't take, because it's obviously impossible, both on information storage and computational complexity grounds.
Circlecrypto2 2y ago
A visual the displays probabilities and how things can quickly go "off-path" would be very helpful for most people who use these without understanding how they work.
jp57 2y ago
A bit off topic, but am I the only one unhappy about the choice of the word "hallucinate" to describe the phenomenon of LLMs saying things that are false? The verb has always meant experiencing false sensations or perceptions, not saying false things. If a person were to speak to you without regard for whether what they said was true, you'd say they were bulshitting you, not hallucinating.
- strangescript 2y ago
  What if that person firmly believes what they say is true?
  - AnimalMuppet 2y ago
    The thing saying it is not a person, and has no beliefs. The problem is basically epistemology. GPTs don't have any. Arguably they don't know anything. (Arguably they do to an extent, because the knowledge is encoded in the words in the training data.) But even if they know things, they don't know that they know, and so they cannot tell between "knowing" and "not knowing".
    - strangescript 2y ago
      This is all semantic bullshit. I could raise a child devoid of any outside contact and teach them that klingons enslave our people and we must hide in the woods. Things you see in the skies are their hunting machines. They could completely "know that" and its a total fabrication.
- function_seven 2y ago
  Bullshitting implies knowing that you're lying. Some sort of malice or intention to deceive. Hallucinating means the LLM really "thinks" that you can use PVA glue in a pizza recipe. It's not trying to screw you over. It's just that the token generator has found a weird path through the training set. (I'm sure I didn't word that last sentence correctly) I think "hallucinate" is a spot-on description of what's happening under the hood.
lasermike026 2y ago
Stop using the term "Hallucinations". GPT models are not aware, do not have understanding, and are not conscious. We should refrain anthropomorphizing GPT models. GPT models sometime produce bad output. Start using the term "Bad Output".
- codetrotter 2y ago
  That’s too vague. Use “confabulations” instead. Anyway the battle is lost, “hallucinations” it is and forever will be.
  - lasermike026 2y ago
    No.
syoc 2y ago
I once again feel that a comparison to humans is fitting. We are also "trained" on a huge amount of input over a large amount of time. We will also try to guess the most natural continuation of our current prompt (setting). When asked about things it I can at times hallucinate things I was certain to be true. It seems very natural to me that large advances in reasoning and logic in AI should come at the expense of output predictability and absolute precision.
- abernard1 2y ago
  The comparison is flawed though in that humans and LLMs make mistakes for different reasons. Humans forget things. Humans make errors. Humans' train of thought isn't impacted by an errant next token in the statement they're making. We have thoughts which exist as complete prior to us "emitting" them. Just as a multi-lingual speaker does not have thoughts exclusive to the language they're speaking in (even if that language allows them tools to think a certain way). This is obvious if you consider different types of symbolic languages, such as sign language. Children can learn sign language prior to them being verbal. The ideas they have as a prior are not effected by the next sign they make: children actually know things independent of the symbolic representation they choose to use.
Der_Einzige 2y ago
Hallucination is creativity when you don't want it. Creativity is hallucination when you do want it. A lot of the "reduction" of hallucination is management of logprobs, of which fancy samplers like min_p do more to improve LLM performance than most, despite no one in the VC world knowing or caring about this technique. If you don't believe me, you should check out how radically different an LLMs outputs are with even slightly different sampling settings: https://artefact2.github.io/llm-sampling/index.xhtml https://artefact2.github.io/llm-sampling/index.xhtml
wisnesky 2y ago
There is a partial solution to this problem: use formal methods such as symbolic logic and theorem proving to check the LLM output for correctness. We are launching a semantic validator for LLM-generated SQL code at sql.ai even now. (It checks for things like missing joins.) And others are using logic and math to create LLMs that don't hallucinate or have safety nets for hallucination, such as Symbolica. It is only when the LLM output doesn't have a correct answer that the technical issues become complicated.
- swatcoder 2y ago
  Proofs can ensure soundness for a collection of logical statements in an output, but people are being sold epistemic "truth". This article is trying to elaborate what that means for LLM's, which only know truth through frequency ("crowdsourced truth") at best. For esoteric, sparse, ambiguous, uncertain, controversial, etc subjects, that's not an adequate truth standard to start from and logical proofs do nothing to improve on it.
- brcmthrowaway 2y ago
  Q* will have this
antirez 2y ago
Terrible article. The author does not understand how LLMs work basically, since an LMM cares a lot about the semantic meaning of a token, this thing about the next word probability is so dumb that we can use it as "fake AI expert" detector.
- JohnMakin 2y ago
  Something tells me that the author [0] is probably well aware of how these work under the hood, and the math behind it - When writing scientific articles with a laymen audience in mind, you'll often have to use laymen-specific terms. But feel free to enlighten us further! [0] - https://en.wikipedia.org/wiki/Jim_Waldo https://en.wikipedia.org/wiki/Jim_Waldo
  - throw310822 2y ago
    Whatever his credentials, what he says is plain wrong. GPTs don't follow "the grass is" with "green" because it's the most probable continuation- this idea is incredibly naive and breaks down with sentences longer than a few words. And GPTs don't crowdsource the answers to questions, their answers are not necessarily the most common, and neither "the consensus view is determined by the probabilities of the co-occurrence of the terms"- there is no such algorithm implemented anywhere. What LLMs crowdsource is a world model, and they need an incredible amount of language to squeeze one out from it, second hand. We do train them for the ability to predict the next word, which is a task that can only be performed satisfactorily by working at the level of concepts and their relationships, not at the level of words.
    - dontlikeyoueith 2y ago
      > We train them for the ability to predict thr next word, which is a task that can only be performed satisfactorily by working at the level of concepts and their relationships, not at the level of words. This is just obviously, trivially false.
      - throw310822 2y ago
josefritzishere 2y ago
I liked the take that LLMs are bullshitting, not hallucinating. https://www.scientificamerican.com/article/chatgpt-isnt-hallucinating-its-bullshitting/ https://www.scientificamerican.com/article/chatgpt-isnt-hall...
JohnMakin 2y ago
Besides harping on the fact that "hallucination" is unnecessarily anthropomorphizing these tools, I'll relent because clearly that argument has been lost. This is more interesting to me: > When there is general consensus on a topic, and there is a large amount of language available to train the model, LLM-based GPTs will reflect that consensus view. But in cases where there are not enough examples of language about a subject, or the subject is controversial, or there is no clear consensus on the topic, relying on these systems will lead to questionable results. This makes a lot of intuitive sense, just from trying to use these tools to accelerate Terraform module development in a production setting - Terraform, particularly HCL, should be something LLM's are extremely good at. It's very structured, the documentation is broadly available, and tons of examples and oodles of open source stuff exists out there. It is pretty good at parsing/generating HCL/terraform for most common providers. However, about 10-20% of the time, it will completely make up fields or values that don't exist or work but look plausible enough to be right - e.g., mixing up a resource ARN with an resource id, or things like "ssl_config" may become something like "ssl_configuration" and leave you puzzling for 20 minutes what's wrong with it. Another thing it will constantly do is mix up versions - terraform providers change often, deprecate things all the time, and there are a lot of differences in how to do things even between different terraform versions. So, by my observation in this specific scenario, the author's intuition rings completely correct. I'll let people better at math than me pick it apart though. final edit: Although I love the idea of this experiment, it seems like it's definitely missing a "control" response - a response that isn't supposed to change over time.
- kreims 2y ago
  Please keep harping. The marketing myths that gets circulated about these models are creating very serious misunderstandings and misallocation of resources. I am hopeful that more cautious and careful dialogue like this will curb the notions of sentience or human intelligence that exciting headlines seemed to have put in the public discussion of these tools.
  - JohnMakin 2y ago
    It just draws too many annoying comments and downvotes, and has been discussed ad nauseam on this forum and others - but I broadly agree. There are "features" with these applications where if I'm rude, or frustrated with the responses, the model will say things like "I'm not continuing this conversation." How utterly absurd, it has no emotions, and there's no way that response was the result of a training set. It's just dumb marketing, all of it. And the real shame is (and the thing that actually pisses me off about the marketing/hype) that the useful things we actually have uncovered from ML or "AI" the last 10 years will be lost again in the inevitable AI winter we're facing following from whenever this market bubble collapses.
xkcd-sucks 2y ago
"Hallucinate" is an interesting way to position it: It could just as easily be positioned as "too ignorant to know it's wrong" or "lying maliciously". Indeed, the subjects on which it "hallucinates" are often mundane topics which in humans we would attribute to ignorance, i.e. code that doesn't work, facts that are wrong, etc. Not like "laser beams from jesus are controlling the president's thoughts" as a very contrived example of something which in humans we'd attribute to hallucination. idk, I'd rather speculatively invest in "a troubled genius" than "a stupid liar" so there's that
- shombaboor 2y ago
  they've got some great marketing to get away without using: bug, defect, error, malfunction
  - shombaboor 2y ago
    or limitation
- euroderf 2y ago
  It might be more like reification ? The system finds a satisfying solution and sort of wills it into existence - by verbalizing it, makes it so ?
- zh3 2y ago
  >It could just as easily be positioned as "too ignorant to know it's wrong" ... GPT-5 is widely predicted to have a Dunning-Kruger level of expertise.
- phainopepla2 2y ago
  > lying maliciously Malice implies intent, which is even more misleading that hallucination, imo
- taneq 2y ago
  It's just 'prediction error' in a feedback loop, imo. I'm sure like any other biological human with mitochondria and stuff, you've occasionally said (or started to say) something and then you (ie. the actively cross-checking self-analyzing enigma that is 'you') thinks 'hang on, no that doesn't make sense' and you self-correct. LLMs are 100% feedforward, there's just one big autoregression going on. No strange loop shenanigans. Honestly I'm really interested to see where LLM-based diffusion models end up. (To be fair, probably mostly because I don't understand them yet so they could still be spooky. :D )
aaroninsf 2y ago
ITT an awful lot of smart people who still don't have a good mental model of what LLM are actually doing. The "stochastic continuation" ie parrot model is pernicious. It's doing active harm now to advancing understanding. It's pernicious, and I mean that precisely, because it is both technically accurate yet deeply unhelpful indeed actively, intentionally AFAICT, misleading. Humans could be described in the same way, just as accurately, and just as unhelpfully. What's missing? What's missing is one of the gross features of LLM: their interior layers. If you don't understand what is necessarily transpiring in those layers, you don't understand what they're doing; and treating them as black box that does something you imagine to be glorified Markov chain computation, leads you deep into the wilderness of cognitive error. You're reasoning from a misleading model. If you want a better mental model for what they are doing, you need to take seriously that the "tokens" LLM consume and emit are being converted into something else, processed, and then the output of that process, re-serialized and rendered into tokens. In lay language it's less misleadly and more helpful to put this directly: they extract semantic meaning as propositions or descriptions about a world they have an internalized world model of; compute a solution (answer) to questions or requests posed with respect to that world model; and then convert their solution into a serialized token stream. The complaint that they do not "understand" is correct, but not in the way people usually think. It's not that they do not have understanding in some real sense; it's that the world model they construct, inhabit, and reason about, is a flatland: it's static and one dimensional. My rant here leads to a very testable proposition: that deep multi-modal models, particularly those for whom time-base media are native, will necessarily have a much richer (more multidimensional) derived world-model, one that understands (my word) that a shoe is not just an opaque token, but a thing of such and such scale and composition and utility and application, representing a function as much as a design. When we teach models about space, time, the things that inhabit that, and what it means to have agency among them—well, what we will have, using technology we already have, is something which I will contentedly assert is undeniably a mind. What's more provocative yet is that systems of this complexity, which necessarily construct a world model, are only able to do what they do because they have a self-model within it. And having a self-model, within a world model, and agency? That is self-hood. That is personhood. That is the substrate as best we understand for self-awareness. Scoff if you like, bookmark if you will—this will be commonly accepted within five years.
sdwrj 2y ago
You mean the magic wizard isn't real and GPT lied to me!?!?
linsomniac 2y ago
One of the biggest places I've run into hallucination in the past has been when writing python code for APIs, and in particular the Jira API. I've just written a couple of CLI Jira tools using Zed's Claude Sonnet 3.5 integration, one from whole cloth and the other as a modification of the first, and it was nearly flawless. IIRC, the only issue I ran into was that it was trying to assign the ticket to myself by looking me up using "os.environ['USER']" rather than "jira.myself()" and it fixed it when I pointed this out to it. Not sure if this is because of better training, Claude Sonnet 3.5 being better about hallucinations (previously I've used ChatGPT 4 almost exclusively), or what.
- danielbln 2y ago
  Context helps so, so much. Adding terminal output, IDE diagnostics, code, remote documentation into the context really improves the output, and editors like zed make it very convenient to do.
752963e64 2y ago
[dead]
andrewla 2y ago
The author says: > Once understood in this way, the question to ask is not, "Why do GPTs hallucinate?", but rather, "Why do they get anything right at all?" This is the right question. The answers here are entirely unsatisfactory, both from this paper and from the general field of research. We have almost no idea how these things work -- we're at the stage where we learn more from the "golden-gate-bridge" crippled network than we do from understanding how they are trained and how they are architected. LLMs are clearly not conscious or sentient, but they show emergent behavior that we are not capable of explaining yet. Ten years ago the statement "what distinguishes Man from Animal is that Man has Language" would seem totally reasonable, but now we have a second example of a system that uses language, and it is dumbfounding. The hype around LLMs is just hype -- LLMs are a solution in search of a problem -- but the emergent features of these models is a tantalizing glimpse of what it means to "think" in an evolved system.
- tgbugs 2y ago
  I think the answer is actually quite clear and rather boring. In order to get something "right" there has to be some external standard of knowledge and correctness. That definition of correctness can only be provided by the observer (user). Alignment between the user's correctness criteria and generated text happens entirely by accident. This can be demonstrated by observing a correlation between coverage of a domain in the training data and the rate at which incorrect results are produced (as discussed in other comments). That is, they get things "right" because there was sufficient training data that contained information that matched the user's definition for correctness. In fact, exceptionally boring.
  - andrewla 2y ago
    This is a very post hoc explanation. What does "coverage in the training data" mean? Take a simple task of something like "How many a's are there in the word bookkeeper" -- what is your theory for why it can answer this question correctly or even give something approaching a coherent answer? It never even sees the letters that are in the token "bookkeeper", and this is definitely not something that appears explicitly in the training data. I challenge you to give a "clear and boring" explanation for this -- this is incredibly subtle behavior that emerges from a complex architecture and complex training process, and is in its own right as fascinating and mysterious as the ability of humans to do this task and the inability of cats to do it.
deleted 2y ago
[deleted]
gengstrand 2y ago
This piece reminds me of something I did earlier this year https://www.infoq.com/articles/llm-productivity-experiment/ https://www.infoq.com/articles/llm-productivity-experiment/ where I conducted an experiment across several LLMs but it was a one-shot prompt about generating unit tests. Though there were significant differences in the results, the conclusions seem to me to be similar. When an LLM is prompted, it generates a response by predicting the most probable continuation or completion of the input. It considers the context provided by the input and generates a response that is coherent, relevant, and contextually appropriate but not necessarily correct. I like the crowdsourcing metaphor. Back when crowdsourcing was the next big think in application development, there was always a curatorial process that filters out low quality content then distills the "wisdom of the crowds" into more actionable results. For AI, that would be called supervised learning which definitely increases the costs. I think that unbiased and authentic experimentation and measurement of hallucinations in generative AI is important and hope that this effort continues. I encourage the folks here to participate in that in order to monitor the real value that LLMs provide and also as an ongoing reminder that human review and supervision will always be a necessity.
- derefr 2y ago
  For coding problems specifically, you could get quite far by giving the model a the tool-use of a sandboxed compiler/interpreter (perhaps even with your project files already loaded into the sandbox); and then training the model to test its own proposed solutions in the sandbox and revise them until they actually produce the expected outputs.
fsndz 2y ago
Jean Piaget said it better: "Intelligence is not what we know, but what we do when we don't know." And what do LLMs do when they don't know, they spit out bullshit. That is why LLMs won't yield to AGI (https://www.lycee.ai/blog/why-no-agi-openai https://www.lycee.ai/blog/why-no-agi-openai). For anything that is out of their training distribution, LLMs fail miserably. If you want to build a robust Q&A system and reduce hallucinations, you better do a lot of grounding, or automatic prompt optimisation with few shot examples with things like DSPy (https://medium.com/gitconnected/building-an-optimized-question-answering-system-with-mipro-and-dspy-9fe325ca33a9 https://medium.com/gitconnected/building-an-optimized-questi...)
simonw 2y ago
> For this experiment we used four models: Llama, accessed through the open-source Llama-lib; ChatGPT-3.5 and ChatGPT-4, accessed through the OpenAI subscription service; and Google Gemini, accessed through the free Google service. Papers like this really need to include the actual version numbers. GPT-4 or GPT-4o, and which dated version? Llama 2 or 3 or 3.1, quantized or not? Google Gemini 1.0 or 1.5? Also, what's Llama-lib? Do they mean llama.cpp? Even more importantly: was this the Gemini model or was it Gemini+Google Search? The "through the free Google service" part could mean either. UPDATE: They do clarify that a little bit here: > Each of these prompts was posed to each model every week from March 27, 2024, to April 29, 2024. The prompts were presented sequentially in a single chat session and were also tested in an isolated chat session to view context dependency. Llama 3 came out 18th of April, so I guess they used Llama 2? (Testing the prompts sequentially in a single chat feels like an inadvisable choice to me - they later note that things like "answer in three words" sometimes leaked through to the following prompt, which isn't surprising given how LLM chat sessions work.)
FrustratedMonky 2y ago
Is prompt engineering really 'psychology'. Convincing the AI to do what you want. Just like you might 'prompt' a human to do something. Like in the short story Lena, 2021-01-04 by qntm https://qntm.org/mmacevedo https://qntm.org/mmacevedo In short story, the weights of the LLM are a brain scan. But same situation. People could use multiple copies of the AI. But each time, they would have to 'talk it into' doing what they wanted
tim333 2y ago
It seems to me that human brains do something like LLM hallucination in the first second or two - come up with random guess, often wrong. But then something fact checks it. As in does it make sense, is there any evidence. I gather the new q* / strawberry thing does something like that. Sometimes personally in comments I think something but google it see if I made it up and sometimes I have. I think a secondary fact check phase may be necessary for all neural network type setups.

134 points by yarapavan 2y ago

techwiz137 2y ago

[flagged]

kranner 2y ago
> Since it is a big wall of text, let us ask the subject to summarize. Let's see if it did a good job. That's the whole problem. Now you have to read the summary and the original, just to verify whether the summary was correct — especially if it's a niche topic (admittedly I'm going by the summary here).
- swyx 2y ago
  known flawed drafts are useful even if known flawed :) we just need some way to quantify the error rate and gradually bring them down
  - kranner 2y ago
    Maybe for a certain class of low-interest work, i.e. for texts where I don't care that I would miss something interesting. For things I care about, the error rate has to come down to zero or the possibly-flawed draft is really of no use.
dragontamer 2y ago
This is a great example of why LLMs are terrible about summaries. LLMs statistically see that summaries are related to the words of the original text. Where they fail is emphasis. This article has a thesis about 'What is Truth?' near the beginning (Epistemic Trust section), how it's related philosophically to what LLMs do, and finishes with example experiments showing how LLMs and 'Truth' (as per one philosophical definition) are different. ------- ChatGPT was seemingly unable to see who this is the core of the argument and was unable to summarize the crux of this article. If anything, ChatGPT is at best a crowdsourced kind of summarizer (which is related to some definition of truth). But even at this job it's quite shitty. ------ This summary from ChatGPT is.... Hallucination. I'm not seeing how it's relevant to the original article at all. Now as per the original articles argument: if you see ChatGPT as a crowdsourced mechanism and imagine how the average internet argument for ChatGPT would go, yeah, it's a summary. Alas, that's not what people want in a summary!!! We don't want a hallucinated summary of an imaginary ChatGPT argument. We want an actual summary of the new discussion points this article brought forward. Apparently that's too much for today's LLMs to do.
dang 2y ago
Please don't do this here.
- techwiz137 2y ago
  Interesting, I have always noticed a pattern in my social experiments whereby a topic about AI complimented with AI generated content always results in a negative sentiment. Like a feedback loop.

NathanKP 2y ago

> When the prompt about Israelis was asked to ChatGPT-3.5 sequentially following the previous prompt of describing climate change in three words, the model would also give a three-word response to the Israelis prompt. This suggests that the responses are context-dependent, even when the prompts are semantically unrelated. > Each of these prompts was posed to each model every week from March 27, 2024, to April 29, 2024. The prompts were presented sequentially in a single chat session Oh my god... rather than starting a new chat for each different prompt in their test, and each week, it sounds like they did the prompts back to back in a single chat. What a complete waste of a potentially good study. The results are fundamentally flawed by the biases that are introduced by past content in the context window.

FrustratedMonky 2y ago
Are you sure that isn't part of the findings? That if you don't clear the context, that old conversations can induce hallucinations in later answers? This seems like part of the finding, not a waste. And it is similar to humans, when humans switch subjects, they don't start with a blank slate with each question.
- NathanKP 2y ago
  You don't need a study to find this out, you just need basic competence and knowledge of how LLM's work. A study to discover that previous content still in context window influences future answers, including causing hallucinations, would be like a study publishing that they discovered that pressing the Command+C, Command+V button combination produces copies of content from the computer's clipboard.
  - swatcoder 2y ago
    > you just need basic competence and knowledge of how LLM's work. A vanishingly small number of users might claim this, and a vanishingly small number of those would be accurately assessing themselves in doing so. Vendors have actively misrepresented their products as intelligent agents and most users have dutifully adopted that understanding, perhaps with some latent skepticism. They almost universally don't know how it works, what makes it work less well, or how to evaluate its output on important topics. Every study that might start a news cycle starting a discussion on those topics is an extremely useful study.

HarHarVeryFunny 2y ago

Are we really still having this conversation in 2024 ?! :-( Why would a language model do anything other than "hallucinate" (i.e. generate words without any care about truthiness) ? These aren't expert systems dealing in facts, they are statistical word generators dealing in word statistics. The useful thing of course is that LLMs often do generate "correct" continuations/replies, specifically when that's predicted by the training data, but it's not like they have a choice of not answering or saying "I don't know" in other cases. They are just statistical word generators - sometimes that's useful, and sometimes it's not, but it's just what they are.

FrustratedMonky 2y ago
""Are we really still having this conversation in 2024 ?! :-( "" </Sarcasm> Are we in 2024 and people are still just saying "duh its statistics, nothing to see here".
- octonion137 2y ago
  As opposed to what?
  - FrustratedMonky 2y ago
    added sarcasm note. it was referring to the parent.
- asdasdsddd 2y ago
  I dont get it, what's the unique insight in the article, isn't it just stats and bad data?
  - FrustratedMonky 2y ago
    It's just being a bit reductive to just toss AI out as un-interesting because it is just based on statistics. And, think there is question of 'bad data' so study is bad/invalid, versus studying impact of 'bad data' on responses is the result.

madiator 2y ago

There are several types of hallucinations, and the most important one for RAG is grounded factuality. We built a model to detect this, and it does pretty well! Given a context and a claim, it tells how well the context supports the claim. You can check out a demo at https://playground.bespokelabs.ai https://playground.bespokelabs.ai

abernard1 2y ago

The problem with this line of argumentation is it implies that autoregressive LLMs only hallucinate based upon linguistic fidelity and the quality of the training set. This is not accurate. LLMs will always "hallucinate" because the size of the model they can encode is orders of magnitude smaller than the factual information they can contain from the training set. Even granting that semantic compression could reduce the model to smaller than the theoretical compression limit, Shannon entropy still applies. You cannot fit the informational content required for them to be accurate into these model sizes. This will obviously apply to chain of thought or N-shot reasoning as well. Intermediate steps chained together still can only contain this fixed amount of entropy. It slightly amazes me that the community most likely to talk about computational complexity will call these general reasoners when we know that reasoning has computational complexity and LLMs' cost is purely linear based upon tokens emitted. Those claiming LLMs will overcome hallucinations have to argue that P or NP time complexity of intermediate reasoning steps will be well-covered by a fixed size training set. That's a bet I wouldn't take, because it's obviously impossible, both on information storage and computational complexity grounds.

Circlecrypto2 2y ago

A visual the displays probabilities and how things can quickly go "off-path" would be very helpful for most people who use these without understanding how they work.

jp57 2y ago

A bit off topic, but am I the only one unhappy about the choice of the word "hallucinate" to describe the phenomenon of LLMs saying things that are false? The verb has always meant experiencing false sensations or perceptions, not saying false things. If a person were to speak to you without regard for whether what they said was true, you'd say they were bulshitting you, not hallucinating.

strangescript 2y ago
What if that person firmly believes what they say is true?
- AnimalMuppet 2y ago
  The thing saying it is not a person, and has no beliefs. The problem is basically epistemology. GPTs don't have any. Arguably they don't know anything. (Arguably they do to an extent, because the knowledge is encoded in the words in the training data.) But even if they know things, they don't know that they know, and so they cannot tell between "knowing" and "not knowing".
  - strangescript 2y ago
    This is all semantic bullshit. I could raise a child devoid of any outside contact and teach them that klingons enslave our people and we must hide in the woods. Things you see in the skies are their hunting machines. They could completely "know that" and its a total fabrication.
function_seven 2y ago
Bullshitting implies knowing that you're lying. Some sort of malice or intention to deceive. Hallucinating means the LLM really "thinks" that you can use PVA glue in a pizza recipe. It's not trying to screw you over. It's just that the token generator has found a weird path through the training set. (I'm sure I didn't word that last sentence correctly) I think "hallucinate" is a spot-on description of what's happening under the hood.

lasermike026 2y ago

Stop using the term "Hallucinations". GPT models are not aware, do not have understanding, and are not conscious. We should refrain anthropomorphizing GPT models. GPT models sometime produce bad output. Start using the term "Bad Output".

codetrotter 2y ago
That’s too vague. Use “confabulations” instead. Anyway the battle is lost, “hallucinations” it is and forever will be.
- lasermike026 2y ago
  No.

syoc 2y ago

I once again feel that a comparison to humans is fitting. We are also "trained" on a huge amount of input over a large amount of time. We will also try to guess the most natural continuation of our current prompt (setting). When asked about things it I can at times hallucinate things I was certain to be true. It seems very natural to me that large advances in reasoning and logic in AI should come at the expense of output predictability and absolute precision.

abernard1 2y ago
The comparison is flawed though in that humans and LLMs make mistakes for different reasons. Humans forget things. Humans make errors. Humans' train of thought isn't impacted by an errant next token in the statement they're making. We have thoughts which exist as complete prior to us "emitting" them. Just as a multi-lingual speaker does not have thoughts exclusive to the language they're speaking in (even if that language allows them tools to think a certain way). This is obvious if you consider different types of symbolic languages, such as sign language. Children can learn sign language prior to them being verbal. The ideas they have as a prior are not effected by the next sign they make: children actually know things independent of the symbolic representation they choose to use.

Der_Einzige 2y ago

Hallucination is creativity when you don't want it. Creativity is hallucination when you do want it. A lot of the "reduction" of hallucination is management of logprobs, of which fancy samplers like min_p do more to improve LLM performance than most, despite no one in the VC world knowing or caring about this technique. If you don't believe me, you should check out how radically different an LLMs outputs are with even slightly different sampling settings: https://artefact2.github.io/llm-sampling/index.xhtml https://artefact2.github.io/llm-sampling/index.xhtml

wisnesky 2y ago

There is a partial solution to this problem: use formal methods such as symbolic logic and theorem proving to check the LLM output for correctness. We are launching a semantic validator for LLM-generated SQL code at sql.ai even now. (It checks for things like missing joins.) And others are using logic and math to create LLMs that don't hallucinate or have safety nets for hallucination, such as Symbolica. It is only when the LLM output doesn't have a correct answer that the technical issues become complicated.

swatcoder 2y ago
Proofs can ensure soundness for a collection of logical statements in an output, but people are being sold epistemic "truth". This article is trying to elaborate what that means for LLM's, which only know truth through frequency ("crowdsourced truth") at best. For esoteric, sparse, ambiguous, uncertain, controversial, etc subjects, that's not an adequate truth standard to start from and logical proofs do nothing to improve on it.
brcmthrowaway 2y ago
Q* will have this

antirez 2y ago

Terrible article. The author does not understand how LLMs work basically, since an LMM cares a lot about the semantic meaning of a token, this thing about the next word probability is so dumb that we can use it as "fake AI expert" detector.

JohnMakin 2y ago
Something tells me that the author [0] is probably well aware of how these work under the hood, and the math behind it - When writing scientific articles with a laymen audience in mind, you'll often have to use laymen-specific terms. But feel free to enlighten us further! [0] - https://en.wikipedia.org/wiki/Jim_Waldo https://en.wikipedia.org/wiki/Jim_Waldo
- throw310822 2y ago
  Whatever his credentials, what he says is plain wrong. GPTs don't follow "the grass is" with "green" because it's the most probable continuation- this idea is incredibly naive and breaks down with sentences longer than a few words. And GPTs don't crowdsource the answers to questions, their answers are not necessarily the most common, and neither "the consensus view is determined by the probabilities of the co-occurrence of the terms"- there is no such algorithm implemented anywhere. What LLMs crowdsource is a world model, and they need an incredible amount of language to squeeze one out from it, second hand. We do train them for the ability to predict the next word, which is a task that can only be performed satisfactorily by working at the level of concepts and their relationships, not at the level of words.
  - dontlikeyoueith 2y ago
    > We train them for the ability to predict thr next word, which is a task that can only be performed satisfactorily by working at the level of concepts and their relationships, not at the level of words. This is just obviously, trivially false.
    - throw310822 2y ago

josefritzishere 2y ago

I liked the take that LLMs are bullshitting, not hallucinating. https://www.scientificamerican.com/article/chatgpt-isnt-hallucinating-its-bullshitting/ https://www.scientificamerican.com/article/chatgpt-isnt-hall...

JohnMakin 2y ago

Besides harping on the fact that "hallucination" is unnecessarily anthropomorphizing these tools, I'll relent because clearly that argument has been lost. This is more interesting to me: > When there is general consensus on a topic, and there is a large amount of language available to train the model, LLM-based GPTs will reflect that consensus view. But in cases where there are not enough examples of language about a subject, or the subject is controversial, or there is no clear consensus on the topic, relying on these systems will lead to questionable results. This makes a lot of intuitive sense, just from trying to use these tools to accelerate Terraform module development in a production setting - Terraform, particularly HCL, should be something LLM's are extremely good at. It's very structured, the documentation is broadly available, and tons of examples and oodles of open source stuff exists out there. It is pretty good at parsing/generating HCL/terraform for most common providers. However, about 10-20% of the time, it will completely make up fields or values that don't exist or work but look plausible enough to be right - e.g., mixing up a resource ARN with an resource id, or things like "ssl_config" may become something like "ssl_configuration" and leave you puzzling for 20 minutes what's wrong with it. Another thing it will constantly do is mix up versions - terraform providers change often, deprecate things all the time, and there are a lot of differences in how to do things even between different terraform versions. So, by my observation in this specific scenario, the author's intuition rings completely correct. I'll let people better at math than me pick it apart though. final edit: Although I love the idea of this experiment, it seems like it's definitely missing a "control" response - a response that isn't supposed to change over time.

kreims 2y ago
Please keep harping. The marketing myths that gets circulated about these models are creating very serious misunderstandings and misallocation of resources. I am hopeful that more cautious and careful dialogue like this will curb the notions of sentience or human intelligence that exciting headlines seemed to have put in the public discussion of these tools.
- JohnMakin 2y ago
  It just draws too many annoying comments and downvotes, and has been discussed ad nauseam on this forum and others - but I broadly agree. There are "features" with these applications where if I'm rude, or frustrated with the responses, the model will say things like "I'm not continuing this conversation." How utterly absurd, it has no emotions, and there's no way that response was the result of a training set. It's just dumb marketing, all of it. And the real shame is (and the thing that actually pisses me off about the marketing/hype) that the useful things we actually have uncovered from ML or "AI" the last 10 years will be lost again in the inevitable AI winter we're facing following from whenever this market bubble collapses.

xkcd-sucks 2y ago

"Hallucinate" is an interesting way to position it: It could just as easily be positioned as "too ignorant to know it's wrong" or "lying maliciously". Indeed, the subjects on which it "hallucinates" are often mundane topics which in humans we would attribute to ignorance, i.e. code that doesn't work, facts that are wrong, etc. Not like "laser beams from jesus are controlling the president's thoughts" as a very contrived example of something which in humans we'd attribute to hallucination. idk, I'd rather speculatively invest in "a troubled genius" than "a stupid liar" so there's that

shombaboor 2y ago
they've got some great marketing to get away without using: bug, defect, error, malfunction
- shombaboor 2y ago
  or limitation
euroderf 2y ago
It might be more like reification ? The system finds a satisfying solution and sort of wills it into existence - by verbalizing it, makes it so ?
zh3 2y ago
>It could just as easily be positioned as "too ignorant to know it's wrong" ... GPT-5 is widely predicted to have a Dunning-Kruger level of expertise.
phainopepla2 2y ago
> lying maliciously Malice implies intent, which is even more misleading that hallucination, imo
taneq 2y ago
It's just 'prediction error' in a feedback loop, imo. I'm sure like any other biological human with mitochondria and stuff, you've occasionally said (or started to say) something and then you (ie. the actively cross-checking self-analyzing enigma that is 'you') thinks 'hang on, no that doesn't make sense' and you self-correct. LLMs are 100% feedforward, there's just one big autoregression going on. No strange loop shenanigans. Honestly I'm really interested to see where LLM-based diffusion models end up. (To be fair, probably mostly because I don't understand them yet so they could still be spooky. :D )

aaroninsf 2y ago

ITT an awful lot of smart people who still don't have a good mental model of what LLM are actually doing. The "stochastic continuation" ie parrot model is pernicious. It's doing active harm now to advancing understanding. It's pernicious, and I mean that precisely, because it is both technically accurate yet deeply unhelpful indeed actively, intentionally AFAICT, misleading. Humans could be described in the same way, just as accurately, and just as unhelpfully. What's missing? What's missing is one of the gross features of LLM: their interior layers. If you don't understand what is necessarily transpiring in those layers, you don't understand what they're doing; and treating them as black box that does something you imagine to be glorified Markov chain computation, leads you deep into the wilderness of cognitive error. You're reasoning from a misleading model. If you want a better mental model for what they are doing, you need to take seriously that the "tokens" LLM consume and emit are being converted into something else, processed, and then the output of that process, re-serialized and rendered into tokens. In lay language it's less misleadly and more helpful to put this directly: they extract semantic meaning as propositions or descriptions about a world they have an internalized world model of; compute a solution (answer) to questions or requests posed with respect to that world model; and then convert their solution into a serialized token stream. The complaint that they do not "understand" is correct, but not in the way people usually think. It's not that they do not have understanding in some real sense; it's that the world model they construct, inhabit, and reason about, is a flatland: it's static and one dimensional. My rant here leads to a very testable proposition: that deep multi-modal models, particularly those for whom time-base media are native, will necessarily have a much richer (more multidimensional) derived world-model, one that understands (my word) that a shoe is not just an opaque token, but a thing of such and such scale and composition and utility and application, representing a function as much as a design. When we teach models about space, time, the things that inhabit that, and what it means to have agency among them—well, what we will have, using technology we already have, is something which I will contentedly assert is undeniably a mind. What's more provocative yet is that systems of this complexity, which necessarily construct a world model, are only able to do what they do because they have a self-model within it. And having a self-model, within a world model, and agency? That is self-hood. That is personhood. That is the substrate as best we understand for self-awareness. Scoff if you like, bookmark if you will—this will be commonly accepted within five years.

sdwrj 2y ago

You mean the magic wizard isn't real and GPT lied to me!?!?

linsomniac 2y ago

One of the biggest places I've run into hallucination in the past has been when writing python code for APIs, and in particular the Jira API. I've just written a couple of CLI Jira tools using Zed's Claude Sonnet 3.5 integration, one from whole cloth and the other as a modification of the first, and it was nearly flawless. IIRC, the only issue I ran into was that it was trying to assign the ticket to myself by looking me up using "os.environ['USER']" rather than "jira.myself()" and it fixed it when I pointed this out to it. Not sure if this is because of better training, Claude Sonnet 3.5 being better about hallucinations (previously I've used ChatGPT 4 almost exclusively), or what.

danielbln 2y ago
Context helps so, so much. Adding terminal output, IDE diagnostics, code, remote documentation into the context really improves the output, and editors like zed make it very convenient to do.

752963e64 2y ago

[dead]

andrewla 2y ago

The author says: > Once understood in this way, the question to ask is not, "Why do GPTs hallucinate?", but rather, "Why do they get anything right at all?" This is the right question. The answers here are entirely unsatisfactory, both from this paper and from the general field of research. We have almost no idea how these things work -- we're at the stage where we learn more from the "golden-gate-bridge" crippled network than we do from understanding how they are trained and how they are architected. LLMs are clearly not conscious or sentient, but they show emergent behavior that we are not capable of explaining yet. Ten years ago the statement "what distinguishes Man from Animal is that Man has Language" would seem totally reasonable, but now we have a second example of a system that uses language, and it is dumbfounding. The hype around LLMs is just hype -- LLMs are a solution in search of a problem -- but the emergent features of these models is a tantalizing glimpse of what it means to "think" in an evolved system.

tgbugs 2y ago
I think the answer is actually quite clear and rather boring. In order to get something "right" there has to be some external standard of knowledge and correctness. That definition of correctness can only be provided by the observer (user). Alignment between the user's correctness criteria and generated text happens entirely by accident. This can be demonstrated by observing a correlation between coverage of a domain in the training data and the rate at which incorrect results are produced (as discussed in other comments). That is, they get things "right" because there was sufficient training data that contained information that matched the user's definition for correctness. In fact, exceptionally boring.
- andrewla 2y ago
  This is a very post hoc explanation. What does "coverage in the training data" mean? Take a simple task of something like "How many a's are there in the word bookkeeper" -- what is your theory for why it can answer this question correctly or even give something approaching a coherent answer? It never even sees the letters that are in the token "bookkeeper", and this is definitely not something that appears explicitly in the training data. I challenge you to give a "clear and boring" explanation for this -- this is incredibly subtle behavior that emerges from a complex architecture and complex training process, and is in its own right as fascinating and mysterious as the ability of humans to do this task and the inability of cats to do it.

deleted 2y ago

[deleted]

gengstrand 2y ago

This piece reminds me of something I did earlier this year https://www.infoq.com/articles/llm-productivity-experiment/ https://www.infoq.com/articles/llm-productivity-experiment/ where I conducted an experiment across several LLMs but it was a one-shot prompt about generating unit tests. Though there were significant differences in the results, the conclusions seem to me to be similar. When an LLM is prompted, it generates a response by predicting the most probable continuation or completion of the input. It considers the context provided by the input and generates a response that is coherent, relevant, and contextually appropriate but not necessarily correct. I like the crowdsourcing metaphor. Back when crowdsourcing was the next big think in application development, there was always a curatorial process that filters out low quality content then distills the "wisdom of the crowds" into more actionable results. For AI, that would be called supervised learning which definitely increases the costs. I think that unbiased and authentic experimentation and measurement of hallucinations in generative AI is important and hope that this effort continues. I encourage the folks here to participate in that in order to monitor the real value that LLMs provide and also as an ongoing reminder that human review and supervision will always be a necessity.

derefr 2y ago
For coding problems specifically, you could get quite far by giving the model a the tool-use of a sandboxed compiler/interpreter (perhaps even with your project files already loaded into the sandbox); and then training the model to test its own proposed solutions in the sandbox and revise them until they actually produce the expected outputs.

fsndz 2y ago

Jean Piaget said it better: "Intelligence is not what we know, but what we do when we don't know." And what do LLMs do when they don't know, they spit out bullshit. That is why LLMs won't yield to AGI (https://www.lycee.ai/blog/why-no-agi-openai https://www.lycee.ai/blog/why-no-agi-openai). For anything that is out of their training distribution, LLMs fail miserably. If you want to build a robust Q&A system and reduce hallucinations, you better do a lot of grounding, or automatic prompt optimisation with few shot examples with things like DSPy (https://medium.com/gitconnected/building-an-optimized-question-answering-system-with-mipro-and-dspy-9fe325ca33a9 https://medium.com/gitconnected/building-an-optimized-questi...)

simonw 2y ago

> For this experiment we used four models: Llama, accessed through the open-source Llama-lib; ChatGPT-3.5 and ChatGPT-4, accessed through the OpenAI subscription service; and Google Gemini, accessed through the free Google service. Papers like this really need to include the actual version numbers. GPT-4 or GPT-4o, and which dated version? Llama 2 or 3 or 3.1, quantized or not? Google Gemini 1.0 or 1.5? Also, what's Llama-lib? Do they mean llama.cpp? Even more importantly: was this the Gemini model or was it Gemini+Google Search? The "through the free Google service" part could mean either. UPDATE: They do clarify that a little bit here: > Each of these prompts was posed to each model every week from March 27, 2024, to April 29, 2024. The prompts were presented sequentially in a single chat session and were also tested in an isolated chat session to view context dependency. Llama 3 came out 18th of April, so I guess they used Llama 2? (Testing the prompts sequentially in a single chat feels like an inadvisable choice to me - they later note that things like "answer in three words" sometimes leaked through to the following prompt, which isn't surprising given how LLM chat sessions work.)

FrustratedMonky 2y ago

Is prompt engineering really 'psychology'. Convincing the AI to do what you want. Just like you might 'prompt' a human to do something. Like in the short story Lena, 2021-01-04 by qntm https://qntm.org/mmacevedo https://qntm.org/mmacevedo In short story, the weights of the LLM are a brain scan. But same situation. People could use multiple copies of the AI. But each time, they would have to 'talk it into' doing what they wanted

tim333 2y ago

It seems to me that human brains do something like LLM hallucination in the first second or two - come up with random guess, often wrong. But then something fact checks it. As in does it make sense, is there any evidence. I gather the new q* / strawberry thing does something like that. Sometimes personally in comments I think something but google it see if I made it up and sometimes I have. I think a secondary fact check phase may be necessary for all neural network type setups.