8 ms·
Are we really still having this conversation in 2024 ?! :-( Why would a language model do anything other than "hallucinate" (i.e. generate words without any ca
by HarHarVeryFunny 2y ago
Are we really still having this conversation in 2024 ?! :-(
Why would a language model do anything other than "hallucinate" (i.e. generate words without any care about truthiness) ? These aren't expert systems dealing in facts, they are statistical word generators dealing in word statistics.
The useful thing of course is that LLMs often do generate "correct" continuations/replies, specifically when that's predicted by the training data, but it's not like they have a choice of not answering or saying "I don't know" in other cases. They are just statistical word generators - sometimes that's useful, and sometimes it's not, but it's just what they are.
- FrustratedMonky 2y ago""Are we really still having this conversation in 2024 ?! :-( "" </Sarcasm> Are we in 2024 and people are still just saying "duh its statistics, nothing to see here".
- octonion137 2y agoAs opposed to what?
- FrustratedMonky 2y agoadded sarcasm note. it was referring to the parent.
- asdasdsddd 2y agoI dont get it, what's the unique insight in the article, isn't it just stats and bad data?
- FrustratedMonky 2y agoIt's just being a bit reductive to just toss AI out as un-interesting because it is just based on statistics. And, think there is question of 'bad data' so study is bad/invalid, versus studying impact of 'bad data' on responses is the result.
- asdasdsddd 2y agoI meant, the answer to the question of why does AI hallucinate is the same answer to the question "why does any statistical system ever produce an incorrect result" which I thought was well established.
- FrustratedMonky 2y agoHow can it be well established when the LLMs are still actively changing and improving. Don't think we can say, statistical systems can have incorrect results, hence we no longer need to study how statistical system produce incorrect results, because we already know they can have incorrect results.
- fnordpiglet 2y agoI am impressed that with multimodal modals in the hands of everyone people still say this. I mean it’s true on the surface but the ability to take a picture of a data sheet and wiring diagram and get it to give me step by step instructions to wire two components together and an example of the UART protocol in cpp that while not directly functional captures all the essential information I need. That’s amazing and it’s not just word prediction, the source is images of documents with diagrams! Anyone who does this and comes away jaded has lost the ability to dream.
- imiric 2y agoIt's not just true on the surface—it's true at its core. All those features you mention are based on statistical relationships. This doesn't mean that these systems can't be very useful, but it also doesn't mean that they are _intelligent_, as much as we like to call them that. They have no understanding of their input or output, beyond being able to pattern match and mechanically decide what the next token should be based on their training data.
- bongodongobob 2y agoThen you need to explain why statistics isn't enough to be intelligent. We are at the point where that isn't an obvious argument anymore and I'll point to the best models as to why you might be wrong.
- imiric 2y agoBecause being able to find patterns in large amounts of data will never make the system intuitively understand math[1], or history[2], or any other field. It will always depend on the data and biases we feed it, at least with the current approaches. You might say that this is how humans learn and demonstrate intelligence, but it's not the same thing. We have the ability to advance our understanding of the universe without being explicitly trained on every topic. Until we can build systems that can do this, I wouldn't label them as intelligent. But, again, this doesn't mean that they can't be useful. [1]: https://towardsdatascience.com/9-11-or-9-9-which-one-is-higher-6efbdbd6a025 https://towardsdatascience.com/9-11-or-9-9-which-one-is-high... [2]: https://www.theguardian.com/technology/2024/mar/08/we-definitely-messed-up-why-did-google-ai-tool-make-offensive-historical-images https://www.theguardian.com/technology/2024/mar/08/we-defini...
- DougWebb 2y agoI've never been worried about LLMs. I've always been worried about how people will use LLMs and how they will interpret the output of LLMs. Especially people who don't understand what LLMs are doing.
- reaperducer 2y agoI've always been worried about how people will use LLMs and how they will interpret the output of LLMs. Especially people who don't understand what LLMs are doing. The problem isn't the people. It's the tech companies. The tech companies are telling people that it's intelligent, and the tech companies are using it to answer people's questions as if they're presenting facts. People are using it the way they're told. If you advertise something as a solution, don't be surprised when people use it to solve things.
- outofpaper 2y agoAlways remember companies are people too... until we make companies powered by AI. Till that day the underlying problem is always with the people.
- s_dev 2y agoWhy is this concern more important the what people interpret from the media, social media and the dissemination of information in general where lies and fabrications are also commonplace? Like surely people will always fall for nonsense, lies or fabrications and there is nothing that can be done about that.
- Spivak 2y agoBecause LLMs strip away all the context surrounding the information it spits out that let you evaluate its trustworthiness. They're incredibly useful tools, I use them constantly when coding but I can do that because I know enough to validate the information and it happens that the cost of validating the output with the docs is shorter than reading them to find the relevant functions. I wouldn't dare try to use an LLM for a chemistry question because I wouldn't be able to tell if it makes any sense or not. But if you're not a "tech person" and all you see is some company advertising their AIs as magical knowledge engines with disclaimer text that wouldn't pass accessibility tests, why wouldn't you assume they know their stuff? The Perplexity ads are bordering on negligent.
- seba_dos1 2y agoAsking "why do large language models hallucinate?" is a good question to ask and answer, just like "why do birds sing?" or "why is the sky blue?" are. The problematic part is when you're somehow surprised that the sky is blue, birds sing and LLMs do nothing but hallucinate.
- lainga 2y agoTiger gotta hunt, bird gotta fly, LLM gotta answer "why, why, why?" Tiger gotta sleep, bird gotta land, LLM gotta produce a token from the context at hand
- rollcat 2y agoI don't even like the word "hallucination", because it's an anthropomorphism, further suggesting/misleading into an impression that LLMs have anything to do with how a human would reason. Fancy autocomplete. Useful, yes. Powerful, definitely. But it's just another tool, limited in applications the same way a hammer is - this is the part that many humans seem to struggle accepting.
- beezlebroxxxxxx 2y agoThe use of "hallucinate" is also my gripe. Philosophy of language, and Wittgenstein in particular, has more specific words for the output insofar as they mean anything to humans: "senseless" or "nonsense".
- SpicyLemonZest 2y agoBut I don't think those terms are accurate. If I say "John Doe was born in 1987", that's a perfectly sensible sentence, even if I don't actually know when he was born and even if it turns out he was actually born in 1971. A nonsense sentence would be something like "John Doe cat five green explicate" or "h4stga3jkui7rutjdyrst".
- beezlebroxxxxxx 2y agoI don't understand your point. You've just re-presented one difference between sense and nonsense in philosophy of language. We might call that first phrase wrong or an error or inaccurate if that was the case. We don't need to use a word like "hallucinate" for your second phrase --- we can just call it nonsense. "Hallucinate" is far more inaccurate and confusing. It paints a false picture that melds 2 distinct things, as you've described, which need not happen at the same time: errors and nonsense.
- Iulioh 2y agoI think the term forst originated with the dream-like images of google deepmind, these were really similar to human visual hallucinations, i guess the term stuck
- golergka 2y ago> Why would a language model do anything other than "hallucinate" (i.e. generate words without any care about truthiness) ? These aren't expert systems dealing in facts, they are statistical word generators dealing in word statistics. That's a false dichotomy.
- Feathercrown 2y agoTheir ability to generate facts is a consequence of the word statistics they are truly using. I think it's fair to say the statistics explanation is a more accurate interpretation.
- golergka 2y agoMore accurate than what? Statistics "explanation" is something which is technically correct. But it also doesn't present the full picture — for example, the fact that LLMs clearly build an internal mental model of the question they're talking about.
- nyrikki 2y agoI can offer cite that hallucinations are an innate property of LLMs, can you provide one that shows they have 'an internal mental model'? https://arxiv.org/abs/2401.11817 https://arxiv.org/abs/2401.11817
- ctoth 2y agoSure! https://thegradient.pub/othello/ https://thegradient.pub/othello/
- Iulioh 2y agoUhm. I don't really know if using an "experiment" is the right way to test this? Like, if you want to test such hypothesis you should first prove that these LLMs were created with that specific function in mind, we are not observing a natural phenomena, it is a manmade(manmade...loosly) algorithm . You don't know the internal architecture of the system and you are testing in a reality narrow situation, this is basically the scientific method applied incorrectly.
- sega_sai 2y agoI do not find this view to be useful. The are cases from my personal experience, where I asked somewhat esoteric practical questions, that likely do not (seem) to have a clear answer in the web and I have got considerable help from ChatGPT. At some point this dichotomy of 'statistical word generator' vs 'true intelligence' should go away as it's just not useful. (I think these discussions always lead to Chinese Room problem; and IMO at some point it does not matter what 'dumb' process is behind, provided it solves a problem or the system behaves like an intelligent agent)
- lanternfish 2y agoExcept there is a qualitative difference in the class of knowledge that a statistical word generator and an expert system would generate. Just because a LLM _can_ offer valuable and insightful information, doesn't mean that it doesn't also hallucinate. The most troubling factor here is that often the hallucinated content also looks like valuable and insightful information, but is just incorrect. This is the use. You have to hold that awareness whenever interacting with these systems.
- outofpaper 2y agoYup exatly. They are dream machines. LLMs without other systems can only work in the flow. The fact that in this word flow the larger LLMs can generate navigation instructions for actual mazes and solve random algorithmicly generated problems doesn't mean they are not hallucinating it just means we're getting wonderfully useful hallucinations.
- refulgentis 2y agoHallucination is a term that means "imagined facts", so it's very hard for me to parse this comment into something meaningful beyond "if we say it always generates hallucination, we can say it always generates hallucinations"
- jrm4 2y ago
- not2b 2y ago"Why would a language model do anything other than "hallucinate" (i.e. generate words without any care about truthiness) ?" That's exactly the question the paper attempts to answer: why do LLMs ever get it right? The answer is that on topics where there's a lot of data and a general consensus on what the right answer is, the statistical model will find that answer, and otherwise you get junk. That's why they work so well for people trying to write Python or Javascript, for example. But I already knew this, you might say. Sure, but the authors produced evidence to back it up.
- jrm4 2y agoThere's really no "right or wrong" per se -- the question that's really being asked is "to what extent does it resonate with a person?"
- danielbln 2y agoAs context sizes grow, it's easier to add lots of information outside of training data via in-context learning, which should offset that issue quite a bit.
- deleted 2y ago[deleted]
- wslh 2y agoI think we're all having different experiences and conversations with LLMs, and sometimes framing them purely as "statistical word generators" or expecting them to function like AI in the traditional sense of automated problem-solving might not capture the whole picture. Like many in this community, I use LLMs daily. My main use case now isn’t software development but rather getting assistance in connecting concepts that I don’t know precisely, guided by my intent. In some ways, it's more akin to "out-of-the-box thinking" but with a tool that helps me explore ideas I might not reach on my own or suffer within the economy of search [1][2]. I might know something about X, Y, and Z, but there's a concept W that ties them all together. Without using W, people in that field might not grasp the connections between X, Y, and Z. Apologies for the abstraction here. I suppose it's my epistemological bias showing! Even if LLMs don't precisely connect the dots, they help my brain connect them faster. [1] https://en.wikipedia.org/wiki/Search_theory https://en.wikipedia.org/wiki/Search_theory [2] https://www.di.ens.fr/~lelarge/soc/varian2.pdf https://www.di.ens.fr/~lelarge/soc/varian2.pdf
- snickell 2y agoYes, they are auto-completers, but they are auto-completers that are layered AND operate in higher dimensional spaces. This throws all intuitions off, and I think makes it misleading to think of them as "just" auto-completers. That's part of the story, but not the whole of it. I suspect we are much closer to auto-completers than most of us like to think, but we're also trained+incentivized by culture, education, parenting, socializing, to produce "useful" results. Maybe part of the problem is in the data set: how much modeling of "how to admit ignorance or uncertainty" are in LLMs training data sets? If you read the internet, all you see is confident replies to other confident replies. Ignorance or non-confidence tends to elicit either a bluff or non-response. If you read technical literature, you see much of the same. Maybe LLMs are trained on a dataset, and thereby inherit a culture that's accidentally biased toward ignorant confidence. In human conversation, if somebody asks a question and I don't know the answer, I say I don't know. On the internet, I just skip it and leave it for somebody else who thinks they know. All this is to say: maybe a statistical autocompleter can admit ignorance instead of firing "neural noise" based on barely-there loose associations. Maybe it just needs a stronger pathway toward talking about not knowing when there's not a strong association.
- 6510 2y agoWasn't it that it by design prefers things expressed with certainty?
- HarHarVeryFunny 2y agoOne problem is that the LLM's own knowledge, or lack of it, doesn't follow from any individual training sample(s). Even if there were a bunch of "I don't know X" samples in the training set, that ought to be trumped by one authoritative "X is ..." one. The next problem is that the LLM doesn't know how reliable it's various training sources are (unlike a human who might trust personal experience > textbook > twitter comment), or even which samples come from which source so that it could learn that.
- chaosist 2y agoI have thought a lot about this and I suspect "I don't know" would be devastating to the model. The magic is in the fact that the model can't say "I don't know". The model would have to have arbitrary thresholds and a type of domain/context classification in order to set this arbitrary threshold for the conversation in order to say "I don't know". It would create all these unsolvable boundary conditions. AGI in this context then would be minimizing the need for "I don't know" until it no longer applies. Is that possible? I don't know :) I would defer to Chomsky also on the subject that we are basically nothing like these models when it comes to language and we are not auto-completers.
- accountnum 2y ago[flagged]
- HarHarVeryFunny 2y ago> Please, explain to me which exact algorithm you use to unify the "facts" you have such magic access to? The "magic access" to "facts" (reality) is called embodiment and personal experience with the world in which we live. An LLM has no embodiment, and therefore no access to reality/facts. All it has is a training set of mixed sources, and a training objective of impersonating those sources (whether an encyclopedia or 4chan comment).
- accountnum 2y ago[flagged]
- famouswaffles 2y agoPretty much all research (and there's a fair few with different methodologies) on this converge on the same conclusion: LLMs internally know a lot more about the uncertainty and factualness of their predictions than they say. "LLMs are always hallucinating" is a popular stance but wrong all the same. Maybe rather than asking Why models hallucinate, the better question is to ask "Why not?". During pre-training, there's close to zero incentive to push any uncertainty to the forefront (words). GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r https://imgur.com/a/3gYel9r Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824 https://arxiv.org/abs/2310.06824 The Internal State of an LLM Knows When It's Lying - https://arxiv.org/abs/2304.13734 https://arxiv.org/abs/2304.13734 LLMs Know More Than What They Say - https://arjunbansal.substack.com/p/llms-know-more-than-what- https://arjunbansal.substack.com/p/llms-know-more-than-what-... Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 https://arxiv.org/abs/2205.14334
- HarHarVeryFunny 2y agoYes, because just like chess ELO that we discussed the other day, they need to learn this in order to do well on their training objective - impersonating (continuing) their training sources. If they are continuing a lie then they need to have recognized the input as having this "context", and take that into account during prediction.
- famouswaffles 2y agoRight but then the problem of hallucination has little to do with statistical generation and much more the utter lack of any incentive in pre-training or otherwise to push features the model has already learnt into the words it generates.
- 2y ago
- andrewla 2y ago> Why would a language model do anything other than "hallucinate" Indeed, why?!?!?! Why do they so often get the correct answer to very direct questions? Saying "it's in the training data" -- I dare you to find anything in the training data that talks about how many "q's" there are in the word "mortuary", and yet it "hallucinates" up answers to this. > without any care What does it mean to care? What question could you ask of an LLM that would allow you to assess how much it "cares" about something? > but it's not like they have a choice of not answering or saying "I don't know" in other cases Is it your contention that the phrase "I don't know" has never occurred in the training data for an LLM? There seems to be a dichotomy of reactions to LLMs. There are technical people saying "it's just an autocomplete engine reciting things from its training set" and there are non-technical people saying "it does more than just next token completion, it is trained to use language". The first group is technically correct but ignores the fact that it can, emergently, do things far outside of an explainable capability, the second group is technically incorrect but correctly perceives that it can use language in novel ways. Neither group captures the fact that we built extremely complex linear algebra machines that for reasons we do not understand, despite being trained on an incredibly simple task (next-token-prediction) are capable of actually using language in a way that ten years ago we assumed only humans could do.
- HarHarVeryFunny 2y ago> Is it your contention that the phrase "I don't know" has never occurred in the training data for an LLM? No, but when it does occur in the training data it's a reflection of that particular source/speaker not knowing, which isn't the same as the LLM not knowing because it was also trained on millions/billions of additional sources. For an LLM to learn to say "I don't know" appropriately, it would need to know when it itself doesn't know (and have that change if you told it), and of course it doesn't have that capability.
- andrewla 2y ago> of course it doesn't have that capability Yes, of course not. Of course. There's no way, for example, that I could ask it if it knew what number I was thinking of, and, after I tell it the number and ask the same question, that it could express that it didn't know before and did know after. Absolutely impossible. Out of the question that it would have this capability. Clearly impossible given its architecture. No way that it could possibly do this task. Why, it would require advances in machine learning and quantum computers and understanding a theory of consciousness and perception at a level that we won't have for centuries. Maybe even a completely new design and training procedure to even begin to approach this insurmountable task. And if it did demonstrate this ability, then clearly your assessment of its capabilities would be completely wrong and you would have to step back and reconsider how much its training data reflects its abilities.
- Nition 2y agoJust a note: Truthiness means a feeling of truthfulness, even if not actually true. I'd argue that LLMs do care about that, but I suspect you meant 'truthfulness'.