39 ms·

Hallucination is inevitable: An innate limitation of large language models

308 points by louthy 3y ago

chrisjj 3y ago
> hallucination is defined as inconsistencies between a computable LLM and a computable ground truth function. That's simply inaccuracy or fabrication. Labelling it hallucination simply panders to the idea these programs are intelligent.
- somewhereoutth 3y ago
  Yes, imagine a pocket calculator that gave a completely wrong answer 10%? of the time, and calling it 'capricious' instead of simply broken.
  - BlueTemplar 3y ago
    A real shame that Douglas Adams didn't live to see all of this...
  - chrisjj 3y ago
    Indeed. The best solution would be to market it as intelligent ;)
- tgv 3y ago
  That statement is also a bit easy on the "computable ground truth." There is no such thing for the tasks we use an LLM for (unless you make up some definition that mangles the definition of each word).
paulsutter 3y ago
The models are just generating probable text. What’s amazing of how often the text is correct. It’s no surprise at all when it’s wrong Their bold confidence to be flat out wrong may be their most human trait
- baq 3y ago
  This again. They’re trained to generate probable text. The mechanisms created in the parameter blob during training to do that are basically a mystery and have to be pulled out of the model with digital brain surgery. E.g. LLMs are reasonable at chess and turns out somewhere in the blob there’s a chessboard representation, and you can make the model believe the board is in a different state by tweaking those parameters. So yeah they generate probable text, sure. Where they get the probabilities is a very good research problem.
  - timr 3y ago
    > E.g. LLMs are reasonable at chess and turns out somewhere in the blob there’s a chessboard representation, and you can make the model believe the board is in a different state by tweaking those parameters. Broadly agreed, but there's no "representation"...the model has no memory, let alone a "concept" of a chessboard. It's just trained on a bunch of textual replays of chess games, and this works well enough for a game with finite probability space. Likewise, I have asked generic LLMs to generate "novel" drugs that solve particular problems, because their training sets have included lots of examples of drug molecules in string form (aka SMILES strings). This works far less well (because chemical space is much larger than a chess game), and the "novel" drugs usually end up looking like some mishmash of existing chemicals for the same problem. This can be a useful tool, but nobody is confusing it for a human brain doing problem solving. People are easily misled by the purported intelligence of these things because they assume that common examples of human intelligence are less probable than they really are. Our languages and games and stories are pretty regular, all things considered. Expand the probability space to something truly vast (even images work for this), and you can easily see the stochastic parrot emerge.
    - baq 3y ago
      The fact that tweaking parameters which appear to store the board makes it play according to the tweaked numbers instead of what was passed to it the context (i.e. working memory) directly contradicts your assertion that LLMs have no memory. The context is their memory. I can’t comment on your drug generation task - they aren’t magic, if the training didn’t result in a working drug model in the billions of params you’ll get garbage output, not very surprising. My point boils down to the ‘what’s more likely’ question: magical stochastic parrots which just by accident manage to create understandable and coherent responses to unreasonably large set of questions or the magic is actually some kind of a world model, or multiple, inside the blob of numbers, outputs of which steer the probabilities, just as this confirmed chess example. My bet is on learned world models because I’m not convinced there’s magic in our physical world.
sschueller 3y ago
You have to very carefully ask your question for it to not make things up. For example don't ask "how do I do this in in x?". Ask "can I do this with x?" These "AI" s are like "yes men". They will say anything to please you even if it's untrue or impossible. I have met people like that and they are very difficult to work with. You can't trust that they will deliver the project they promised and you always have to double check everything. You also can't trust them that what they promised is even possible.
- deleted 3y ago
  [deleted]
- ithkuil 3y ago
  The analogy is apt indeed. It's all about training and selection. As long as the incentives are that you either behave in that way "or else" it's unsurprising that we end up with a system that uses its intelligence to meet the goals you've set up. Fortunately that doesn't tell much about the limitations of the underlying intelligence but mostly about the limitations of the incentive structure you put around it
- impulsivepuppet 3y ago
  Before ChatGPT, human language translation had a similar problem but people weren't as vocal about it. What I find frustrating that it's increasingly challenging to have DeepL translate thou -> du, as this was my go-to "hack" to overcome the incompatibility of the English language due to its missing features. To somewhat remedy the "yes man" problem, one needs to become a pedantic mathematician about posing your questions and I don't believe that LLM technology alone is capable of overcoming it entirely. As silly as it sounds, I must concede to the existence of "prompt engineering" as I can forsee the development of abstractions aimed to decompose questions for you.
- tudorw 3y ago
  I've had some success with 'Ask follow up questions where additional clarity is required'. My best experiences start with a much more freeform conversation about what we are going to attempt to do, doing a Q&A first to make sure that both I and the AI are thinking about the same domain and sharing associated nomenclature seems to help.
golol 3y ago
This is just another diagonalization against some finite thing, like the no free lunch theorem. An "LLM" in their definition is essentially any finite thing which predicts the next token. The same result applies to your brain too, for example. Like all such diagonalization results, it is not really relevant for real world considerations. The reason is that it does not matter if your model fails on none, finitely many or infinitely many inputs. In reality the space of possible inputs is equipped with a probability measure, and the size of the hallucinating inputs set w.r.t. that measure is relevant. Diagonalization arguments usually, make no claim to the size of that set, and it is most likely negligible in the real world.
FeepingCreature 3y ago
It seems a stretch to call it "inevitable". "Inevitable given the current architecture without modifications" at most. Also, I'm missing a section on how (if) human brains manage to avoid hallucinations in this. Also, it doesn't have to never hallucinate, it just has to hallucinate less than we do.
- bamboozled 3y ago
  According to Buddhist philosophy, our whole identity is a hallucination :) I kind of concur.
  - jetanoia 3y ago
    Username checks out :)
    - bamboozled 3y ago
      I'll honestly take this is a compliment.
- SubNoize 3y ago
  Or catch itself that it's hallucinating? I feel like humans would do that a fair bit. How often do we sit somewhere thinking about random scenarios that won't ever happen and are filled with wild thoughts and sometimes completely out of the world situations.. then we shake our heads and throw away the impossible from that thought train and only use what was based in reality
- dkjaudyeqooe 3y ago
  Because we have parts of our brain that supervise other parts of our brain and evaluate its output. For example: if you smoke pot and get paranoid, it's because pot dials back the work of the part of your brain that prunes thought paths that are not applicable. Normally, paranoid thoughts do not make sense, so they are discarded. That's also why you're more 'creative' when you smoke pot, less thought paths are pruned and more stuff that doesn't quite make sense gets through. Or thoughts that overly focus on some details get through, which are normally not required. Our brains are inherently "higher level", current AI is hopelessly simplistic by comparison.
t_mann 3y ago
I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute some functions, but the hallucination problem isn't about LLMs needing to know everything. The problem that we care about is the 'I don't know'-answering problem, which may still be computable.
- az09mugen 3y ago
  I think there is no easy way to make an LLM answer "I don't know". For that, it should learn among all the stuff ingested when people effectively don't know. But most people on internet write down irrelevant stuff even when they don't know instead of simply writing "I don't know". That's a very good point.
  - timini 3y ago
    I think its fairly simple, it needs a certain level of proof e.g references to authoritative sources, if not say "i don't know".
    - cubefox 3y ago
      Then it is nothing more than a summarizer for search engine results.
      - amarant 3y ago
        A lot of people have said chat-gpt/copilot is a lot like having a robotic junior dev around. I think perhaps your description is more succinct
    - az09mugen 3y ago
      I'm really curious about one would implement that. By pondering weigths from certain sources ?
karol 3y ago
Organisms that evolved to perceive true reality instead of the "user interface" have smaller chances of survival. Donald Hoffman
- tibbydudeza 3y ago
  The brain fakes it to approximate reality - more so for reasons of the limitations of the wetware it runs on than anything else. For others here is a TED talk. https://www.youtube.com/watch?v=oYp5XuGYqqY https://www.youtube.com/watch?v=oYp5XuGYqqY
Borealid 3y ago
> hallucination is defined as inconsistencies between a computable LLM and a computable ground truth function. With this definition, you can trivially prove the titular sentence - "hallucination is inevitable" - is untrue. Let your LLM have a fixed input context length of one byte. Continue training the LLM until such a time as it replies to the input "A" with "yes" and all other inputs with "no". Define your computable ground truth function such that the correct output for the input "A" is "yes" and the correct output for all other inputs is "no". This LLM provable never hallucinates - we have exhaustively verified that its output matches the ground truth function for all possible inputs. There is nothing stopping inductively increasing the size of the input context and the number of entries in the ground truth table arbitrarily, and at no step do hallucinations become "inevitable".
- resolutebat 3y ago
  > Continue training the LLM until such a time as it replies to the input "A" with "yes" and all other inputs with "no". This is basically the same as saying "train your LLM until they never hallucinate", which reduces your claim to a tautology: an LLM trained not to hallucinate does not hallucinate. The trick is making that happen.
  - ProxCoques 3y ago
    As I always tell my students: the solution to unreliable code is not to put the bugs in there in the first place.
    - paxcoder 3y ago
      [dead]
  - Borealid 3y ago
    It's a tautology that for a given truth table mapping inputs to "correct" outputs there exists a function that produces that mapping. Saying that you can't train an LLM to NOT hallucinate is saying that it's impossible for any LLM to always produce output matching any particular truth table. There may exist truth tables where it's not possible to produce an LLM to match them (for some finite size of LLM perhaps), but my claim isn't a tautology - it's just an assertion that there exist some truth tables which an LLM can be trained to match. It may be tricky to make the LLM in the first place, but it's certainly not as tricky to verify it. You can test it with every input you consider to be present in the truth table, record its results, and throw it away if it doesn't match. The only possible results here are either non-convergeance or a "perfect" LLM. You can get rid of the non-convergeance if you bound the size of the LLM and literally iteratively verify every single possible model. The authors of the paper didn't do that. For trivially sized models and inputs, that's completely possible. For a 7B parameter model, nobody is doing that ever. But you can prove the title statement wrong with a ten-parameter model and a one-byte input.
Sparkyte 3y ago
Definitely a given, it isn't like AI has an actual brain capable of resolving and forming new connections. The LLM and human brains is that LLMs are interactive compendiums and our brains organize and sort information that ensures survival as an organism. There is no survival of whether or not LLMs are accurate and a machine wouldn't understand what is good or bad without weighted context. Its good for analyze, process, storing, retrieving and decomposing information. It isn't good at understanding, validating and forming connections between the things it says and what you want of it. It lacks comprehension, it doesn't lack composure.
- golol 3y ago
  The result in the theorem applies to your brain. Your brain can be modelled as an LLM in the sense of the paper up to arbitrarily small error. The result is a diagonalization argument that is not very relevant for the real world.
  - wredue 3y ago
    >your brain can be modelled as an LLM in the sense of the paper The vast majority of people actually writing LLMs don’t claim this, and in fact, actually claim the very opposite: that LLMs do not accurately model a human brain in any capacity. The fact is that science has no clue what happens in the nucleus of a neuron, so claiming that computer scientists must is… well. You fill in the word.
    - golol 3y ago
      Definition 2 (Large Language Model). Let S be a computable seta of all the finite-length strings of alphabet A and (s0, s1, . . .) be an one-to-one enumeration of all the elements in S. A large language model, denoted h, is a function that completes the input string s ∈ S using the function’s predicted tokens h(s), in a finite time. Function h is attained procedurally using a set of training samples of input-completion pairs. For an arbtrarily large duration and an arbitrarily small error with respect to any definition of error you choose, there exists an LLM in the above sense which models the dynamics of your brain with that small error.
carlossouza 3y ago
Someone smart once said: If it is good, we call it "creativity." If it is bad, we call it "hallucination." This isn't a bug (or limitation, as the authors say). It's a feature.
- tgv 3y ago
  Asking it to write code for you is basically asking it to hallucinate.
  - gardenhedge 3y ago
    I don't think so. I think it's asking it to repeat code it has been trained on
    - tgv 3y ago
      Search for a piece of code you wrote. If it's more than 15 lines and not boilerplate, chances are you won't find it anywhere on the net.
      - gardenhedge 3y ago
        To be honest, I imagine I would. Variables named could be different but it would largely be the same as code others have written. For example, I am creating an app in React Native at the moment. My app will not be unique in terms of code but instead in business domain.
      - Jensson 3y ago
        But there are many pieces of code that I've written that you can find in many places on the net, having a tool that can adapt that to your codebase in seconds is useful. It doesn't have to be smart, just pasting in an function and fitting that to your code is useful.
        tgv 3y ago
        Sure, but the point is it will have to adapt it to your code, if only in naming. So it has to make up things, i.e. hallucinate. It can't just reproduce the best match in memory.
precompute 3y ago
Ah, the AI hype is now entering the "let's be real" phase. Haven't seen a frenzied post on alignment in a while now.
- Culonavirus 3y ago
  The hype is insane. Listen, I think LLMs still have a lot of room to grow and they're already very useful, but like some excellent researchers say, they're not the holy grail. If we want AGI, LLMs are not it. A lot of people seem to think this is an engineering issue and that LLMs can get us there, but they can't, because it is not an engineering issue.
  - hnfong 3y ago
    Do you have evidence to back your claims up besides "the hype is overblown"? Because hype only indicates that the precise hyped up claims are wrong, it doesn't imply the opposite extreme (i.e. LLMs can never achieve AGI) must be true.
    - deleted 3y ago
      [deleted]
  - mistermann 3y ago
    What kind(s) of an issue do you think it is fundamentally?
  - PopePompus 3y ago
    I don't think you can say with confidence that the LLM approach will not lead to AGI, unless you understand in detail how human intelligence operates, and can show that no modification to current LLM architectures can achieve the same or superior results. I think the fact that adding "attention" to LLMs made a huge difference means that we are probably still in the low hanging fruit stage of LLM architecture development, and a few more design improvements on a par with "attention" might lead to something that could legitimately be called AGI. Many people, myself included, believe that LLMs are now exhibiting emergent behavior properties. If that's true, then saying that LLMs are not intelligent because they just predict the next token of output is like saying collections of neurons cannot be intelligent because they just stimulate other neurons chemically.
bbor 3y ago
I’m sorry… does this paper just point out that LLMs by definition are not as good at holding data as a direct database? Cause A) duh and b) who cares, they’re intuitive language transformers, not knowledge models. Maybe I’m missing something obvious? This seems like someone torturing math to imply outlandish conclusions that fit their (in this case anti-“AI”) agenda.
- anonylizard 3y ago
  It at least disproves LLMs from being 'god models'. They will never be able to solve every problem perfectly.
  - ninetyninenine 3y ago
    Humans aren't God models either. The goal is to get this thing to the level of a human. God like levels are not possible imo.
TylerE 3y ago
I miss the days when HN posts about hallucinating were about microdosing.
gokulkrishh09 3y ago
[dead]
hyperpape 3y ago
The result seems to rely on stipulating the LLM must answer true or false to all its questions, and can't say "I don't know." So it's an interesting result, but it's not obvious that it tells us much about our actual problem, which is 100% about how to get a system that accurately understand the level of confidence it should have.
cjdell 3y ago
I thought the industry was already experimenting with the idea that you have another LLM observing the output of the primary LLM which is trained more towards safety than creativity. On top of that it would be good if the safety LLM could give a confidence score in the answer given by the main LLM. Then you can try multiple attempts with different parameters and only show the highest confidence answer to the user.
franze 3y ago
Hallucinations and Ideas are the same thing.
DebtDeflation 3y ago
There used to be an entire sub-field of NLP called Open Domain Question Answering (ODQA). It extensively studied the problem of selecting the best answer from the set of plausible answers and devised a number of potential strategies. Like everything else in AI/ML it fell victim to the "bitter lesson", in this case that scaling up "predict the next token" beats an ensemble of specialized linguistic-based methods.
- falling_myshkin 3y ago
  For those who don't know: http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html I agree with you for the NLP domain, but I wonder if there will also be a bitter lesson learned about the perceived generality of language for universal applications.
sgammon 3y ago
Splinters inevitable: An innate limitation of working with wood /s
- HKH2 3y ago
  People expect splinter-free epistemology instead of just putting on some gloves.
Scene_Cast2 3y ago
I don't think anyone has mentioned Bayesian Neural Nets (I forget the exact term). Sure, the paradigm adds an order of magnitude overhead (at least - and that's why I've never seen it used in the industry), but you can bolt it on to existing architectures. The basic idea is that besides the probabilities, the network also spits out confidence (IIRC based on how out-of-distribution the input is). There's been a ton of work on getting confidence values out of existing neural nets without as much overhead, but I've never seen those approaches replicate in the industry.
- wnkrshm 3y ago
  I would imagine that to propagate any confidence value through the system you'd need to have priors for the confidence of correctness for all data in your training set. (and those priors change over time)
somewhereoutth 3y ago
For production systems the considerations are: - how often does it error? - how bad are the errors? - how tolerable are the errors? - how detectable are the errors? - how recoverable are the errors? For example, a pocket calculator that occasionally was out at the 3rd decimal place might do much more damage than one that quite regularly returned NaN. LLMs error both regularly and badly, so detectability and recoverability are going to be crucial for useful deployment in systems outside of those that have a high tolerance for errors (e.g. algorithmic feeds).
- intended 3y ago
  Yes. If you want to make something that works, your life is figuring out evaluation and LLM ops. At that point, you know its not thinking, its doing token prediction.
macrolime 3y ago
They define an LLM as "a probabilistic model of a string that conditions the output at time t based on all the tokens that come before it in the string". I think that definition is wide enough to include human intelligence, so their finding should be equally valid for humans.
- jddj 3y ago
  Silk silk silk silk silk silk silk. What do cows drink?
- moooo99 3y ago
  > I think that definition is wide enough to include human intelligence, so their finding should be equally valid for humans. Which is definitely true. Human memory and the ability to correctly recall things we though we remembered is affected by a whole bunch of things and at times very unreliable. However, human intelligence, unlike LLMs, is not limited to recalling information we once learned. We are also able to do logical reasoning, which seems to improve in LLMs, but is far from being perfect. Another problem is how different we treat the reliability of information depending on the source, especially based on personal bias. I think that is a huge factor, because in my experience, LLMs tend to quickly fall over and change their opinion based on user input.
  - ben_w 3y ago
    We can do logical reasoning, but we're very bad at it and often take shortcuts either via pattern matching, memory, or "common sense". Baseball and bat together cost $1.10, the bat is $1 more than the ball, how much does the ball cost? A French plane filled with Spanish passengers crashes over Italy, where are the survivors buried? An armed man enters a store, tells the cashier to hand over the money, and when he departs the cashier calls the police. Was this a robbery?
    - mistermann 3y ago
      Humans also have various culturally flavored, implicit "you know what I mean" algorithms on each end to smooth out "irrelevant" misunderstandings and ensure a cordial interaction, a cultural prime directive.
Almondsetat 3y ago
Of course it's inevitable. Things can be facts or deductions of facts (or both). If I ask an LLM the date of birth of Napoleon and it doesn't have it in its dataset there are only 2 options: either it has other facts from which Napoleon's birthday can be deduced or it doesn't. If it does then by improving the LLM we will be able to make more and more deductions, it if doesn't then it can only hallucinate. Since there will always be a set of facts that the LLM is not aware of and which cannot be deduced we will always have hallucinations
- jtc331 3y ago
  Is this a way of saying that large language models don’t have the concept of “I don’t know”?
- andybak 3y ago
  Your "Of course" is a red flag. Unless you have a very precise definition of "LLM" then there's no "of course". It's possible to imagine a hypothetical software system capable of returning "I don't know" or "I'm not sure". I haven't RTFA but I presume they are arguing within specific constraints. The important point is - could an augmented LLM or an "LLM plus something else" solve this. I don't have an answer to that but I don't think it's an "of course" type answer.
- samatman 3y ago
  I don't see why that's inevitable at all. I immediately recognize that I don't happen to know Napoleon's birthday, and act on that basis.
keepamovin 3y ago
This is why you need to pair language learning with real world experience. These robots need to be given a world to explore -- even a virtual one -- and have consequences within, and to survive it. Otherwise it's all unrooted sign and symbol systems untethered to experience.
- phh 3y ago
  I think I agree with you (I even upvoted), but this might be an anthropomorphism. Back like 3-5 years ago, we already thought that about LLMs: They couldn't answer questions about what would fall when stuff are attached together in some non-obvious way, and the argument back then was that you had to /experience/ it to realize it. But LLMs have long fixed those kind of issues. The way LLMs "resolve" questions is very different from us. At this point, I think that if we want to prove that LLMs need to be rooted in the real world to achieve intelligence, we need to find some real-world phenomenon that is so obvious that noone ever wrote about it... but then we'd have written about it?
  - barnacs 3y ago
    Think of it this way: Intelligent beings in the real world have a very complex built-in biological error function rooted in real world experiences: sensory inputs, feelings, physical and temporal limitations and so on. You feel pain, joy, fear, have a limited lifetime, etc. "AI" on the other hand only have an external error function, usually roughly designed to minimize the difference of the output from that of an actually intelligent real world being.
    - keepamovin 3y ago
      Yeah, that's exactly my thinking man. We have to root intelligence in the real world, otherwise it will endlessly spin in these abstract loops. Akin to how logic -- untethered by emotion, intuition and experience (wisdom, maybe if you want? Understanding? Sure) -- can justify any obscene conclusion, and can not discriminate between morality. Reward functions or a system of values -- these things are rooted in real world experience. Logic is required, sure, but insufficient. At least alone! Haha :)
pfdietz 3y ago
A hallucinator seems like an excellent thing to have as a component to an intelligent system, but it must be coupled with evaluators. Such an architecture seems plausible for the human brain as well.
viktour19 3y ago
If hallucination is inevitable? What should developers do? Design user experiences that align users with this behaviour! Relatedly, I built a game to demonstrate how one might calibrate users to the responses of LLMs: https://news.ycombinator.com/item?id=39255583 https://news.ycombinator.com/item?id=39255583
demondemidi 3y ago
I bet that, basically, an LLM is just a part of a future AI. The same way that a distributor is part of an internal combustion engine. Or the way Minsk’s described The Society of Mind. Eager to see if an LLM can be bolted on to some new kind of giant model that does something like mitigate the LLMs weaknesses. Maybe it’ll be a collection of radically different models working together in 20 years, and not just a single model. Like, uh, our own brains. It reminds me of how lex and yacc are super important as grammar and tokenizer, but are only just the front end to much larger projects.
WirelessGigabit 3y ago
I feel the same way about information provided by LLMs as with lots of pre-LLM articles and overall YouTube 'information' videos. Sources matter. You're either rehashing information from a collection of sources or you have your own data to prove your statements.
- coffeefirst 3y ago
  Yeah, this is why I really like Kagi's approach: it's clearly marked for what is, and cites its sources so you can verify the quality of the answer (or at least get a ballpark idea of where it's coming from) yourself.
lebuffon 3y ago
Complete AI layman here but it seems to me that the human mind's architecture has an overarching "executive" module that is involved in managing the numerous "expert" systems that do other stuff for us. (split brain experiments demonstrate multiple "minds" in the brain) Based on what we know about different systems in the brain it might be a stretch to expect hallucination free AGI, using only a single LLM.
- Swizec 3y ago
  > Based on what we know about different systems in the brain it might be a stretch to expect hallucination free AGI, using only a single LLM. Based on what we know about brains, it might be a stretch to expect hallucination free AGI. I’ve yet to meet a general intelligence that doesn’t hallucinate. Lots of fun examples from chickens who turn off when they see a straight line to cats who get spooked by cucumbers and the endless fun cognitive biases humans fall for.
pylua 3y ago
Sometimes hallucination is sort of a feature instead of a bug. For instance, if I ask it how to do something and it hallucinates , usually it is perfectly logical for it to work the way it suggests, even though it is wrong. In other words, it can be a good feature request or idea.
Fripplebubby 3y ago
The core argument in this paper it seems to me from scanning it is that because P != NP therefore LLMs will hallucinate answers to NP-complete problems. I think this is a clever point and an interesting philosophical question (about math, computer science, and language), but I think people are mostly trying to apply this using our commonsense notions of "LLM hallucination" rather than the formal notion they use in this paper, and I don't see an obvious connection, since commonsense hallucinations (eg inventing chapters of a novel when asked to produce summaries, inventing specific details when asked) don't seem to be NP-complete problems but rather are hallucinatory for some other interesting reason. (I apologize if I have not captured the paper correctly and would welcome correction on that, I read it quickly) The statement that the formal world (the world of math and logic and formal grammars) is a subset of the "real" world (or perhaps, the world of natural language) is really interesting to me as well. Most humans can't solve formal logic problems and parse formal grammars but don't suffer from a (strong) hallucination effect, and can work in natural language in great proficiency. Is hallucination inevitable in humans since we also can't solve certain NP-complete problems? We have finite lifespans, after all, so even with the capabilities we might never complete a certain problem.
- foobarian 3y ago
  Humans have some amount of ability to recognize they hit a wall and adjust accordingly. On the other hand this (completeness theorems, Kolmogorov complexity, complexity theory) was only arrived at what, in the 20th century?
  - digitalsushi 3y ago
    'Adjust accordingly' includes giving up and delivering something similar to what I asked, but not what I asked; is this the point at which the circle is complete and AI has fully replaced my dev team?
    - foobarian 3y ago
      Well in the example of an NP complete problem, a human might realize they are having trouble coming up with an optimal solution and start analyzing complexity. And once they have a proof might advise you accordingly and perhaps suggest a good enough heuristic.
ninetyninenine 3y ago
Fiction and story writing is hallucination. It is the opposite of a stochastic parrot. We've achieved both extremes of AI. Computers can be both logical machines and hallucinators. Our goal is to create a machine that can be both at the same time and can differentiate between both. That's the key. Hallucination is important but the key is for the computer to be self aware about when it's hallucinating. Of course it's a hard problem but even humans hallucinate massively. Just look at religion. Only one religion can be right or none, that must mean,logically speaking all other religions are hallucinations.
- beardedwizard 3y ago
  It is not the opposite of stochastic parrot, it is exactly the same thing only the predictions are worse due to sparse training data.
- elicksaur 3y ago
  Comparing religion to LLM mistakes is a case of the very prevalent anthropomorphism in society currently. I fear this misunderstanding and conflation will prevent us actually improving the tech. Coming to an incorrect explanation such as, “Helios pulls the Sun across the sky every day,” is categorically different than a math program incorrectly returning the most likely next token in a sequence. LLMs don’t have beliefs at all. Helios answers a question “Why does the Sun rise?” Holding such a belief shows a logical understanding that some force must be doing this, but due to a lack of knowledge of the world, the person comes up with an incorrect explanation. LLMs can’t pose and reason about such questions. It is just not the same class of “hallucinations.” Assuming we’ve figured out cognition via word prediction is going to get us nowhere fast in the long term.
  - ninetyninenine 3y ago
    No. I never said we figured out cognition. The LLM is still a black box feed forward network. It is the intricacies of how signals interact with each other in this network that we don't fully understand. Word prediction and curve fitting are high level concepts we used to build an intelligence we don't fully understand. Also belief and understanding are orthogonal concepts. I can believe something I don't understand and I can understand something I don't belief. My claim here is that LLMs understand things from a certain aspect because LLMs can produce output indistinguishable from understanding. Also because both the human brain and the LLM are black boxes there is no other metric we can use to gauge the level of understanding an LLM has other than comparing it's inputs and outputs to the human brain.
wseqyrku 3y ago
So they always hallucinate, it's just sometimes good enough?
throwawaaarrgh 3y ago
LLMs literally just place words one in front of another based on a probability and the "goodness" of training data. Of course it's going to make stuff up.
graemebenzie 3y ago
I think part of understanding is filling in the gaps between facts. AIs can't recognize when that gap is too large
graemebenzie 3y ago
I think understanding comes from filling the gaps between the facts. AIs can't tell when the gap between points of knowledge is too large to interpolate.
valine 3y ago
This paper is arguing that it’s impossible for an LLM to know the answer to every question, therefore it’s impossible to eliminate hallucination. It’s easy to imagine an LLM that responds “I don’t know” to all questions. An LLM like that isn’t very useful, but it also doesn’t hallucinate. Eliminating hallucination by teaching it to recognize what it doesn’t know is probably a more sane approach than teaching an LLM to know literally everything in the universe.
zyklonix 3y ago
Hallucinations are essential for divergent thinking. Not everything is solved following goal driven approaches. Check out DreamGPT: https://github.com/DivergentAI/dreamGPT https://github.com/DivergentAI/dreamGPT
zuminator 3y ago
This is sort of like the compression algorithm "problem." For the overwhelming majority of inputs, compression algorithms don't compress, and it can be proven that on average they don't work. But we're not really interested in compressing things on average. What we use compression for amounts to edge cases of highly regularized or repeatable data. Thus the fact that LLMs can be proven in general to hallucinate doesn't necessarily imply that they must hallucinate in the types of situations for which we use them for. The paper itself discusses a number of mitigating strategies -- such as supplementing their training data with current information or using multiple LLMs to vote on the accuracy of the outcome -- only to basically brush them aside and advise not to use LLMs in any sort of critical situation. And that's probably true enough today, but in the future I think these strategies will greatly reduce the severity of these hallucinations. Just as we as human beings have developed strategies to reduce our reliance on pure memory. This reminds me of a deposition I had to give a number of years back. One of the lawyers asked me if I remembered how the plaintiff and I came to discuss a certain accusation leveled at him by the defendant. And I confidently stated, "Sure, he and I used to have various conversations about the issue and one day he the plaintiff brought up this thing that defendant said to him." And the lawyer says, if you want to, you can refer to your phone text log to refresh your memory. Then I looked at my phone, and the truth was that I myself had spoken to the defendant, and she told me the accusation, and then I went and shared it with the plaintiff. So, I basically remembered the situation exactly backwards, i.e., a hallucination, which I was able to repair by referring to real world information instead of just my memory.
educaysean 3y ago
Is it a theoretical distinction of "we can't get to 0%, but we can virtually trivialize it by reducing its frequency down to to 1x10^-8%" type of scenario? Or is it something that requires an external layer of control?
DinaCoder99 3y ago
"Hallucination" implies perception of non-real things, not generation of phrases that map poorly to reality (or are simply incoherent). It seems like a really bad term for this phenomenon.
- TheBlight 3y ago
  "Bullsh***ing" seems more apt.
- earthwalker99 3y ago
  It makes being wrong sound impressive and mysterious, so it's here to stay.
- bonzaidrinkingb 3y ago
  We use "confabulation". It is a feature, not a bug. "confabulation" could be "solved" when LLMs realize they are uncertain on a reply and making things up. But that requires the LLM saying "I don't know" and rewarding that more than a wrong guess. That requires a change in loss functions and not even sure if all users desire that.
nedt 3y ago
Well humans believe that vacination either kills people or gives them chips for tracking and the top politicians are lizard people drinking the blood of children kept in caves and they had to fake a pandemic to get them out. I'd say an A.I. hallucinating isn't that far off from real humans. It's rather the recipient that needs to interpret any response from either.
- katehikes88 3y ago
  Have you considered that parts of what you said might be true but you ridicule it only because you associate with the others might be untrue and maybe even ridiculous?
  - nedt 3y ago
    It might be true, or not. It might be ridiculous, or not. With _it_ being a message from a human or an AI. A hallucination is not so much a problem as long as there is not blind trust or a single source of truth. And oh boy would I like to be pure ridiculous or satirical with the example of what humans are believing.
- bonzaidrinkingb 3y ago
  Around 12k fatal outcomes have been reported in the EU after vaccination, but it is not certain in all cases that vaccines were the cause. The vaccine tracking chips come from two Microsoft (-affiliate) patents, one about using chips to track body activity to reward in cryptocurrency, and another about putting a vaccine passport chip in the hands of African immigrants. That vaccines contain tracking chips is a fabricated conspiracy to ridicule and obfuscate. Lizard people is often an anti-semitic dog whistle. Rich elites use blood transfusions of young people to combat aging and age-related disease. Children have been kept in cages and feral children have lived in caves. You likely made up the part about faking a pandemic to get children out of caves, unless you can point to discussion about these beliefs. Real humans do hallucinate all the time.
  - dragonwriter 3y ago
whycome 3y ago
Maybe this is why we need to force LLMs to dream. To get all the hallucinations out of the way during ‘down time’ and clean things up for ‘waking hours’.
emrah 3y ago
There is a fine line between hallucination and creativity. When we know the exact answer and seek it, we label misfires "hallucination". Is it that the answer was not in the data set? Or that the LLM chose not to use it? Or something else? I find it ironic that people claim LLMs are creative and then tryy to eliminate hallucinations. Maybe we need an explicit switch to turn on/off creative elements in general
dorkwood 3y ago
These scientists could use a bit of coaching from HN's resident LLM experts. Don't they know that if you write a detailed enough prompt, and ask the LLM to explain itself each step along the way, that you can eliminate hallucination?

308 points by louthy 3y ago

> hallucination is defined as inconsistencies between a computable LLM and a computable ground truth function. That's simply inaccuracy or fabrication. Labelling it hallucination simply panders to the idea these programs are intelligent.

somewhereoutth 3y ago
Yes, imagine a pocket calculator that gave a completely wrong answer 10%? of the time, and calling it 'capricious' instead of simply broken.
- BlueTemplar 3y ago
  A real shame that Douglas Adams didn't live to see all of this...
- chrisjj 3y ago
  Indeed. The best solution would be to market it as intelligent ;)
tgv 3y ago
That statement is also a bit easy on the "computable ground truth." There is no such thing for the tasks we use an LLM for (unless you make up some definition that mangles the definition of each word).

paulsutter 3y ago

The models are just generating probable text. What’s amazing of how often the text is correct. It’s no surprise at all when it’s wrong Their bold confidence to be flat out wrong may be their most human trait

baq 3y ago
This again. They’re trained to generate probable text. The mechanisms created in the parameter blob during training to do that are basically a mystery and have to be pulled out of the model with digital brain surgery. E.g. LLMs are reasonable at chess and turns out somewhere in the blob there’s a chessboard representation, and you can make the model believe the board is in a different state by tweaking those parameters. So yeah they generate probable text, sure. Where they get the probabilities is a very good research problem.
- timr 3y ago
  > E.g. LLMs are reasonable at chess and turns out somewhere in the blob there’s a chessboard representation, and you can make the model believe the board is in a different state by tweaking those parameters. Broadly agreed, but there's no "representation"...the model has no memory, let alone a "concept" of a chessboard. It's just trained on a bunch of textual replays of chess games, and this works well enough for a game with finite probability space. Likewise, I have asked generic LLMs to generate "novel" drugs that solve particular problems, because their training sets have included lots of examples of drug molecules in string form (aka SMILES strings). This works far less well (because chemical space is much larger than a chess game), and the "novel" drugs usually end up looking like some mishmash of existing chemicals for the same problem. This can be a useful tool, but nobody is confusing it for a human brain doing problem solving. People are easily misled by the purported intelligence of these things because they assume that common examples of human intelligence are less probable than they really are. Our languages and games and stories are pretty regular, all things considered. Expand the probability space to something truly vast (even images work for this), and you can easily see the stochastic parrot emerge.
  - baq 3y ago
    The fact that tweaking parameters which appear to store the board makes it play according to the tweaked numbers instead of what was passed to it the context (i.e. working memory) directly contradicts your assertion that LLMs have no memory. The context is their memory. I can’t comment on your drug generation task - they aren’t magic, if the training didn’t result in a working drug model in the billions of params you’ll get garbage output, not very surprising. My point boils down to the ‘what’s more likely’ question: magical stochastic parrots which just by accident manage to create understandable and coherent responses to unreasonably large set of questions or the magic is actually some kind of a world model, or multiple, inside the blob of numbers, outputs of which steer the probabilities, just as this confirmed chess example. My bet is on learned world models because I’m not convinced there’s magic in our physical world.

sschueller 3y ago

You have to very carefully ask your question for it to not make things up. For example don't ask "how do I do this in in x?". Ask "can I do this with x?" These "AI" s are like "yes men". They will say anything to please you even if it's untrue or impossible. I have met people like that and they are very difficult to work with. You can't trust that they will deliver the project they promised and you always have to double check everything. You also can't trust them that what they promised is even possible.

deleted 3y ago
[deleted]
ithkuil 3y ago
The analogy is apt indeed. It's all about training and selection. As long as the incentives are that you either behave in that way "or else" it's unsurprising that we end up with a system that uses its intelligence to meet the goals you've set up. Fortunately that doesn't tell much about the limitations of the underlying intelligence but mostly about the limitations of the incentive structure you put around it
impulsivepuppet 3y ago
Before ChatGPT, human language translation had a similar problem but people weren't as vocal about it. What I find frustrating that it's increasingly challenging to have DeepL translate thou -> du, as this was my go-to "hack" to overcome the incompatibility of the English language due to its missing features. To somewhat remedy the "yes man" problem, one needs to become a pedantic mathematician about posing your questions and I don't believe that LLM technology alone is capable of overcoming it entirely. As silly as it sounds, I must concede to the existence of "prompt engineering" as I can forsee the development of abstractions aimed to decompose questions for you.
tudorw 3y ago
I've had some success with 'Ask follow up questions where additional clarity is required'. My best experiences start with a much more freeform conversation about what we are going to attempt to do, doing a Q&A first to make sure that both I and the AI are thinking about the same domain and sharing associated nomenclature seems to help.

golol 3y ago

This is just another diagonalization against some finite thing, like the no free lunch theorem. An "LLM" in their definition is essentially any finite thing which predicts the next token. The same result applies to your brain too, for example. Like all such diagonalization results, it is not really relevant for real world considerations. The reason is that it does not matter if your model fails on none, finitely many or infinitely many inputs. In reality the space of possible inputs is equipped with a probability measure, and the size of the hallucinating inputs set w.r.t. that measure is relevant. Diagonalization arguments usually, make no claim to the size of that set, and it is most likely negligible in the real world.

FeepingCreature 3y ago

It seems a stretch to call it "inevitable". "Inevitable given the current architecture without modifications" at most. Also, I'm missing a section on how (if) human brains manage to avoid hallucinations in this. Also, it doesn't have to never hallucinate, it just has to hallucinate less than we do.

bamboozled 3y ago
According to Buddhist philosophy, our whole identity is a hallucination :) I kind of concur.
- jetanoia 3y ago
  Username checks out :)
  - bamboozled 3y ago
    I'll honestly take this is a compliment.
SubNoize 3y ago
Or catch itself that it's hallucinating? I feel like humans would do that a fair bit. How often do we sit somewhere thinking about random scenarios that won't ever happen and are filled with wild thoughts and sometimes completely out of the world situations.. then we shake our heads and throw away the impossible from that thought train and only use what was based in reality
dkjaudyeqooe 3y ago
Because we have parts of our brain that supervise other parts of our brain and evaluate its output. For example: if you smoke pot and get paranoid, it's because pot dials back the work of the part of your brain that prunes thought paths that are not applicable. Normally, paranoid thoughts do not make sense, so they are discarded. That's also why you're more 'creative' when you smoke pot, less thought paths are pruned and more stuff that doesn't quite make sense gets through. Or thoughts that overly focus on some details get through, which are normally not required. Our brains are inherently "higher level", current AI is hopelessly simplistic by comparison.

t_mann 3y ago

I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute some functions, but the hallucination problem isn't about LLMs needing to know everything. The problem that we care about is the 'I don't know'-answering problem, which may still be computable.

az09mugen 3y ago
I think there is no easy way to make an LLM answer "I don't know". For that, it should learn among all the stuff ingested when people effectively don't know. But most people on internet write down irrelevant stuff even when they don't know instead of simply writing "I don't know". That's a very good point.
- timini 3y ago
  I think its fairly simple, it needs a certain level of proof e.g references to authoritative sources, if not say "i don't know".
  - cubefox 3y ago
    Then it is nothing more than a summarizer for search engine results.
    - amarant 3y ago
      A lot of people have said chat-gpt/copilot is a lot like having a robotic junior dev around. I think perhaps your description is more succinct
  - az09mugen 3y ago
    I'm really curious about one would implement that. By pondering weigths from certain sources ?

karol 3y ago

Organisms that evolved to perceive true reality instead of the "user interface" have smaller chances of survival. Donald Hoffman

tibbydudeza 3y ago
The brain fakes it to approximate reality - more so for reasons of the limitations of the wetware it runs on than anything else. For others here is a TED talk. https://www.youtube.com/watch?v=oYp5XuGYqqY https://www.youtube.com/watch?v=oYp5XuGYqqY

Borealid 3y ago

> hallucination is defined as inconsistencies between a computable LLM and a computable ground truth function. With this definition, you can trivially prove the titular sentence - "hallucination is inevitable" - is untrue. Let your LLM have a fixed input context length of one byte. Continue training the LLM until such a time as it replies to the input "A" with "yes" and all other inputs with "no". Define your computable ground truth function such that the correct output for the input "A" is "yes" and the correct output for all other inputs is "no". This LLM provable never hallucinates - we have exhaustively verified that its output matches the ground truth function for all possible inputs. There is nothing stopping inductively increasing the size of the input context and the number of entries in the ground truth table arbitrarily, and at no step do hallucinations become "inevitable".

resolutebat 3y ago
> Continue training the LLM until such a time as it replies to the input "A" with "yes" and all other inputs with "no". This is basically the same as saying "train your LLM until they never hallucinate", which reduces your claim to a tautology: an LLM trained not to hallucinate does not hallucinate. The trick is making that happen.
- ProxCoques 3y ago
  As I always tell my students: the solution to unreliable code is not to put the bugs in there in the first place.
  - paxcoder 3y ago
    [dead]
- Borealid 3y ago
  It's a tautology that for a given truth table mapping inputs to "correct" outputs there exists a function that produces that mapping. Saying that you can't train an LLM to NOT hallucinate is saying that it's impossible for any LLM to always produce output matching any particular truth table. There may exist truth tables where it's not possible to produce an LLM to match them (for some finite size of LLM perhaps), but my claim isn't a tautology - it's just an assertion that there exist some truth tables which an LLM can be trained to match. It may be tricky to make the LLM in the first place, but it's certainly not as tricky to verify it. You can test it with every input you consider to be present in the truth table, record its results, and throw it away if it doesn't match. The only possible results here are either non-convergeance or a "perfect" LLM. You can get rid of the non-convergeance if you bound the size of the LLM and literally iteratively verify every single possible model. The authors of the paper didn't do that. For trivially sized models and inputs, that's completely possible. For a 7B parameter model, nobody is doing that ever. But you can prove the title statement wrong with a ten-parameter model and a one-byte input.

Sparkyte 3y ago

Definitely a given, it isn't like AI has an actual brain capable of resolving and forming new connections. The LLM and human brains is that LLMs are interactive compendiums and our brains organize and sort information that ensures survival as an organism. There is no survival of whether or not LLMs are accurate and a machine wouldn't understand what is good or bad without weighted context. Its good for analyze, process, storing, retrieving and decomposing information. It isn't good at understanding, validating and forming connections between the things it says and what you want of it. It lacks comprehension, it doesn't lack composure.

golol 3y ago
The result in the theorem applies to your brain. Your brain can be modelled as an LLM in the sense of the paper up to arbitrarily small error. The result is a diagonalization argument that is not very relevant for the real world.
- wredue 3y ago
  >your brain can be modelled as an LLM in the sense of the paper The vast majority of people actually writing LLMs don’t claim this, and in fact, actually claim the very opposite: that LLMs do not accurately model a human brain in any capacity. The fact is that science has no clue what happens in the nucleus of a neuron, so claiming that computer scientists must is… well. You fill in the word.
  - golol 3y ago
    Definition 2 (Large Language Model). Let S be a computable seta of all the finite-length strings of alphabet A and (s0, s1, . . .) be an one-to-one enumeration of all the elements in S. A large language model, denoted h, is a function that completes the input string s ∈ S using the function’s predicted tokens h(s), in a finite time. Function h is attained procedurally using a set of training samples of input-completion pairs. For an arbtrarily large duration and an arbitrarily small error with respect to any definition of error you choose, there exists an LLM in the above sense which models the dynamics of your brain with that small error.

carlossouza 3y ago

Someone smart once said: If it is good, we call it "creativity." If it is bad, we call it "hallucination." This isn't a bug (or limitation, as the authors say). It's a feature.

tgv 3y ago
Asking it to write code for you is basically asking it to hallucinate.
- gardenhedge 3y ago
  I don't think so. I think it's asking it to repeat code it has been trained on
  - tgv 3y ago
    Search for a piece of code you wrote. If it's more than 15 lines and not boilerplate, chances are you won't find it anywhere on the net.
    - gardenhedge 3y ago
      To be honest, I imagine I would. Variables named could be different but it would largely be the same as code others have written. For example, I am creating an app in React Native at the moment. My app will not be unique in terms of code but instead in business domain.
    - Jensson 3y ago
      But there are many pieces of code that I've written that you can find in many places on the net, having a tool that can adapt that to your codebase in seconds is useful. It doesn't have to be smart, just pasting in an function and fitting that to your code is useful.
      - tgv 3y ago
        Sure, but the point is it will have to adapt it to your code, if only in naming. So it has to make up things, i.e. hallucinate. It can't just reproduce the best match in memory.

precompute 3y ago

Ah, the AI hype is now entering the "let's be real" phase. Haven't seen a frenzied post on alignment in a while now.

Culonavirus 3y ago
The hype is insane. Listen, I think LLMs still have a lot of room to grow and they're already very useful, but like some excellent researchers say, they're not the holy grail. If we want AGI, LLMs are not it. A lot of people seem to think this is an engineering issue and that LLMs can get us there, but they can't, because it is not an engineering issue.
- hnfong 3y ago
  Do you have evidence to back your claims up besides "the hype is overblown"? Because hype only indicates that the precise hyped up claims are wrong, it doesn't imply the opposite extreme (i.e. LLMs can never achieve AGI) must be true.
  - deleted 3y ago
    [deleted]
- mistermann 3y ago
  What kind(s) of an issue do you think it is fundamentally?
- PopePompus 3y ago
  I don't think you can say with confidence that the LLM approach will not lead to AGI, unless you understand in detail how human intelligence operates, and can show that no modification to current LLM architectures can achieve the same or superior results. I think the fact that adding "attention" to LLMs made a huge difference means that we are probably still in the low hanging fruit stage of LLM architecture development, and a few more design improvements on a par with "attention" might lead to something that could legitimately be called AGI. Many people, myself included, believe that LLMs are now exhibiting emergent behavior properties. If that's true, then saying that LLMs are not intelligent because they just predict the next token of output is like saying collections of neurons cannot be intelligent because they just stimulate other neurons chemically.

bbor 3y ago

I’m sorry… does this paper just point out that LLMs by definition are not as good at holding data as a direct database? Cause A) duh and b) who cares, they’re intuitive language transformers, not knowledge models. Maybe I’m missing something obvious? This seems like someone torturing math to imply outlandish conclusions that fit their (in this case anti-“AI”) agenda.

anonylizard 3y ago
It at least disproves LLMs from being 'god models'. They will never be able to solve every problem perfectly.
- ninetyninenine 3y ago
  Humans aren't God models either. The goal is to get this thing to the level of a human. God like levels are not possible imo.

TylerE 3y ago

I miss the days when HN posts about hallucinating were about microdosing.

gokulkrishh09 3y ago

[dead]

hyperpape 3y ago

The result seems to rely on stipulating the LLM must answer true or false to all its questions, and can't say "I don't know." So it's an interesting result, but it's not obvious that it tells us much about our actual problem, which is 100% about how to get a system that accurately understand the level of confidence it should have.

cjdell 3y ago

I thought the industry was already experimenting with the idea that you have another LLM observing the output of the primary LLM which is trained more towards safety than creativity. On top of that it would be good if the safety LLM could give a confidence score in the answer given by the main LLM. Then you can try multiple attempts with different parameters and only show the highest confidence answer to the user.

franze 3y ago

Hallucinations and Ideas are the same thing.

DebtDeflation 3y ago

There used to be an entire sub-field of NLP called Open Domain Question Answering (ODQA). It extensively studied the problem of selecting the best answer from the set of plausible answers and devised a number of potential strategies. Like everything else in AI/ML it fell victim to the "bitter lesson", in this case that scaling up "predict the next token" beats an ensemble of specialized linguistic-based methods.

falling_myshkin 3y ago
For those who don't know: http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html I agree with you for the NLP domain, but I wonder if there will also be a bitter lesson learned about the perceived generality of language for universal applications.

sgammon 3y ago

Splinters inevitable: An innate limitation of working with wood /s

HKH2 3y ago
People expect splinter-free epistemology instead of just putting on some gloves.

Scene_Cast2 3y ago

I don't think anyone has mentioned Bayesian Neural Nets (I forget the exact term). Sure, the paradigm adds an order of magnitude overhead (at least - and that's why I've never seen it used in the industry), but you can bolt it on to existing architectures. The basic idea is that besides the probabilities, the network also spits out confidence (IIRC based on how out-of-distribution the input is). There's been a ton of work on getting confidence values out of existing neural nets without as much overhead, but I've never seen those approaches replicate in the industry.

wnkrshm 3y ago
I would imagine that to propagate any confidence value through the system you'd need to have priors for the confidence of correctness for all data in your training set. (and those priors change over time)

somewhereoutth 3y ago

For production systems the considerations are: - how often does it error? - how bad are the errors? - how tolerable are the errors? - how detectable are the errors? - how recoverable are the errors? For example, a pocket calculator that occasionally was out at the 3rd decimal place might do much more damage than one that quite regularly returned NaN. LLMs error both regularly and badly, so detectability and recoverability are going to be crucial for useful deployment in systems outside of those that have a high tolerance for errors (e.g. algorithmic feeds).

intended 3y ago
Yes. If you want to make something that works, your life is figuring out evaluation and LLM ops. At that point, you know its not thinking, its doing token prediction.

macrolime 3y ago

They define an LLM as "a probabilistic model of a string that conditions the output at time t based on all the tokens that come before it in the string". I think that definition is wide enough to include human intelligence, so their finding should be equally valid for humans.

jddj 3y ago
Silk silk silk silk silk silk silk. What do cows drink?
moooo99 3y ago
> I think that definition is wide enough to include human intelligence, so their finding should be equally valid for humans. Which is definitely true. Human memory and the ability to correctly recall things we though we remembered is affected by a whole bunch of things and at times very unreliable. However, human intelligence, unlike LLMs, is not limited to recalling information we once learned. We are also able to do logical reasoning, which seems to improve in LLMs, but is far from being perfect. Another problem is how different we treat the reliability of information depending on the source, especially based on personal bias. I think that is a huge factor, because in my experience, LLMs tend to quickly fall over and change their opinion based on user input.
- ben_w 3y ago
  We can do logical reasoning, but we're very bad at it and often take shortcuts either via pattern matching, memory, or "common sense". Baseball and bat together cost $1.10, the bat is $1 more than the ball, how much does the ball cost? A French plane filled with Spanish passengers crashes over Italy, where are the survivors buried? An armed man enters a store, tells the cashier to hand over the money, and when he departs the cashier calls the police. Was this a robbery?
  - mistermann 3y ago
    Humans also have various culturally flavored, implicit "you know what I mean" algorithms on each end to smooth out "irrelevant" misunderstandings and ensure a cordial interaction, a cultural prime directive.

Almondsetat 3y ago

Of course it's inevitable. Things can be facts or deductions of facts (or both). If I ask an LLM the date of birth of Napoleon and it doesn't have it in its dataset there are only 2 options: either it has other facts from which Napoleon's birthday can be deduced or it doesn't. If it does then by improving the LLM we will be able to make more and more deductions, it if doesn't then it can only hallucinate. Since there will always be a set of facts that the LLM is not aware of and which cannot be deduced we will always have hallucinations

jtc331 3y ago
Is this a way of saying that large language models don’t have the concept of “I don’t know”?
andybak 3y ago
Your "Of course" is a red flag. Unless you have a very precise definition of "LLM" then there's no "of course". It's possible to imagine a hypothetical software system capable of returning "I don't know" or "I'm not sure". I haven't RTFA but I presume they are arguing within specific constraints. The important point is - could an augmented LLM or an "LLM plus something else" solve this. I don't have an answer to that but I don't think it's an "of course" type answer.
samatman 3y ago
I don't see why that's inevitable at all. I immediately recognize that I don't happen to know Napoleon's birthday, and act on that basis.

keepamovin 3y ago

This is why you need to pair language learning with real world experience. These robots need to be given a world to explore -- even a virtual one -- and have consequences within, and to survive it. Otherwise it's all unrooted sign and symbol systems untethered to experience.

phh 3y ago
I think I agree with you (I even upvoted), but this might be an anthropomorphism. Back like 3-5 years ago, we already thought that about LLMs: They couldn't answer questions about what would fall when stuff are attached together in some non-obvious way, and the argument back then was that you had to /experience/ it to realize it. But LLMs have long fixed those kind of issues. The way LLMs "resolve" questions is very different from us. At this point, I think that if we want to prove that LLMs need to be rooted in the real world to achieve intelligence, we need to find some real-world phenomenon that is so obvious that noone ever wrote about it... but then we'd have written about it?
- barnacs 3y ago
  Think of it this way: Intelligent beings in the real world have a very complex built-in biological error function rooted in real world experiences: sensory inputs, feelings, physical and temporal limitations and so on. You feel pain, joy, fear, have a limited lifetime, etc. "AI" on the other hand only have an external error function, usually roughly designed to minimize the difference of the output from that of an actually intelligent real world being.
  - keepamovin 3y ago
    Yeah, that's exactly my thinking man. We have to root intelligence in the real world, otherwise it will endlessly spin in these abstract loops. Akin to how logic -- untethered by emotion, intuition and experience (wisdom, maybe if you want? Understanding? Sure) -- can justify any obscene conclusion, and can not discriminate between morality. Reward functions or a system of values -- these things are rooted in real world experience. Logic is required, sure, but insufficient. At least alone! Haha :)

pfdietz 3y ago

A hallucinator seems like an excellent thing to have as a component to an intelligent system, but it must be coupled with evaluators. Such an architecture seems plausible for the human brain as well.

viktour19 3y ago

If hallucination is inevitable? What should developers do? Design user experiences that align users with this behaviour! Relatedly, I built a game to demonstrate how one might calibrate users to the responses of LLMs: https://news.ycombinator.com/item?id=39255583 https://news.ycombinator.com/item?id=39255583

demondemidi 3y ago

I bet that, basically, an LLM is just a part of a future AI. The same way that a distributor is part of an internal combustion engine. Or the way Minsk’s described The Society of Mind. Eager to see if an LLM can be bolted on to some new kind of giant model that does something like mitigate the LLMs weaknesses. Maybe it’ll be a collection of radically different models working together in 20 years, and not just a single model. Like, uh, our own brains. It reminds me of how lex and yacc are super important as grammar and tokenizer, but are only just the front end to much larger projects.

WirelessGigabit 3y ago

I feel the same way about information provided by LLMs as with lots of pre-LLM articles and overall YouTube 'information' videos. Sources matter. You're either rehashing information from a collection of sources or you have your own data to prove your statements.

coffeefirst 3y ago
Yeah, this is why I really like Kagi's approach: it's clearly marked for what is, and cites its sources so you can verify the quality of the answer (or at least get a ballpark idea of where it's coming from) yourself.

lebuffon 3y ago

Complete AI layman here but it seems to me that the human mind's architecture has an overarching "executive" module that is involved in managing the numerous "expert" systems that do other stuff for us. (split brain experiments demonstrate multiple "minds" in the brain) Based on what we know about different systems in the brain it might be a stretch to expect hallucination free AGI, using only a single LLM.

Swizec 3y ago
> Based on what we know about different systems in the brain it might be a stretch to expect hallucination free AGI, using only a single LLM. Based on what we know about brains, it might be a stretch to expect hallucination free AGI. I’ve yet to meet a general intelligence that doesn’t hallucinate. Lots of fun examples from chickens who turn off when they see a straight line to cats who get spooked by cucumbers and the endless fun cognitive biases humans fall for.

pylua 3y ago

Sometimes hallucination is sort of a feature instead of a bug. For instance, if I ask it how to do something and it hallucinates , usually it is perfectly logical for it to work the way it suggests, even though it is wrong. In other words, it can be a good feature request or idea.

Fripplebubby 3y ago

The core argument in this paper it seems to me from scanning it is that because P != NP therefore LLMs will hallucinate answers to NP-complete problems. I think this is a clever point and an interesting philosophical question (about math, computer science, and language), but I think people are mostly trying to apply this using our commonsense notions of "LLM hallucination" rather than the formal notion they use in this paper, and I don't see an obvious connection, since commonsense hallucinations (eg inventing chapters of a novel when asked to produce summaries, inventing specific details when asked) don't seem to be NP-complete problems but rather are hallucinatory for some other interesting reason. (I apologize if I have not captured the paper correctly and would welcome correction on that, I read it quickly) The statement that the formal world (the world of math and logic and formal grammars) is a subset of the "real" world (or perhaps, the world of natural language) is really interesting to me as well. Most humans can't solve formal logic problems and parse formal grammars but don't suffer from a (strong) hallucination effect, and can work in natural language in great proficiency. Is hallucination inevitable in humans since we also can't solve certain NP-complete problems? We have finite lifespans, after all, so even with the capabilities we might never complete a certain problem.

foobarian 3y ago
Humans have some amount of ability to recognize they hit a wall and adjust accordingly. On the other hand this (completeness theorems, Kolmogorov complexity, complexity theory) was only arrived at what, in the 20th century?
- digitalsushi 3y ago
  'Adjust accordingly' includes giving up and delivering something similar to what I asked, but not what I asked; is this the point at which the circle is complete and AI has fully replaced my dev team?
  - foobarian 3y ago
    Well in the example of an NP complete problem, a human might realize they are having trouble coming up with an optimal solution and start analyzing complexity. And once they have a proof might advise you accordingly and perhaps suggest a good enough heuristic.

ninetyninenine 3y ago

Fiction and story writing is hallucination. It is the opposite of a stochastic parrot. We've achieved both extremes of AI. Computers can be both logical machines and hallucinators. Our goal is to create a machine that can be both at the same time and can differentiate between both. That's the key. Hallucination is important but the key is for the computer to be self aware about when it's hallucinating. Of course it's a hard problem but even humans hallucinate massively. Just look at religion. Only one religion can be right or none, that must mean,logically speaking all other religions are hallucinations.

beardedwizard 3y ago
It is not the opposite of stochastic parrot, it is exactly the same thing only the predictions are worse due to sparse training data.
elicksaur 3y ago
Comparing religion to LLM mistakes is a case of the very prevalent anthropomorphism in society currently. I fear this misunderstanding and conflation will prevent us actually improving the tech. Coming to an incorrect explanation such as, “Helios pulls the Sun across the sky every day,” is categorically different than a math program incorrectly returning the most likely next token in a sequence. LLMs don’t have beliefs at all. Helios answers a question “Why does the Sun rise?” Holding such a belief shows a logical understanding that some force must be doing this, but due to a lack of knowledge of the world, the person comes up with an incorrect explanation. LLMs can’t pose and reason about such questions. It is just not the same class of “hallucinations.” Assuming we’ve figured out cognition via word prediction is going to get us nowhere fast in the long term.
- ninetyninenine 3y ago
  No. I never said we figured out cognition. The LLM is still a black box feed forward network. It is the intricacies of how signals interact with each other in this network that we don't fully understand. Word prediction and curve fitting are high level concepts we used to build an intelligence we don't fully understand. Also belief and understanding are orthogonal concepts. I can believe something I don't understand and I can understand something I don't belief. My claim here is that LLMs understand things from a certain aspect because LLMs can produce output indistinguishable from understanding. Also because both the human brain and the LLM are black boxes there is no other metric we can use to gauge the level of understanding an LLM has other than comparing it's inputs and outputs to the human brain.

wseqyrku 3y ago

So they always hallucinate, it's just sometimes good enough?

throwawaaarrgh 3y ago

LLMs literally just place words one in front of another based on a probability and the "goodness" of training data. Of course it's going to make stuff up.

graemebenzie 3y ago

I think part of understanding is filling in the gaps between facts. AIs can't recognize when that gap is too large

graemebenzie 3y ago

I think understanding comes from filling the gaps between the facts. AIs can't tell when the gap between points of knowledge is too large to interpolate.

valine 3y ago

This paper is arguing that it’s impossible for an LLM to know the answer to every question, therefore it’s impossible to eliminate hallucination. It’s easy to imagine an LLM that responds “I don’t know” to all questions. An LLM like that isn’t very useful, but it also doesn’t hallucinate. Eliminating hallucination by teaching it to recognize what it doesn’t know is probably a more sane approach than teaching an LLM to know literally everything in the universe.

zyklonix 3y ago

Hallucinations are essential for divergent thinking. Not everything is solved following goal driven approaches. Check out DreamGPT: https://github.com/DivergentAI/dreamGPT https://github.com/DivergentAI/dreamGPT

zuminator 3y ago

This is sort of like the compression algorithm "problem." For the overwhelming majority of inputs, compression algorithms don't compress, and it can be proven that on average they don't work. But we're not really interested in compressing things on average. What we use compression for amounts to edge cases of highly regularized or repeatable data. Thus the fact that LLMs can be proven in general to hallucinate doesn't necessarily imply that they must hallucinate in the types of situations for which we use them for. The paper itself discusses a number of mitigating strategies -- such as supplementing their training data with current information or using multiple LLMs to vote on the accuracy of the outcome -- only to basically brush them aside and advise not to use LLMs in any sort of critical situation. And that's probably true enough today, but in the future I think these strategies will greatly reduce the severity of these hallucinations. Just as we as human beings have developed strategies to reduce our reliance on pure memory. This reminds me of a deposition I had to give a number of years back. One of the lawyers asked me if I remembered how the plaintiff and I came to discuss a certain accusation leveled at him by the defendant. And I confidently stated, "Sure, he and I used to have various conversations about the issue and one day he the plaintiff brought up this thing that defendant said to him." And the lawyer says, if you want to, you can refer to your phone text log to refresh your memory. Then I looked at my phone, and the truth was that I myself had spoken to the defendant, and she told me the accusation, and then I went and shared it with the plaintiff. So, I basically remembered the situation exactly backwards, i.e., a hallucination, which I was able to repair by referring to real world information instead of just my memory.

educaysean 3y ago

Is it a theoretical distinction of "we can't get to 0%, but we can virtually trivialize it by reducing its frequency down to to 1x10^-8%" type of scenario? Or is it something that requires an external layer of control?

DinaCoder99 3y ago

"Hallucination" implies perception of non-real things, not generation of phrases that map poorly to reality (or are simply incoherent). It seems like a really bad term for this phenomenon.

TheBlight 3y ago
"Bullsh***ing" seems more apt.
earthwalker99 3y ago
It makes being wrong sound impressive and mysterious, so it's here to stay.
bonzaidrinkingb 3y ago
We use "confabulation". It is a feature, not a bug. "confabulation" could be "solved" when LLMs realize they are uncertain on a reply and making things up. But that requires the LLM saying "I don't know" and rewarding that more than a wrong guess. That requires a change in loss functions and not even sure if all users desire that.

nedt 3y ago

Well humans believe that vacination either kills people or gives them chips for tracking and the top politicians are lizard people drinking the blood of children kept in caves and they had to fake a pandemic to get them out. I'd say an A.I. hallucinating isn't that far off from real humans. It's rather the recipient that needs to interpret any response from either.

katehikes88 3y ago
Have you considered that parts of what you said might be true but you ridicule it only because you associate with the others might be untrue and maybe even ridiculous?
- nedt 3y ago
  It might be true, or not. It might be ridiculous, or not. With _it_ being a message from a human or an AI. A hallucination is not so much a problem as long as there is not blind trust or a single source of truth. And oh boy would I like to be pure ridiculous or satirical with the example of what humans are believing.
bonzaidrinkingb 3y ago
Around 12k fatal outcomes have been reported in the EU after vaccination, but it is not certain in all cases that vaccines were the cause. The vaccine tracking chips come from two Microsoft (-affiliate) patents, one about using chips to track body activity to reward in cryptocurrency, and another about putting a vaccine passport chip in the hands of African immigrants. That vaccines contain tracking chips is a fabricated conspiracy to ridicule and obfuscate. Lizard people is often an anti-semitic dog whistle. Rich elites use blood transfusions of young people to combat aging and age-related disease. Children have been kept in cages and feral children have lived in caves. You likely made up the part about faking a pandemic to get children out of caves, unless you can point to discussion about these beliefs. Real humans do hallucinate all the time.
- dragonwriter 3y ago

whycome 3y ago

Maybe this is why we need to force LLMs to dream. To get all the hallucinations out of the way during ‘down time’ and clean things up for ‘waking hours’.

emrah 3y ago

There is a fine line between hallucination and creativity. When we know the exact answer and seek it, we label misfires "hallucination". Is it that the answer was not in the data set? Or that the LLM chose not to use it? Or something else? I find it ironic that people claim LLMs are creative and then tryy to eliminate hallucinations. Maybe we need an explicit switch to turn on/off creative elements in general

dorkwood 3y ago

These scientists could use a bit of coaching from HN's resident LLM experts. Don't they know that if you write a detailed enough prompt, and ask the LLM to explain itself each step along the way, that you can eliminate hallucination?