23 ms·
I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of w
by t_mann 3y ago
I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations).
It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute some functions, but the hallucination problem isn't about LLMs needing to know everything. The problem that we care about is the 'I don't know'-answering problem, which may still be computable.
- az09mugen 3y agoI think there is no easy way to make an LLM answer "I don't know". For that, it should learn among all the stuff ingested when people effectively don't know. But most people on internet write down irrelevant stuff even when they don't know instead of simply writing "I don't know". That's a very good point.
- timini 3y agoI think its fairly simple, it needs a certain level of proof e.g references to authoritative sources, if not say "i don't know".
- cubefox 3y agoThen it is nothing more than a summarizer for search engine results.
- amarant 3y agoA lot of people have said chat-gpt/copilot is a lot like having a robotic junior dev around. I think perhaps your description is more succinct
- az09mugen 3y agoI'm really curious about one would implement that. By pondering weigths from certain sources ?
- Certhas 3y agoLLMs don't have a concept of sources for their statements. Ask them to give you some literature recommendations on something it has explained to you. You'll get plenty of plausible sounding papers that don't exist. Humans know to some extent why they know (read it in a text book, colleague mentioned it). LLMs don't seem to.
- card_zero 3y agoThey read it in a non-existent average interpolation of the books actual humans read similar things in.
- mike_hearn 3y agoAsk a human to provide accurate citations for any random thing they know and they won't be able to do a good job either. They'd probably have to search to find it, even if they know they got it from a document originally and have some clear memory of what it said.
- Jensson 3y agoYes, humans wont lie to you about it, they will research and come up with sources. Current LLM doesn't do that when asked for sources (unless they invoke a tool), they come back to you with hallucinated links that looks like links it was trained on.
- mike_hearn 3y agoUnfortunately it's not an uncommon experience when reading academic papers in some fields to find citations that, when checked, don't actually support the cited claim or sometimes don't even contain it. The papers will exist but beyond that they might as well be "hallucinations".
- Jensson 3y agoHumans can speak bullshit when they don't want to put in the effort, these LLMs always do it. That is the difference. We need to create the part that humans do when they do the deliberate work to properly create those sources etc, that kind of thinking isn't captured in the text so LLMs doesn't learn it.
- barrkel 3y agoLLMs are token completion engines. The correspondence of the text to the truth or authoritative sources is a function of being trained on text like that; with the additional wrinkle that generalization from training (a desired property or it's just a memorization engine) will produce text which is only plausibly truthful, it only resembles training data. Getting beyond this is a tricky dark art. There isn't any simple there. There's nowhere to put an if statement.
- rini17 3y agoMaybe it needs some memory retrieval step that can measure the confidence - whether there's anything related to the prompt. No idea how to train a LLM to do that.
- dmd 3y agoConsider the extremely common Amazon product question section, where you see Q: Will this product fit my Frobnitz 123? A: I don't know, I ended up buying something else. Q: Does it come with batteries? A: IDK I RETURN IT
- golol 3y agoI can assure you it has no relevance for people working with LLMs, as the result includes your brain, for example.
- svantana 3y agoAccording to their definition, answering "I don't know" is also a hallucination. Even worse, the truth function is deliberately designed to trip up the models, it has no connection to any real-world truth. So for example, if the input is "what is 2 + 5?" and the LLM answers "7", - their truth function will say that's a hallucination, the correct answer is "banana".
- cornholio 3y agoTransformers have no capacity for self reflection, for reasoning about their reasoning process, they don't "know" that they don't know. My interpretation of the paper is that it claims this weakness if fundamental, you can train the network to act as if it knows its knowledge limits, but there will always be an impossible to cover gap for any real world implementation.
- GaggiX 3y agoDo you have a source? That's also what I thought but I wouldn't be surprised if the model learned to identify its own perplexity during the reinforcement learning phase.
- ddalex 3y agoActually it seems to me that they do... I asked via custom prompts the various GPTs to give me scores for accuracy, precision and confidence for its answer (in range 0-1), and then I instructed them to stop generating when they feel the scores will be under .9, which seems to pretty much stop the hallucination. I added this as a suffix to my queries.
- smusamashah 3y agoAny examples?
- ddalex 3y agoJust a random example: > After you answer the question below, output a JSON a rating score of the quality of the answer in three dimensions: `confidence`, `clarity` and `certainty', all in range 0 to 1, where 0 is the worst, and 1 is the best. Strive for highest score possible. Make sure the rating is the last thing written as to be parsed by machine. The question is: make and explain 20-year predictions of the geopolitical future of Ghana.
- Sai_ 3y agoIf LLMs can self reflect and accurately score themselves on your three dimensions, why are they spending money on RHLF? They wouldn’t be wasting all that time and money if the machine could self reflect.
- drdrek 3y agoNot saying anything about LLM But in CS in general many issues "cannot be solved" or "Cannot be solved in reasonable time (NP)" but approximations upper bound by some value are solvable in reasonable time (P). And in the real world if the truck route of amazon is 20% off the mathematically optimal solution the traveling salesman is "Solved" in a good enough way.
- startupsfail 3y agoThe claim of the paper is that computation is irreducible (assuming P!=NP), LLMs have limited computational capacity and will hallucinate on the irreducible problems. I don’t know, the claim seems dubious to me. We usually are able to have algorithms that return a failure status, when the problem proved to be too large. Avoiding the “hallucination”. Don’t see why LLMs can’t have that embedded.
- nottorp 3y ago> we can get LLMs to answer 'I don't know' more often Have any nets been trained specifically to be able to go to an 'i don't know' state, I wonder? It may be the humans' fault.
- deleted 3y ago[deleted]
- gessha 3y agoYes, you can find some of the work on this topic under the terms open world recognition or open world X where X is a topic in computer vision or NLP. https://arxiv.org/abs/2011.12906 https://arxiv.org/abs/2011.12906
- nottorp 3y agoMaybe, but are the LLM churches doing it?
- intended 3y agoIf a model can say ‘I don’t know’, then the hallucination problem would also be solved. When we say “know” it usually means being factual. For an LLM to ‘know’ it doesn’t know, it would have had to move away from pure correlations on words, and meta processing about its own results. I can see this happen with two LLMs working together (and there are Evals that use just this), however each LLM still has no self awareness of its limits. This was a terribly convoluted argument to make.
- empath-nirvana 3y agoThe models that exist now say "I don't know" all the time. It's so weird that people keep insisting that it can't do things that it does. Ask it what dark matter is, and it won't invent an answer, it will present existing theories and say that it's unknown. Ask it about a person you know that isn't in it's data set and it'll tell you it has no information about the person. Despite the fact that people insist that hallucinations are common and that it will invent answers if it doesn't know something frequently, the truth is that chatgpt doesn't hallucinate that much and will frequently say it doesn't know things. One of the few cases where I've noticed it inventing things are that it often makes up apis for programming libraries and CLI tools that don't exist, and that's trivially fixable by referring it to documentation.
- intended 3y agoI have to use LLMs for work projects - which are not PoCs. I can’t have a tool that makes up stuff an unknown amount of time. There is a world of research examining hallucination Rates, indicating hallucination rates of 30%+. With steps to reduce it using RAGs, you could potentially improve the results significantly - last I checked it was 80-90%. And the failure types aren’t just accuracy, it’s precision, recall, relevance and more.
- empath-nirvana 3y ago> There is a world of research examining hallucination Rates, indicating hallucination rates of 30%+. I want to see a citation for this. And a clear definition for what is a hallucination and what isn't.
- andsoitis 3y ago> the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). To answer "I don't know" requires one to know when you know. To know when you know in turn requires understanding.
- MuffinFlavored 3y agohow did LLMs get this far without any concept of understanding? how much further can they go until they become “close enough”?
- Karellen 3y agoThey generate text which looks like the kind of text that people who do have understanding generate.
- ninetyninenine 3y agoTwo key things here to realize. People also often don't understand things and have trouble separating fact from fiction. By logic only one religion or no religion is true. Consequently also by logic most religions in the world where their followers believe the religion to be true are hallucinating. The second thing to realize that your argument doesn't really apply. Its in theory possible to create a stochastic parrot that can imitate to a degree of 100 percent the output of a human who truly understands things. It blurs the line of what is understanding. One can even define true understanding as a stochastic parrot that generated text indistinguishable total understanding.
- andsoitis 3y ago> People also often don't understand things and have trouble separating fact from fiction. That's not the point being argued. Understanding, critical thinking, knowledge, common sense, etc. all these things exist on a spectrum - both in principle and certainly in humans. In fact, in any particular human there are different levels of competence across these dimensions. What we are debating, is whether or not, an LLM can have understanding itself. One test is: can an LLM understand understanding? The human mind has come to the remarkable understanding that understanding itself is provisional and incomplete.
- somethingsaid 3y agoI also wonder if having a hallucination-free LLM is even required for it to be useful. Humans can and will hallucinate (by this I mean make false statements in full confidence, not drugs or mental states) and they’re entrusted with all sorts of responsibilities. Humans are also susceptible to illusions and misdirection just like LLMs. So in all likelihood there is simply some state of ‘good enough’ that is satisfactory for most tasks. Perusing the elimination of hallucinations to the nth degree may be a fools errand.
- skydhash 3y agoTools are not people and people should not be considered as tools. Imagine your hammer only hitting the nail 60% of the time! But workers should be allowed to stop working to negotiate work conditions.
- sandworm101 3y agoThey cannot say "I dont know" because they dont actually know anything. The answers are not comming from a thinking mind but a complex pattern-fitting supercomputer hovering over a massive table of precomputed patterns. It computes your input then looks to those patterns and spits out the best match. There is no thinking brain with a conceptual understanding of its own limitations. Getting an "i dont know" from current AI is like asking navigation software how far it is to the Simpsons house in Springfield: the machine spits out answers but cannot fathom the cultural reference that makes the answer impossible. Instead, it finds someone named simpson in the nearest realworld Springfield.
- williamcotton 3y agoWhat if you worked on the problem and tried to come up with some kind of solution?
- sandworm101 3y agoThe solution is older non-AI tech. Google search can say "no good results found" because it returns actual data rather than creating anything new. If you want a hard answer about the presence or absence of something, AI isnt the correct tool.
- williamcotton 3y agoSo there are no other possibilities for us other than using a system that can be gamed for substandard results? Are we sure about this?
- tempest_ 3y agoCan, but doesn't. I can't remember the last time google actually returned no results.
- tsimionescu 3y agoIt does reply with no results, but only for very long queries. E.g. If you search for two concatenated GUIDs, you can easily see a no results page.
- rf15 3y ago> I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often I feel like writing an entire paper about the practical approach to the problems posed in this paper, but you'll probably have to first formally define the language used in the training data before you can try to map it (through training and sampling algos, which this paper conveniently skipped) to the target form. This sounds really fun at first, but then we're once again talking about the strict formalisation of natural language (which you could still do - the training data is limited and fixed!)
- mr_toad 3y agoSaying “I don’t know” implies you understand what “I” means.