6 ms·

A big red flag for me was that Sundar was prompting the model to report lots of facts that can be either true or false. We all saw the benchmark figures that t

A big red flag for me was that Sundar was prompting the model to report lots of facts that can be either true or false. We all saw the benchmark figures that they published and the results mostly showed marginal improvements. In other words, the issue of hallucination has not been solved. But the demo seemed to imply that it had. My conclusion was that they had mostly cherry picked instances in which the model happened to report correct or consistent information. They oversold its capabilities, but it does still seem that multi-modal models are going to be a requirement for AI to converge on a consistent idea of what kinds of phenomena are truly likely to be observed across modalities. So it's a good step forward. Now if they can just show us convincingly that a given architecture is actually modeling causality.

LesZedCB 3y ago
i think this was demonstrated in that mark rober promo video[1] where he asked why the paper airplane stalled by blatantly leading the witness. "do you believe that a pocket of hot air would lead to lower air pressure causing my plane to stall?" he could barely even phrase the question correctly because it was so awkward. just embarrassing. [1] https://www.youtube.com/watch?v=mHZSrtl4zX0&t=277s https://www.youtube.com/watch?v=mHZSrtl4zX0&t=277s
- dom96 3y ago
  Yeah, this was so obvious too. Clearly Mark Rober tried to ask it what to try and got stupid answers, then tried to give it clues and had to get really specific before he got a usable answer.
- ryeights 3y ago
  This has got to be satire! That is too funny.
calf 3y ago
Ever since the "stochastic parrots" and "super-autocomplete" criticisms of LLMs, the question is whether hallucinations are solvable in principle at all. And if hallucinations are solvable, it would of such basic and fundamental scientific importance that I think would be another mini-breakthrough in AI.
- adriand 3y ago
  An interesting perspective on this I’ve heard discussed is whether hallucinations ought to be solved at all, or whether they are core to the way human intelligence works as well, in the sense that that is what is needed to produce narratives. I believe it is Hinton that prefers “confabulation” to “hallucination” because it’s more accurate. The example in the discussion about hallucination/confabulation was that of someone who had been present in the room during Nixon’s Watergate conversations. Interviewed about what he heard, he provided a narrative that got many facts wrong (who said what, and what exactly was said). Later, when audio tapes surfaced, the inaccuracies in his testimony became known. However, he had “confabulated truthfully”. That is, he had made up a narrative that fit his recall as best as he was able, and the gist of it was true. Without the ability to confabulate, he would have been unable to tell his story. (Incidentally, because I did not check the facts of what I just recounted, I just did the same thing…)
  - TerrifiedMouse 3y ago
    > Without the ability to confabulate, he would have been unable to tell his story. You can tell a story without making up fiction. Just say you don’t know when you don’t know. Inaccurate information is worse than no information.
    - lmm 3y ago
      > You can tell a story without making up fiction. Just say you don’t know when you don’t know. The point is that humans can't in general, because we don't actually know which parts of what we "remember" are real and which parts are our brain filling in the blanks. And maybe it's the same for nonhuman intelligences too.
      - eloisius 3y ago
        It’s hard even when you don’t take into consideration that you don’t know what you’ve misremembered. Try writing a memoir. You’ll realize you never actually remember what anyone says, but caveating your dialog with “and then I think they said something like” would make horrible reading.
plaidfuji 3y ago
These LLMs do not have a concept of factual correctness and are not trained/optimized as such. I find it laughable that people expect these things to act like quiz bots - this misunderstands the nature of a generative LLM entirely. It simply spits out whatever output sequence it feels is most likely to occur after your input sequence. How it defines “most likely” is the subject of much research, but to optimize for factual correctness is a completely different endeavor. In certain cases (like coding problems) it can sound smart enough because for certain prompts, the approximate consensus of all available text on the internet is pretty much true and is unpolluted by garbage content from laypeople. It is also good at generating generic fluffy “content” although the value of this feature escapes me. In the end the quality of the information it will get back to you is no better than the quality of a thorough google search.. it will just get you a more concise and well-formatted answer faster.
- eurekin 3y ago
  The first question I always ask myself in such cases: how much input data has a simple "I don't know" lines? This is clearly a concept (not knowing sth) that has to be learned in order to be expressed in the output.
  - visarga 3y ago
    What stops you from asking the same question multiple times, and seeing if the answers are consistent. I am sure the capital of France is always going to come out Paris, but the name of a river passing a small village might be hallucinated differently. Even better - use two different models, if they agree it's probably true. And probably the best - provide the data to the model in context, if you have a good source. Don't use the model as fact knowledge base, use RAG.
    - ntlk 3y ago
      Can’t speak for other people but I find it more time consuming to get ChatGPT to correct its mistakes than to do the work myself.
nblgbg 3y ago
The issue of hallucinations won't be solved with the RAG approach. It requires a fundamentally different architecture. These aren't my words but Yann LeCun's. You could easily understand if you spend some time playing around. The autoregressive nature won't allow the LLMs to create an internally consistent model before answering the question. We have approaches like Chain of Thought and others, but they are merely band-aids and superficially address the issue.
- golol 3y ago
  If you build a complex Chain if Thought style Agent and then train/finetune further by reinforcement learning with this architecture then it is not a band-aid anymore, it is an integral part of the model and the weights will optimize to make use of this CoT ability.
  - phire 3y ago
    It's been 3.5 years since GPT-3 was released, and just over a year since ChatGPT was released to the public. If it was possible to solve LLM hallucinations with simple Chain-of-Thought style agents, someone would have done that and released a product by now. The fact that nobody has released such a product, is pretty strong evidence that you can't fix hallucinations via Chain-of-Thought or Retrieval-Augmented Generation, or any other band-aid approaches.
    - wokwokwok 3y ago
      I agree: but I just wanted to say that there are specific subdomains where you can mitigate some of these issues. For example, generating json. You can explicitly follow a defined grammar to get what will always be a valid json output. Similarly, structured output such as code can be passed to other tools such as compilers, type checkers and test suites to ensure that at a minimum the output you selected passes some minimum threshold of “isn’t total rubbish”. For unstructured output this a much harder problem, and bluntly, it doesn’t seem like there’s any kind of meaningful solution to it. …but the current generation of LLMs are driven by probabilistic sampling functions. Over the probability curve you’ll always get some rubbish, but if you sample many times for structure and verifiable output you can, to a reasonable degree, mitigate the impact that hallucinations have. Currently that’s computationally expensive, to drive the chance of error down to a useful level, but compute scales. We may seem some quite reasonable outputs from similar architectures wrapped in validation frameworks in the future, I guess. …for, a very specific subset of types of output.
kristopolous 3y ago
I mean it's a demo. Isn't this kinda what they all do

LesZedCB 3y ago

i think this was demonstrated in that mark rober promo video[1] where he asked why the paper airplane stalled by blatantly leading the witness. "do you believe that a pocket of hot air would lead to lower air pressure causing my plane to stall?" he could barely even phrase the question correctly because it was so awkward. just embarrassing. [1] https://www.youtube.com/watch?v=mHZSrtl4zX0&t=277s https://www.youtube.com/watch?v=mHZSrtl4zX0&t=277s

dom96 3y ago
Yeah, this was so obvious too. Clearly Mark Rober tried to ask it what to try and got stupid answers, then tried to give it clues and had to get really specific before he got a usable answer.
ryeights 3y ago
This has got to be satire! That is too funny.

calf 3y ago

Ever since the "stochastic parrots" and "super-autocomplete" criticisms of LLMs, the question is whether hallucinations are solvable in principle at all. And if hallucinations are solvable, it would of such basic and fundamental scientific importance that I think would be another mini-breakthrough in AI.

adriand 3y ago
An interesting perspective on this I’ve heard discussed is whether hallucinations ought to be solved at all, or whether they are core to the way human intelligence works as well, in the sense that that is what is needed to produce narratives. I believe it is Hinton that prefers “confabulation” to “hallucination” because it’s more accurate. The example in the discussion about hallucination/confabulation was that of someone who had been present in the room during Nixon’s Watergate conversations. Interviewed about what he heard, he provided a narrative that got many facts wrong (who said what, and what exactly was said). Later, when audio tapes surfaced, the inaccuracies in his testimony became known. However, he had “confabulated truthfully”. That is, he had made up a narrative that fit his recall as best as he was able, and the gist of it was true. Without the ability to confabulate, he would have been unable to tell his story. (Incidentally, because I did not check the facts of what I just recounted, I just did the same thing…)
- TerrifiedMouse 3y ago
  > Without the ability to confabulate, he would have been unable to tell his story. You can tell a story without making up fiction. Just say you don’t know when you don’t know. Inaccurate information is worse than no information.
  - lmm 3y ago
    > You can tell a story without making up fiction. Just say you don’t know when you don’t know. The point is that humans can't in general, because we don't actually know which parts of what we "remember" are real and which parts are our brain filling in the blanks. And maybe it's the same for nonhuman intelligences too.
    - eloisius 3y ago
      It’s hard even when you don’t take into consideration that you don’t know what you’ve misremembered. Try writing a memoir. You’ll realize you never actually remember what anyone says, but caveating your dialog with “and then I think they said something like” would make horrible reading.

plaidfuji 3y ago

These LLMs do not have a concept of factual correctness and are not trained/optimized as such. I find it laughable that people expect these things to act like quiz bots - this misunderstands the nature of a generative LLM entirely. It simply spits out whatever output sequence it feels is most likely to occur after your input sequence. How it defines “most likely” is the subject of much research, but to optimize for factual correctness is a completely different endeavor. In certain cases (like coding problems) it can sound smart enough because for certain prompts, the approximate consensus of all available text on the internet is pretty much true and is unpolluted by garbage content from laypeople. It is also good at generating generic fluffy “content” although the value of this feature escapes me. In the end the quality of the information it will get back to you is no better than the quality of a thorough google search.. it will just get you a more concise and well-formatted answer faster.

eurekin 3y ago
The first question I always ask myself in such cases: how much input data has a simple "I don't know" lines? This is clearly a concept (not knowing sth) that has to be learned in order to be expressed in the output.
- visarga 3y ago
  What stops you from asking the same question multiple times, and seeing if the answers are consistent. I am sure the capital of France is always going to come out Paris, but the name of a river passing a small village might be hallucinated differently. Even better - use two different models, if they agree it's probably true. And probably the best - provide the data to the model in context, if you have a good source. Don't use the model as fact knowledge base, use RAG.
  - ntlk 3y ago
    Can’t speak for other people but I find it more time consuming to get ChatGPT to correct its mistakes than to do the work myself.

nblgbg 3y ago

The issue of hallucinations won't be solved with the RAG approach. It requires a fundamentally different architecture. These aren't my words but Yann LeCun's. You could easily understand if you spend some time playing around. The autoregressive nature won't allow the LLMs to create an internally consistent model before answering the question. We have approaches like Chain of Thought and others, but they are merely band-aids and superficially address the issue.

golol 3y ago
If you build a complex Chain if Thought style Agent and then train/finetune further by reinforcement learning with this architecture then it is not a band-aid anymore, it is an integral part of the model and the weights will optimize to make use of this CoT ability.
- phire 3y ago
  It's been 3.5 years since GPT-3 was released, and just over a year since ChatGPT was released to the public. If it was possible to solve LLM hallucinations with simple Chain-of-Thought style agents, someone would have done that and released a product by now. The fact that nobody has released such a product, is pretty strong evidence that you can't fix hallucinations via Chain-of-Thought or Retrieval-Augmented Generation, or any other band-aid approaches.
  - wokwokwok 3y ago
    I agree: but I just wanted to say that there are specific subdomains where you can mitigate some of these issues. For example, generating json. You can explicitly follow a defined grammar to get what will always be a valid json output. Similarly, structured output such as code can be passed to other tools such as compilers, type checkers and test suites to ensure that at a minimum the output you selected passes some minimum threshold of “isn’t total rubbish”. For unstructured output this a much harder problem, and bluntly, it doesn’t seem like there’s any kind of meaningful solution to it. …but the current generation of LLMs are driven by probabilistic sampling functions. Over the probability curve you’ll always get some rubbish, but if you sample many times for structure and verifiable output you can, to a reasonable degree, mitigate the impact that hallucinations have. Currently that’s computationally expensive, to drive the chance of error down to a useful level, but compute scales. We may seem some quite reasonable outputs from similar architectures wrapped in validation frameworks in the future, I guess. …for, a very specific subset of types of output.

kristopolous 3y ago

I mean it's a demo. Isn't this kinda what they all do