4 ms·

I think LLMs need to be taught to say "I don't know"/"I am not sure" or something to that effect. Another approach might be to introduce an adversarial "censor

I think LLMs need to be taught to say "I don't know"/"I am not sure" or something to that effect. Another approach might be to introduce an adversarial "censor" model to guard against hallucination (or inappropriate answers).

ben_w 3y ago
Isn't the fundamental issue that it doesn't have any way to tell if what it thinks it knows is or isn't true? This article sounds like an idea I had independent not too long ago, but with a different goal: LLMs are great at natural language comprehension, but also have a lot of neurons dedicated to factoids. Using neurons that way is really inefficient, can we split the "language" capability from the "knowledge" capability and have the former just look things up in a database? My question was more about reducing the size of the network rather than reducing hallucinations, but it's still a separately updatable knowledge resource. (The answer may actually be "no"; I don't study this professionally, but technical jargon is kinda both factual domain knowledge and also linguistic comprehension, which is why Oracle isn't competing with Starbucks for Java beans).
- RyEgswuCsn 3y ago
  Many classic statistical modelling techniques have ways to produce some measure of confidence for their predictions; perhaps LLMs could incorporate that as well, e.g. assign probabilities(/perplexity?) to each of the tokens they generate. > LLMs are great at natural language comprehension, but also have a lot of neurons dedicated to factoids. Using neurons that way is really inefficient, can we split the "language" capability from the "knowledge" capability and have the former just look things up in a database? I think the beauty of LLMs is exactly that all we need to do is to feed them raw text --- the hope, I guess, had been that the models will be able to develop human-like insights by learning to understand and "speak" languages on its own. Introducing "feature engineering" (e.g. the distinction as you suggested) would defeat that goal.
  - IanCal 3y ago
    > perhaps LLMs could incorporate that as well, e.g. assign probabilities(/perplexity?) to each of the tokens they generate. They do, that's how tokens are selected. Locally run models or non-chat ones from openai can return the probabilities and you can do things like modify or filter them.
    - RyEgswuCsn 3y ago
      I was thinking more of the entropy of the distribution over the distribution sort of thing --- one may say a model is uncertain about its statements if all its potential variants are equally probable and vice versa.
- cubefox 3y ago
  I don't think that's possible in the current autocomplete based paradigm. But the LLM can be made to ignore most of its own knowledge. For example, Bing answers most questions by performing an Internet search, even if it could have answered without one. (Sometimes this makes it actually worse, e.g. when it is asked to solve a puzzle, and it gives answers to similar ones from the web, without trying to solve it itself.)
Silphendio 3y ago
I don't see why this isn't a focus in RLHF. Just heavily discourage confident lies and encourage admission of ignorance. It can't be that much harder than "safety" training. But then we would probably end up with things like "As an AI language model, my knowledge of the world is limited, but if I were to guess, ..."
yifanl 3y ago
That sounds like admitting there is no secret sauce? The superpower has been the ability to synthesize output from very disparate training sources, and any answer for "I don't know" would come from needing to synthesize disparate training sources.

ben_w 3y ago

Isn't the fundamental issue that it doesn't have any way to tell if what it thinks it knows is or isn't true? This article sounds like an idea I had independent not too long ago, but with a different goal: LLMs are great at natural language comprehension, but also have a lot of neurons dedicated to factoids. Using neurons that way is really inefficient, can we split the "language" capability from the "knowledge" capability and have the former just look things up in a database? My question was more about reducing the size of the network rather than reducing hallucinations, but it's still a separately updatable knowledge resource. (The answer may actually be "no"; I don't study this professionally, but technical jargon is kinda both factual domain knowledge and also linguistic comprehension, which is why Oracle isn't competing with Starbucks for Java beans).

RyEgswuCsn 3y ago
Many classic statistical modelling techniques have ways to produce some measure of confidence for their predictions; perhaps LLMs could incorporate that as well, e.g. assign probabilities(/perplexity?) to each of the tokens they generate. > LLMs are great at natural language comprehension, but also have a lot of neurons dedicated to factoids. Using neurons that way is really inefficient, can we split the "language" capability from the "knowledge" capability and have the former just look things up in a database? I think the beauty of LLMs is exactly that all we need to do is to feed them raw text --- the hope, I guess, had been that the models will be able to develop human-like insights by learning to understand and "speak" languages on its own. Introducing "feature engineering" (e.g. the distinction as you suggested) would defeat that goal.
- IanCal 3y ago
  > perhaps LLMs could incorporate that as well, e.g. assign probabilities(/perplexity?) to each of the tokens they generate. They do, that's how tokens are selected. Locally run models or non-chat ones from openai can return the probabilities and you can do things like modify or filter them.
  - RyEgswuCsn 3y ago
    I was thinking more of the entropy of the distribution over the distribution sort of thing --- one may say a model is uncertain about its statements if all its potential variants are equally probable and vice versa.
cubefox 3y ago
I don't think that's possible in the current autocomplete based paradigm. But the LLM can be made to ignore most of its own knowledge. For example, Bing answers most questions by performing an Internet search, even if it could have answered without one. (Sometimes this makes it actually worse, e.g. when it is asked to solve a puzzle, and it gives answers to similar ones from the web, without trying to solve it itself.)

Silphendio 3y ago

I don't see why this isn't a focus in RLHF. Just heavily discourage confident lies and encourage admission of ignorance. It can't be that much harder than "safety" training. But then we would probably end up with things like "As an AI language model, my knowledge of the world is limited, but if I were to guess, ..."

yifanl 3y ago

That sounds like admitting there is no secret sauce? The superpower has been the ability to synthesize output from very disparate training sources, and any answer for "I don't know" would come from needing to synthesize disparate training sources.