3 ms·

> I wonder how much the source content is the cause of hallucinations rather than anything inherent to LLMs I mean, it's inherent to LLMs to be unable to answe

> I wonder how much the source content is the cause of hallucinations rather than anything inherent to LLMs I mean, it's inherent to LLMs to be unable to answer "I don't know" as a result of not knowing the answer. An LLM never "doesn't know" the answer. But they'll gladly answer "I don't know" if that's statistically the most likely response, right? (Although current public offerings are probably trained against ever saying that.)

aidenn0 3y ago
LLMs work at all because of the high correlation between the statistically most likely response and the most reasonable answer.
- digging 3y ago
  That's an explanation of why their answers can be useful, but doesn't relate to their ability to "not know" an answer
  - ben_w 3y ago
    I suspect this is going to be a disagreement on the meaning of "to know". On the same lines as why people argue if a tree falling in a wood where nobody can hear it makes sound because some people implicitly regard sound is the qualia while others regard it as the vibrations in the air.
    - digging 3y ago
      Not really. An LLM should have no problem replying "I don't know" if that's the most statistically likely answer to a given question, and if it's not trained against such a response. What it fundamentally can't do is introspect and determine it doesn't have enough information to answer the question. It always has an answer. (disclaimer: I don't know jack about the actual mechanics. It's possible something could be constructed which does have that ability and still be considered an "LLM". But the ones we have now can't do that.)
    - weebull 3y ago
      LLMs don't know anything except the most frequent observed response to a context made up of a sequence of tokens. How often do the words "I don't know" get uttered in books, papers, articles, stack overflow, or any other resource of knowledge?
      - ben_w 3y ago
        What does it mean for a human to "know" something? I have some representation in my mind; as someone who doesn't have aphantasia, this representation comes with a mental image. Tower? Tall, linear, and in my case a skyscraper by default. Eiffel Tower? Paying attention to the extra context, the first word transforms the second into the eponymous structure. Model Eiffel Tower? Now the context makes it a tchotchke, probably 10cm tall. Lego model Eiffel Tower? The 1-ish meter tall one on display in the Lego shop. Is my "knowledge" the abstract representation that is in my case connected to a mental image? The attention process can reasonably be considered as developing a vector in a very high dimensional concept space, and the next token comes from what would best suit the current location in that high dimensional space. It's entirely possible that the concept of "ignorance" is linearly separable within that space (much as gender is, see the word2vec trick with "king" - "queen" ~= "man" - "woman"), and the corresponding "ignorance" vector can be associated with the sequence of words "I don't know". I think it would take actual research into the internal vector space to answer that, and while I'd like to do that research, I have some higher priorities right now. Is "knowledge" any belief? Any true belief? Any justified true belief? https://en.wikipedia.org/wiki/Gettier_problem https://en.wikipedia.org/wiki/Gettier_problem I take the position that there is no such thing as knowledge, and instead the best we can have is belief. But then, what is "belief", and can the information within an LLM said to meet whichever definition you give?
  - TeMPOraL 3y ago
    FWIW, we train into kids that "I don't know" is a valid response, and when to utter it. That training is more RLHF-type than source-materal-type, too.
    - digging 3y ago
      I don't follow, what does this mean to the conversation?
      - TeMPOraL 3y ago
        That knowing to say "I don't know" instead of extrapolating is an explicitly learned skill in humans, not something innate or inherent in structure of language, so we shouldn't expect LLMs to pick that ex nihilo either.
  - TrevorJ 3y ago
    Sure it does, if those tokens appear in the training data.
    - digging 3y ago
      No, that's the same misunderstanding previously stated. Answering "I don't know" because it a likely response to a particular string is completely different from being aware that one does not know the answer and saying so. Both motivations lead to the same outcome, but they're unrelated processes. The response "I don't know" can represent either: 1. The most likely answer to a particular question, based on statistical data; or 2. An expression of an agent's internal state. Figuring out that distinction is perhaps one of the most important questions ever raised.

aidenn0 3y ago

LLMs work at all because of the high correlation between the statistically most likely response and the most reasonable answer.

digging 3y ago
That's an explanation of why their answers can be useful, but doesn't relate to their ability to "not know" an answer
- ben_w 3y ago
  I suspect this is going to be a disagreement on the meaning of "to know". On the same lines as why people argue if a tree falling in a wood where nobody can hear it makes sound because some people implicitly regard sound is the qualia while others regard it as the vibrations in the air.
  - digging 3y ago
    Not really. An LLM should have no problem replying "I don't know" if that's the most statistically likely answer to a given question, and if it's not trained against such a response. What it fundamentally can't do is introspect and determine it doesn't have enough information to answer the question. It always has an answer. (disclaimer: I don't know jack about the actual mechanics. It's possible something could be constructed which does have that ability and still be considered an "LLM". But the ones we have now can't do that.)
  - weebull 3y ago
    LLMs don't know anything except the most frequent observed response to a context made up of a sequence of tokens. How often do the words "I don't know" get uttered in books, papers, articles, stack overflow, or any other resource of knowledge?
    - ben_w 3y ago
      What does it mean for a human to "know" something? I have some representation in my mind; as someone who doesn't have aphantasia, this representation comes with a mental image. Tower? Tall, linear, and in my case a skyscraper by default. Eiffel Tower? Paying attention to the extra context, the first word transforms the second into the eponymous structure. Model Eiffel Tower? Now the context makes it a tchotchke, probably 10cm tall. Lego model Eiffel Tower? The 1-ish meter tall one on display in the Lego shop. Is my "knowledge" the abstract representation that is in my case connected to a mental image? The attention process can reasonably be considered as developing a vector in a very high dimensional concept space, and the next token comes from what would best suit the current location in that high dimensional space. It's entirely possible that the concept of "ignorance" is linearly separable within that space (much as gender is, see the word2vec trick with "king" - "queen" ~= "man" - "woman"), and the corresponding "ignorance" vector can be associated with the sequence of words "I don't know". I think it would take actual research into the internal vector space to answer that, and while I'd like to do that research, I have some higher priorities right now. Is "knowledge" any belief? Any true belief? Any justified true belief? https://en.wikipedia.org/wiki/Gettier_problem https://en.wikipedia.org/wiki/Gettier_problem I take the position that there is no such thing as knowledge, and instead the best we can have is belief. But then, what is "belief", and can the information within an LLM said to meet whichever definition you give?
- TeMPOraL 3y ago
  FWIW, we train into kids that "I don't know" is a valid response, and when to utter it. That training is more RLHF-type than source-materal-type, too.
  - digging 3y ago
    I don't follow, what does this mean to the conversation?
    - TeMPOraL 3y ago
      That knowing to say "I don't know" instead of extrapolating is an explicitly learned skill in humans, not something innate or inherent in structure of language, so we shouldn't expect LLMs to pick that ex nihilo either.
- TrevorJ 3y ago
  Sure it does, if those tokens appear in the training data.
  - digging 3y ago
    No, that's the same misunderstanding previously stated. Answering "I don't know" because it a likely response to a particular string is completely different from being aware that one does not know the answer and saying so. Both motivations lead to the same outcome, but they're unrelated processes. The response "I don't know" can represent either: 1. The most likely answer to a particular question, based on statistical data; or 2. An expression of an agent's internal state. Figuring out that distinction is perhaps one of the most important questions ever raised.