4 ms·

I was hoping this would link me to a deeper discussion on hallucination. I'm intrigued that it's hallucinating sequences that appear to have never written befo

I was hoping this would link me to a deeper discussion on hallucination. I'm intrigued that it's hallucinating sequences that appear to have never written before (at least not on Google) and not just recalling some crappy training data. Anecdotally (and expectedly) it happens a lot on ChatGPT with specialized scientific questions (random radiology and medical stuff). I am assuming some of this is due to the training corpus although Galactica suffered from the same thing, and the GPT3 corpora would have included a lot of scientific webpages. Anyone have any resources that investigate why this happens?

cloudking 4y ago
My laymen understanding is that it's trained to learn the mostly likely next word in a sequence of words. So when you feed it a prompt, it's looking at the combination of words in the prompt and predicting the next word, then repeating that sequence until it feels like there are no good words left to predict. In essence, all of it's responses are generated based on probabilities from prior learning on the fly, not necessarily repeating training data verbatim. So it's likely at some point in this sequence-guessing game, it predicts the wrong words too. Thus hallucinating an answer that sounds plausible in the context of the sentence structure, but isn't factual.
- haldujai 4y ago
  Yeah that's correct, but sometimes it seems what we are calling hallucination is recall of bad training data, Sebastian Raschka had a short post about this[1]. I was asking it a medical question about the imaging criteria for characterizing a renal cyst and part of it's response included "septations that do not reach the cyst wall" which is a physical impossibility (a septation is a division/partition of a cyst which arises from the wall by definition) and to my medical knowledge/quick search that sequence of words has never been put together by a human anywhere. I get next token prediction, but I find it non-intuitive it's outputting sequences that are very incorrect and have never appeared in the same context window over a variant of the many more accurate sequences it has definitely seen during training, shouldn't there be many permutations of more probable sequences before it hallucinates this? [1] https://sebastianraschka.com/blog/2023/chatgpt-dilemma.html https://sebastianraschka.com/blog/2023/chatgpt-dilemma.html
  - roflyear 4y ago
    How about words that are semantically the same, or similar? I also feel like GPT models output "tangential" phrases, though unsure exactly why that would be...
  - ShamelessC 4y ago
    > it has definitely seen during training The size of the training data far exceeds the size of the model's weights. As loss is minimized, the model begins to develop various "strategies" across several "genre" of text. We of course have trouble interpreting these and have to make guesses without advanced analysis. > very incorrect What do you mean? It got _very_ close to the correct answer. One useful thing to know is that GPT-3 cannot "go back" and correct early mistakes. This may have happened here. I think in this case it just didn't remember much about it, and may have a sort of general strategy invoked for "include the definition of the word in some sort of counterfactual" process?
    - haldujai 4y ago
      > The size of the training data far exceeds the size of the model's weights. I'm not expecting strict recall but I imagine there is a non-insignificant amount of text in the training data about cysts relative to other entities in the "medical genre" as cysts of all form are one of the most common medical conditions and would be discussed, so I would have expected more probable sequences than the one generated. Doesn't this alsois then also raises the question of what parameter-corpus size works best? At 120b parameters to 106b tokens Galactica was still hallucinating quite a bit. > What do you mean? It got _very_ close to the correct answer. One useful thing to know is that GPT-3 cannot "go back" and correct early mistakes. This may have happened here. I meant that with the negation it became a physical impossibility and therefore very incorrect, but if not negated it would be a correct statement. Your explanation sounds right in this instance, at some point it decided to negate the sentence and it went from being correct (although not relevant to the prompt) to an incorrect statement. This also suggests to me that ChatGPT doesn't have a good enough understanding of negation. This is a challenge for many models in the medical domain as the frequency of negated statements is much higher than in general texts, but very intuitive for any human. I think part of the problem is ChatGPT works so well most of the time I get surprised when it fails in seemingly obvious ways, granted to someone with expertise in the field. It's interesting to probe.
oidar 4y ago
Do you have any idea where they started applying the word "hallucination" to making stuff by an LLM? It seems that the more proper word, would be confabulating. B/c everything that the LLM does is a "hallucination" but a confabulation completely made up and the model thinks it accurate - much like a person who confabulates. A person who hallucinates pretty much knows that they are hallucinating at some point.
- yaakov34 4y ago
  I don't know who applied it to LLMs, but it is/was the standard term used for an image processing model producing a detailed signal not justified by its inputs. For example, "face hallucination" means that the model produces a detailed-looking face when given very noisy data, but of course the face will not actually be the original face. In fact, the original image may have had no face at all. Hallucination can be either desired (as a kind of generative technique) or very harmful - imagine using image enhancement to identify a criminal in a noisy image, and getting a detailed face looking like someone in your training set - but not the right person's. Any image enhancement technique, deep learning-based or not, can result in hallucination - you're producing information which was not in your input, which you're able to do because you have priors. But this can always result in incorrect information.
joshxyz 4y ago
Same problem, on programming questions it provides functions that dont exist.

cloudking 4y ago

My laymen understanding is that it's trained to learn the mostly likely next word in a sequence of words. So when you feed it a prompt, it's looking at the combination of words in the prompt and predicting the next word, then repeating that sequence until it feels like there are no good words left to predict. In essence, all of it's responses are generated based on probabilities from prior learning on the fly, not necessarily repeating training data verbatim. So it's likely at some point in this sequence-guessing game, it predicts the wrong words too. Thus hallucinating an answer that sounds plausible in the context of the sentence structure, but isn't factual.

haldujai 4y ago
Yeah that's correct, but sometimes it seems what we are calling hallucination is recall of bad training data, Sebastian Raschka had a short post about this[1]. I was asking it a medical question about the imaging criteria for characterizing a renal cyst and part of it's response included "septations that do not reach the cyst wall" which is a physical impossibility (a septation is a division/partition of a cyst which arises from the wall by definition) and to my medical knowledge/quick search that sequence of words has never been put together by a human anywhere. I get next token prediction, but I find it non-intuitive it's outputting sequences that are very incorrect and have never appeared in the same context window over a variant of the many more accurate sequences it has definitely seen during training, shouldn't there be many permutations of more probable sequences before it hallucinates this? [1] https://sebastianraschka.com/blog/2023/chatgpt-dilemma.html https://sebastianraschka.com/blog/2023/chatgpt-dilemma.html
- roflyear 4y ago
  How about words that are semantically the same, or similar? I also feel like GPT models output "tangential" phrases, though unsure exactly why that would be...
- ShamelessC 4y ago
  > it has definitely seen during training The size of the training data far exceeds the size of the model's weights. As loss is minimized, the model begins to develop various "strategies" across several "genre" of text. We of course have trouble interpreting these and have to make guesses without advanced analysis. > very incorrect What do you mean? It got _very_ close to the correct answer. One useful thing to know is that GPT-3 cannot "go back" and correct early mistakes. This may have happened here. I think in this case it just didn't remember much about it, and may have a sort of general strategy invoked for "include the definition of the word in some sort of counterfactual" process?
  - haldujai 4y ago
    > The size of the training data far exceeds the size of the model's weights. I'm not expecting strict recall but I imagine there is a non-insignificant amount of text in the training data about cysts relative to other entities in the "medical genre" as cysts of all form are one of the most common medical conditions and would be discussed, so I would have expected more probable sequences than the one generated. Doesn't this alsois then also raises the question of what parameter-corpus size works best? At 120b parameters to 106b tokens Galactica was still hallucinating quite a bit. > What do you mean? It got _very_ close to the correct answer. One useful thing to know is that GPT-3 cannot "go back" and correct early mistakes. This may have happened here. I meant that with the negation it became a physical impossibility and therefore very incorrect, but if not negated it would be a correct statement. Your explanation sounds right in this instance, at some point it decided to negate the sentence and it went from being correct (although not relevant to the prompt) to an incorrect statement. This also suggests to me that ChatGPT doesn't have a good enough understanding of negation. This is a challenge for many models in the medical domain as the frequency of negated statements is much higher than in general texts, but very intuitive for any human. I think part of the problem is ChatGPT works so well most of the time I get surprised when it fails in seemingly obvious ways, granted to someone with expertise in the field. It's interesting to probe.

oidar 4y ago

Do you have any idea where they started applying the word "hallucination" to making stuff by an LLM? It seems that the more proper word, would be confabulating. B/c everything that the LLM does is a "hallucination" but a confabulation completely made up and the model thinks it accurate - much like a person who confabulates. A person who hallucinates pretty much knows that they are hallucinating at some point.

yaakov34 4y ago
I don't know who applied it to LLMs, but it is/was the standard term used for an image processing model producing a detailed signal not justified by its inputs. For example, "face hallucination" means that the model produces a detailed-looking face when given very noisy data, but of course the face will not actually be the original face. In fact, the original image may have had no face at all. Hallucination can be either desired (as a kind of generative technique) or very harmful - imagine using image enhancement to identify a criminal in a noisy image, and getting a detailed face looking like someone in your training set - but not the right person's. Any image enhancement technique, deep learning-based or not, can result in hallucination - you're producing information which was not in your input, which you're able to do because you have priors. But this can always result in incorrect information.

joshxyz 4y ago

Same problem, on programming questions it provides functions that dont exist.