4 ms·

My laymen understanding is that it's trained to learn the mostly likely next word in a sequence of words. So when you feed it a prompt, it's looking at the comb

My laymen understanding is that it's trained to learn the mostly likely next word in a sequence of words. So when you feed it a prompt, it's looking at the combination of words in the prompt and predicting the next word, then repeating that sequence until it feels like there are no good words left to predict. In essence, all of it's responses are generated based on probabilities from prior learning on the fly, not necessarily repeating training data verbatim. So it's likely at some point in this sequence-guessing game, it predicts the wrong words too. Thus hallucinating an answer that sounds plausible in the context of the sentence structure, but isn't factual.

haldujai 4y ago
Yeah that's correct, but sometimes it seems what we are calling hallucination is recall of bad training data, Sebastian Raschka had a short post about this[1]. I was asking it a medical question about the imaging criteria for characterizing a renal cyst and part of it's response included "septations that do not reach the cyst wall" which is a physical impossibility (a septation is a division/partition of a cyst which arises from the wall by definition) and to my medical knowledge/quick search that sequence of words has never been put together by a human anywhere. I get next token prediction, but I find it non-intuitive it's outputting sequences that are very incorrect and have never appeared in the same context window over a variant of the many more accurate sequences it has definitely seen during training, shouldn't there be many permutations of more probable sequences before it hallucinates this? [1] https://sebastianraschka.com/blog/2023/chatgpt-dilemma.html https://sebastianraschka.com/blog/2023/chatgpt-dilemma.html
- roflyear 4y ago
  How about words that are semantically the same, or similar? I also feel like GPT models output "tangential" phrases, though unsure exactly why that would be...
- ShamelessC 4y ago
  > it has definitely seen during training The size of the training data far exceeds the size of the model's weights. As loss is minimized, the model begins to develop various "strategies" across several "genre" of text. We of course have trouble interpreting these and have to make guesses without advanced analysis. > very incorrect What do you mean? It got _very_ close to the correct answer. One useful thing to know is that GPT-3 cannot "go back" and correct early mistakes. This may have happened here. I think in this case it just didn't remember much about it, and may have a sort of general strategy invoked for "include the definition of the word in some sort of counterfactual" process?
  - haldujai 4y ago
    > The size of the training data far exceeds the size of the model's weights. I'm not expecting strict recall but I imagine there is a non-insignificant amount of text in the training data about cysts relative to other entities in the "medical genre" as cysts of all form are one of the most common medical conditions and would be discussed, so I would have expected more probable sequences than the one generated. Doesn't this alsois then also raises the question of what parameter-corpus size works best? At 120b parameters to 106b tokens Galactica was still hallucinating quite a bit. > What do you mean? It got _very_ close to the correct answer. One useful thing to know is that GPT-3 cannot "go back" and correct early mistakes. This may have happened here. I meant that with the negation it became a physical impossibility and therefore very incorrect, but if not negated it would be a correct statement. Your explanation sounds right in this instance, at some point it decided to negate the sentence and it went from being correct (although not relevant to the prompt) to an incorrect statement. This also suggests to me that ChatGPT doesn't have a good enough understanding of negation. This is a challenge for many models in the medical domain as the frequency of negated statements is much higher than in general texts, but very intuitive for any human. I think part of the problem is ChatGPT works so well most of the time I get surprised when it fails in seemingly obvious ways, granted to someone with expertise in the field. It's interesting to probe.
    - ShamelessC 4y ago
      It's certainly interesting. I think it is basically inevitable that we will use language models and other tricks for diagnosis. There's a lot of shame in listing all your symptoms to an actual person. But a language model feels non-judgemental and doesn't even necessarily remember what you said outside it's context window (although chatGPT works around this). Until it "just works", however, probably not a good idea to use in a medical context.
- baq 4y ago
  There’s definitely something going on, in non-English it sometimes emits words which don’t exist in any dictionary and are used by absolutely noone, but they’re based on existing words and convey correct meaning (humans understand and are able to say which existing word should be used instead).
- theGnuMe 4y ago
  You need some randomization to hallucinate which is the Gaussian diffusion stuff in Stable Diffusion.

haldujai 4y ago

Yeah that's correct, but sometimes it seems what we are calling hallucination is recall of bad training data, Sebastian Raschka had a short post about this[1]. I was asking it a medical question about the imaging criteria for characterizing a renal cyst and part of it's response included "septations that do not reach the cyst wall" which is a physical impossibility (a septation is a division/partition of a cyst which arises from the wall by definition) and to my medical knowledge/quick search that sequence of words has never been put together by a human anywhere. I get next token prediction, but I find it non-intuitive it's outputting sequences that are very incorrect and have never appeared in the same context window over a variant of the many more accurate sequences it has definitely seen during training, shouldn't there be many permutations of more probable sequences before it hallucinates this? [1] https://sebastianraschka.com/blog/2023/chatgpt-dilemma.html https://sebastianraschka.com/blog/2023/chatgpt-dilemma.html

roflyear 4y ago
How about words that are semantically the same, or similar? I also feel like GPT models output "tangential" phrases, though unsure exactly why that would be...
ShamelessC 4y ago
> it has definitely seen during training The size of the training data far exceeds the size of the model's weights. As loss is minimized, the model begins to develop various "strategies" across several "genre" of text. We of course have trouble interpreting these and have to make guesses without advanced analysis. > very incorrect What do you mean? It got _very_ close to the correct answer. One useful thing to know is that GPT-3 cannot "go back" and correct early mistakes. This may have happened here. I think in this case it just didn't remember much about it, and may have a sort of general strategy invoked for "include the definition of the word in some sort of counterfactual" process?
- haldujai 4y ago
  > The size of the training data far exceeds the size of the model's weights. I'm not expecting strict recall but I imagine there is a non-insignificant amount of text in the training data about cysts relative to other entities in the "medical genre" as cysts of all form are one of the most common medical conditions and would be discussed, so I would have expected more probable sequences than the one generated. Doesn't this alsois then also raises the question of what parameter-corpus size works best? At 120b parameters to 106b tokens Galactica was still hallucinating quite a bit. > What do you mean? It got _very_ close to the correct answer. One useful thing to know is that GPT-3 cannot "go back" and correct early mistakes. This may have happened here. I meant that with the negation it became a physical impossibility and therefore very incorrect, but if not negated it would be a correct statement. Your explanation sounds right in this instance, at some point it decided to negate the sentence and it went from being correct (although not relevant to the prompt) to an incorrect statement. This also suggests to me that ChatGPT doesn't have a good enough understanding of negation. This is a challenge for many models in the medical domain as the frequency of negated statements is much higher than in general texts, but very intuitive for any human. I think part of the problem is ChatGPT works so well most of the time I get surprised when it fails in seemingly obvious ways, granted to someone with expertise in the field. It's interesting to probe.
  - ShamelessC 4y ago
    It's certainly interesting. I think it is basically inevitable that we will use language models and other tricks for diagnosis. There's a lot of shame in listing all your symptoms to an actual person. But a language model feels non-judgemental and doesn't even necessarily remember what you said outside it's context window (although chatGPT works around this). Until it "just works", however, probably not a good idea to use in a medical context.
baq 4y ago
There’s definitely something going on, in non-English it sometimes emits words which don’t exist in any dictionary and are used by absolutely noone, but they’re based on existing words and convey correct meaning (humans understand and are able to say which existing word should be used instead).
theGnuMe 4y ago
You need some randomization to hallucinate which is the Gaussian diffusion stuff in Stable Diffusion.