4 ms·

> AI hallucinations resulting in widespread persistent errors that cause an epistemological crisis. Are AI hallucinations just made up references? Or does that

> AI hallucinations resulting in widespread persistent errors that cause an epistemological crisis. Are AI hallucinations just made up references? Or does that term have a more general or specific meaning?

knodi123 4y ago
It means being confidently wrong. Like I asked facebook's llama the other day which dog was best, and it said golden retrievers because they are eager to please and don't shed (which makes them suitable for people with dog allergies). It was quite confident-sounding, but, uh..... no.
- cjohnson318 4y ago
  Got it. Yeah, you can generally tell if someone lives in the same house as a golden retriever.
HarHarVeryFunny 4y ago
It's a rather meaningless term that people are using to describe situations when one of these LLM/AIs generates something that isn't true in a context where they are expecting something true. What's going on here is that at the simplest level of explanation these models are statistical word generators: you feed some input (words) in and ask the model to generate a word that it thinks is a statistically likely continuation/follow-on to that input. If you want more than one word of output then take the previous output word, add that to the input, and ask the model to generate the next word, etc, etc... What's happening with "hallucinations" is a bit like when we accuse a human of "starting to speak before engaging their brain"... the model generates words one at a time as asked (i.e. "starts speaking") but isn't explicitly planning ahead (hasn't engaged it's brain) to any great extent. If it starts down a path of "the source for that is ..." (not because it actually has a source, but rather because it thought it statistically likely that it would be quoting one), then to complete that sentence it's going to have to quote a source - as usual what ever it thinks is most statistically likely in the context (maybe something highly relevant from it's training set, maybe something of poor relevance, or maybe something whose only relevance is that it looks source-y).
- sanxiyn 4y ago
  Let's not be ridiculous. Of course LLMs are planning ahead, not generating words one at a time. You can't even generate correct indefinite articles (a/an) without planning one word (or token) ahead. Surely you are not suggesting LLMs pick a/an according to statistical distribution and then choose vowel starting/non-starting words, not vice versa? This is testable: let it complete "I climbed up the apple/pear tree and picked". It always picks "an" for apple and "a" for pear, and never reaches a dead end by picking the wrong article so that it either fails syntax (wrong article) or semantics (pear growing on apple tree). We actually have fairly good idea how LLMs pick a/an. Have a look at https://www.lesswrong.com/posts/cgqh99SHsCv3jJYDS/we-found-an-neuron-in-gpt-2 https://www.lesswrong.com/posts/cgqh99SHsCv3jJYDS/we-found-a....
  - HarHarVeryFunny 4y ago
    I'm not sure that this apple/pear example proves anything. In the context of "apple tree and picked" in the input, it's enough simply that the word "an" received positive feedback during training as the next word, and having generated "an" then "apple" becomes almost certain as the next word after that. For sure there has to be some internal high level representation of the entire input, and I would expect also some representation of possible continuations at a higher level than just alternate word choices, but I wouldn't exactly characterize this as a plan since it's so statistical in nature, and the model has to continually adapt to what is actually generated (sampled). At a low temperature setting "an" might always be picked, making "apple" more likely to follow, but at a higher temperature maybe another word such as "the" might be chosen, perhaps with a continuation of "biggest apple i could find", etc. In a 350B token training set I'm sure there are a number of other alternatives too. So, does the model plan ahead ? Well, sort of, but this "plan" has to represent multiple branching alternate futures, and continually adapt as the output is sampled, so it's a very fluid kind of plan. It's also not clear how far ahead the plan needs to represent .. maybe one word only (like a state machine), even if the reason those alternate one-word continuations are there is because they were seen as part of specific longer continuations. The lack of any internal state preservation from one word generated to the next puts a huge limit on what can be planned, as does the extent of how complex a plan can be generated from the flow of a single input sequence forwards thru the layers of the model. You can see the limitations of this with prompts such as "pick two primes > 200, and tell me their product, but not the numbers themselves"... if you play with it and analyze the mistakes, it makes it clear how much the output really is generated word by word, adapting to what came before, much of the time.

knodi123 4y ago

It means being confidently wrong. Like I asked facebook's llama the other day which dog was best, and it said golden retrievers because they are eager to please and don't shed (which makes them suitable for people with dog allergies). It was quite confident-sounding, but, uh..... no.

cjohnson318 4y ago
Got it. Yeah, you can generally tell if someone lives in the same house as a golden retriever.

HarHarVeryFunny 4y ago

It's a rather meaningless term that people are using to describe situations when one of these LLM/AIs generates something that isn't true in a context where they are expecting something true. What's going on here is that at the simplest level of explanation these models are statistical word generators: you feed some input (words) in and ask the model to generate a word that it thinks is a statistically likely continuation/follow-on to that input. If you want more than one word of output then take the previous output word, add that to the input, and ask the model to generate the next word, etc, etc... What's happening with "hallucinations" is a bit like when we accuse a human of "starting to speak before engaging their brain"... the model generates words one at a time as asked (i.e. "starts speaking") but isn't explicitly planning ahead (hasn't engaged it's brain) to any great extent. If it starts down a path of "the source for that is ..." (not because it actually has a source, but rather because it thought it statistically likely that it would be quoting one), then to complete that sentence it's going to have to quote a source - as usual what ever it thinks is most statistically likely in the context (maybe something highly relevant from it's training set, maybe something of poor relevance, or maybe something whose only relevance is that it looks source-y).

sanxiyn 4y ago
Let's not be ridiculous. Of course LLMs are planning ahead, not generating words one at a time. You can't even generate correct indefinite articles (a/an) without planning one word (or token) ahead. Surely you are not suggesting LLMs pick a/an according to statistical distribution and then choose vowel starting/non-starting words, not vice versa? This is testable: let it complete "I climbed up the apple/pear tree and picked". It always picks "an" for apple and "a" for pear, and never reaches a dead end by picking the wrong article so that it either fails syntax (wrong article) or semantics (pear growing on apple tree). We actually have fairly good idea how LLMs pick a/an. Have a look at https://www.lesswrong.com/posts/cgqh99SHsCv3jJYDS/we-found-an-neuron-in-gpt-2 https://www.lesswrong.com/posts/cgqh99SHsCv3jJYDS/we-found-a....
- HarHarVeryFunny 4y ago
  I'm not sure that this apple/pear example proves anything. In the context of "apple tree and picked" in the input, it's enough simply that the word "an" received positive feedback during training as the next word, and having generated "an" then "apple" becomes almost certain as the next word after that. For sure there has to be some internal high level representation of the entire input, and I would expect also some representation of possible continuations at a higher level than just alternate word choices, but I wouldn't exactly characterize this as a plan since it's so statistical in nature, and the model has to continually adapt to what is actually generated (sampled). At a low temperature setting "an" might always be picked, making "apple" more likely to follow, but at a higher temperature maybe another word such as "the" might be chosen, perhaps with a continuation of "biggest apple i could find", etc. In a 350B token training set I'm sure there are a number of other alternatives too. So, does the model plan ahead ? Well, sort of, but this "plan" has to represent multiple branching alternate futures, and continually adapt as the output is sampled, so it's a very fluid kind of plan. It's also not clear how far ahead the plan needs to represent .. maybe one word only (like a state machine), even if the reason those alternate one-word continuations are there is because they were seen as part of specific longer continuations. The lack of any internal state preservation from one word generated to the next puts a huge limit on what can be planned, as does the extent of how complex a plan can be generated from the flow of a single input sequence forwards thru the layers of the model. You can see the limitations of this with prompts such as "pick two primes > 200, and tell me their product, but not the numbers themselves"... if you play with it and analyze the mistakes, it makes it clear how much the output really is generated word by word, adapting to what came before, much of the time.