3 ms·
The author says: > Once understood in this way, the question to ask is not, "Why do GPTs hallucinate?", but rather, "Why do they get anything right at all?" T
by andrewla 2y ago
The author says:
> Once understood in this way, the question to ask is not, "Why do GPTs hallucinate?", but rather, "Why do they get anything right at all?"
This is the right question. The answers here are entirely unsatisfactory, both from this paper and from the general field of research. We have almost no idea how these things work -- we're at the stage where we learn more from the "golden-gate-bridge" crippled network than we do from understanding how they are trained and how they are architected.
LLMs are clearly not conscious or sentient, but they show emergent behavior that we are not capable of explaining yet. Ten years ago the statement "what distinguishes Man from Animal is that Man has Language" would seem totally reasonable, but now we have a second example of a system that uses language, and it is dumbfounding.
The hype around LLMs is just hype -- LLMs are a solution in search of a problem -- but the emergent features of these models is a tantalizing glimpse of what it means to "think" in an evolved system.
- tgbugs 2y agoI think the answer is actually quite clear and rather boring. In order to get something "right" there has to be some external standard of knowledge and correctness. That definition of correctness can only be provided by the observer (user). Alignment between the user's correctness criteria and generated text happens entirely by accident. This can be demonstrated by observing a correlation between coverage of a domain in the training data and the rate at which incorrect results are produced (as discussed in other comments). That is, they get things "right" because there was sufficient training data that contained information that matched the user's definition for correctness. In fact, exceptionally boring.
- andrewla 2y agoThis is a very post hoc explanation. What does "coverage in the training data" mean? Take a simple task of something like "How many a's are there in the word bookkeeper" -- what is your theory for why it can answer this question correctly or even give something approaching a coherent answer? It never even sees the letters that are in the token "bookkeeper", and this is definitely not something that appears explicitly in the training data. I challenge you to give a "clear and boring" explanation for this -- this is incredibly subtle behavior that emerges from a complex architecture and complex training process, and is in its own right as fascinating and mysterious as the ability of humans to do this task and the inability of cats to do it.
- tgbugs 2y agoAre you surprised that jpg compression algorithms can reproduce input data that bears striking resemblance to the uncompressed input image across a variety of compression levels?
- HarHarVeryFunny 2y agoTry asking Claude Sonnet 3.5 (one of today's best models) "how many p's in Lypophrenia - just a number please" I tried it a second ago, and it said "1". To get these correct requires splitting tokens into letters and counting. I'd not be surprised if most models are either trained on token splitting or have learnt to do it. "Counting" number of occurrences of letters in an arbitrary separated sequence is the harder part, and where I'd guess it might be failing.
- andrewla 2y agoYes! They are commonly wrong in this, and that's fascinating too. Because they are not solving the problem by looking at the letter in the word because they are not architecturally capable of enumerating the letters in the word. The fact that they can do it at all could be the stuff of an entire phd thesis, and could tell us more about the nature of LLM hallucination than a bunch of rambling about "how much coverage in the training set" when our determination of the coverage is based on human semantic similarity.
- HarHarVeryFunny 2y agoAbility to split words into letters isn't architecturally limited - it's just a matter of training data, and made easier by the fact that the input is tokens representing short letter sequences rather than words of which there are more. It's quite possible that more recent training data deliberately includes word/token -> letter sequence samples, but even if not I'd expect there is going to be enough spelling examples naturally occurring in the training data for the model to learn the token (not word) -> letter sequence rules (which will be consistent/reinforced across all spelling samples), which it can then apply to arbitrary words.
- lottin 2y ago> now we have a second example of a system that uses language, and it is dumbfounding An LLM 'uses' language, in the same sense that a calculator 'uses' arithmetic. It's a figure of speech.
- andrewla 2y agoThat's not the sense of the metaphor that I'm applying when I say "uses language". That's closer to saying that "Alexa uses language", where "uses" here is analogous to what a calculator does. To avoid using anthropomorphic terms, an LLM can take natural language from a human, integrate information from those expressions together with information held in its (opaque) store, and return natural language that a human can understand that reflects that information. I am not aware of any other systems besides humans that can accomplish that task. Some animals can be trained to do some parts of this, but really until now humans are the only ones that could do the full loop.
- lottin 2y agoOkay... but computers perform many other tasks that only humans can perform, for example, do square roots or play chess. The point being LLMs are just another program running on an integrated circuit. In short, I fail to see how LLMs blur the line between man and machine but pocket calculators do not.