4 ms·

At a high level, what causes hallucinations is an easier question than how to solve them. LLMs are pretrained to maximize the probability of predicting the n+1

At a high level, what causes hallucinations is an easier question than how to solve them. LLMs are pretrained to maximize the probability of predicting the n+1 token given n tokens. To do this reliably, the model learns statistical patterns in the source data and transformer models are very good at doing that when large enough and given enough data. It is therefore suspect to any statistical biases in the training data because despite many advances in guiding LLMs, e.g. RLHF, LLMs are not sentient and most approaches to get around that such as the current reasoning models are hacks over a fundamental problem with the approach. It also doesn't help that when sampling the tokens, the default temperature of most LLM UIs is 1.0, with the argument that it is better for creativity. If you have access to the API and want a specific answer more reliably, I recommend setting temperature = 0.0, in which case the model will always select the token with the highest probability and tends to be more correct.

rzz3 2y ago
Thanks for the explanation, it makes a lot of sense. One thing I keep wondering though is why we can’t teach them logic at some sort of base layer. Not to anthropomorphize, but couldn’t the actual next-token prediction itself embed some sort of consultation against another (non-LLM) model that can evaluate a probability of logicality of the overall output that would be produced be selecting next-token-A vs next-token-B? I’m sure it’s obviously way more complicated than this, but still.
- actsasbuffoon 2y ago
  I know this is reductive, but at their core, LLMs really are just very fancy autocomplete. The process they go through bears no resemblance to reasoning. Grab the best model you’ve got access to (o4, Gemini Pro 2.5, Claude Sonnet 3.7, etc) and try playing chess with it. The results are astonishingly bad. It’s not just that LLMs make poorly thought out moves. They regularly make completely illegal moves (making a knight move like a pawn, for example), hallucinate new pieces into existence, change pieces from one color to another, etc. More words have probably been written about Chess than any other game. You can literally buy a book that’s just about chess openings. There has to be a staggering amount of text about chess in the training data set. And yet, even the best models are far worse at chess than I was in 3rd grade (and I’m not great). They do pretty well for the first few moves, often playing well known chess openings. But once they get to the part where you need to reason, they fall apart in the most outrageous ways. I suspect this reveals something about how they deal with other logic-based tasks. I’ve wondered for a while now if they basically code through using a vast amount of training data from Stack Overflow, which is why they’re so good at helping with common error codes, and so useless when doing something novel. These new chess experiments have been eye-opening about how incapable the best models are of basic reasoning. Unless something about that changes drastically, I think LLMs have pretty much plateaued. There will undoubtedly be some advancements, but LLMs are never going to reach AGI without reasoning, nor will they be able to do most jobs.
- Jensson 2y ago
  > One thing I keep wondering though is why we can’t teach them logic at some sort of base layer That is what they did with these "chain of thought" models. Maybe they didn't do it in the optimal way, but they did train them on their ability to answer certain questions. So the low hanging fruits from this style of training has already been plucked.

rzz3 2y ago

Thanks for the explanation, it makes a lot of sense. One thing I keep wondering though is why we can’t teach them logic at some sort of base layer. Not to anthropomorphize, but couldn’t the actual next-token prediction itself embed some sort of consultation against another (non-LLM) model that can evaluate a probability of logicality of the overall output that would be produced be selecting next-token-A vs next-token-B? I’m sure it’s obviously way more complicated than this, but still.

actsasbuffoon 2y ago
I know this is reductive, but at their core, LLMs really are just very fancy autocomplete. The process they go through bears no resemblance to reasoning. Grab the best model you’ve got access to (o4, Gemini Pro 2.5, Claude Sonnet 3.7, etc) and try playing chess with it. The results are astonishingly bad. It’s not just that LLMs make poorly thought out moves. They regularly make completely illegal moves (making a knight move like a pawn, for example), hallucinate new pieces into existence, change pieces from one color to another, etc. More words have probably been written about Chess than any other game. You can literally buy a book that’s just about chess openings. There has to be a staggering amount of text about chess in the training data set. And yet, even the best models are far worse at chess than I was in 3rd grade (and I’m not great). They do pretty well for the first few moves, often playing well known chess openings. But once they get to the part where you need to reason, they fall apart in the most outrageous ways. I suspect this reveals something about how they deal with other logic-based tasks. I’ve wondered for a while now if they basically code through using a vast amount of training data from Stack Overflow, which is why they’re so good at helping with common error codes, and so useless when doing something novel. These new chess experiments have been eye-opening about how incapable the best models are of basic reasoning. Unless something about that changes drastically, I think LLMs have pretty much plateaued. There will undoubtedly be some advancements, but LLMs are never going to reach AGI without reasoning, nor will they be able to do most jobs.
Jensson 2y ago
> One thing I keep wondering though is why we can’t teach them logic at some sort of base layer That is what they did with these "chain of thought" models. Maybe they didn't do it in the optimal way, but they did train them on their ability to answer certain questions. So the low hanging fruits from this style of training has already been plucked.