6 ms·

This is what happens when your objective function is to generate plausible or well-aligned next-word predictions - the model will converge on agreeable nonsense

This is what happens when your objective function is to generate plausible or well-aligned next-word predictions - the model will converge on agreeable nonsense that sounds credible but is increasingly divorced from the truth. RLHF is a lousy patch to fix this problem. The real solution to grounding LLMs is by training it on hard-to-solve but easy-to-verify (i.e., NP-complete / hard-on-average problems), then formally verifying the generated outputs. While this may be difficult for prose, by including a large amount of training data containing formal reasoning tasks and programmatic text, this puts a strong bias towards correctness.

gfodor 3y ago
Not really, newer LLMs do a better job than older LLMs at not making things up, and there is a lot of research going into this issue, since it’s the most obvious problem with LLMs right now.
- bmc7505 3y ago
  The current generation of LLMs will never generalize on problems whose expected complexity is significantly harder than the training set. One way to think of why this is the case is because transformers are learning shortcuts to multistep reasoning tasks [1, 2], but do not learn to reason systematically and unsurprisingly fail to generalize when presented with more challenging tasks. The proper way to evaluate systematic reasoning is by training on a curriculum of problems of increasing average-case complexity. Think random k-SAT in the critical region, bisimulation games, or interpolation in certain logical theories. Although there is a growing evidence that Transformers and GNNs have foundational limits to expressivity [3, 4, 5], if the scaling crowd could show strong generalization results on these sorts of problems they would have the last laugh... [1]: https://arxiv.org/pdf/2210.10749.pdf https://arxiv.org/pdf/2210.10749.pdf [2]: https://arxiv.org/pdf/2305.18654.pdf https://arxiv.org/pdf/2305.18654.pdf [3]: https://arxiv.org/pdf/2106.16213.pdf https://arxiv.org/pdf/2106.16213.pdf [4]: https://arxiv.org/pdf/2301.10743.pdf https://arxiv.org/pdf/2301.10743.pdf [5]: https://arxiv.org/pdf/2303.04613.pdf https://arxiv.org/pdf/2303.04613.pdf
  - HarHarVeryFunny 3y ago
    > The current generation of LLMs will never generalize on problems whose expected complexity is significantly harder than the training set. I think it could be done, using same method that has recently been successfully used to distill reasoning power of a large model into a smaller one... The trick would be to self-generate a fine tuning dataset by generating "think step by step" responses to challenging problems. By training on "step by step" output you teach the model how to reason, rather than just what is the desired output (which without step by step might be outside of it's capabilities).
  - ticviking 3y ago
    It seems like you will have to actually instruct the model like a human pupil with explicit feedback on what is correct and why.
- roughly 3y ago
  An LLM is Always making things up - that’s what it Does. It is a corpus of word (token) probabilities and a function to generate probabilistic paths through them from a starting point. It is always hallucinating, it just often hallucinating things that also happen to exist.
  - gfodor 3y ago
    You've constructed a very elegant frame here that says basically nothing at all.
naveen99 3y ago
You mean like brute force password search or prime factorization ? That won’t help. There is no free lunch in compression, search, or learning from data.
kunai 3y ago
Isn't this exactly one of the problems that GPT4 has been aiming to solve? It'll be interesting to see if this is the tyranny of AI -- whether degradation of NLMs is an increasingly insurmountable barrier to good performance or whether it's something that can be massaged out with better verification.

gfodor 3y ago

Not really, newer LLMs do a better job than older LLMs at not making things up, and there is a lot of research going into this issue, since it’s the most obvious problem with LLMs right now.

bmc7505 3y ago
The current generation of LLMs will never generalize on problems whose expected complexity is significantly harder than the training set. One way to think of why this is the case is because transformers are learning shortcuts to multistep reasoning tasks [1, 2], but do not learn to reason systematically and unsurprisingly fail to generalize when presented with more challenging tasks. The proper way to evaluate systematic reasoning is by training on a curriculum of problems of increasing average-case complexity. Think random k-SAT in the critical region, bisimulation games, or interpolation in certain logical theories. Although there is a growing evidence that Transformers and GNNs have foundational limits to expressivity [3, 4, 5], if the scaling crowd could show strong generalization results on these sorts of problems they would have the last laugh... [1]: https://arxiv.org/pdf/2210.10749.pdf https://arxiv.org/pdf/2210.10749.pdf [2]: https://arxiv.org/pdf/2305.18654.pdf https://arxiv.org/pdf/2305.18654.pdf [3]: https://arxiv.org/pdf/2106.16213.pdf https://arxiv.org/pdf/2106.16213.pdf [4]: https://arxiv.org/pdf/2301.10743.pdf https://arxiv.org/pdf/2301.10743.pdf [5]: https://arxiv.org/pdf/2303.04613.pdf https://arxiv.org/pdf/2303.04613.pdf
- HarHarVeryFunny 3y ago
  > The current generation of LLMs will never generalize on problems whose expected complexity is significantly harder than the training set. I think it could be done, using same method that has recently been successfully used to distill reasoning power of a large model into a smaller one... The trick would be to self-generate a fine tuning dataset by generating "think step by step" responses to challenging problems. By training on "step by step" output you teach the model how to reason, rather than just what is the desired output (which without step by step might be outside of it's capabilities).
- ticviking 3y ago
  It seems like you will have to actually instruct the model like a human pupil with explicit feedback on what is correct and why.
roughly 3y ago
An LLM is Always making things up - that’s what it Does. It is a corpus of word (token) probabilities and a function to generate probabilistic paths through them from a starting point. It is always hallucinating, it just often hallucinating things that also happen to exist.
- gfodor 3y ago
  You've constructed a very elegant frame here that says basically nothing at all.

naveen99 3y ago

You mean like brute force password search or prime factorization ? That won’t help. There is no free lunch in compression, search, or learning from data.

kunai 3y ago

Isn't this exactly one of the problems that GPT4 has been aiming to solve? It'll be interesting to see if this is the tyranny of AI -- whether degradation of NLMs is an increasingly insurmountable barrier to good performance or whether it's something that can be massaged out with better verification.