9 ms·

Not really, newer LLMs do a better job than older LLMs at not making things up, and there is a lot of research going into this issue, since it’s the most obviou

by gfodor 3y ago

Not really, newer LLMs do a better job than older LLMs at not making things up, and there is a lot of research going into this issue, since it’s the most obvious problem with LLMs right now.

bmc7505 3y ago
The current generation of LLMs will never generalize on problems whose expected complexity is significantly harder than the training set. One way to think of why this is the case is because transformers are learning shortcuts to multistep reasoning tasks [1, 2], but do not learn to reason systematically and unsurprisingly fail to generalize when presented with more challenging tasks. The proper way to evaluate systematic reasoning is by training on a curriculum of problems of increasing average-case complexity. Think random k-SAT in the critical region, bisimulation games, or interpolation in certain logical theories. Although there is a growing evidence that Transformers and GNNs have foundational limits to expressivity [3, 4, 5], if the scaling crowd could show strong generalization results on these sorts of problems they would have the last laugh... [1]: https://arxiv.org/pdf/2210.10749.pdf https://arxiv.org/pdf/2210.10749.pdf [2]: https://arxiv.org/pdf/2305.18654.pdf https://arxiv.org/pdf/2305.18654.pdf [3]: https://arxiv.org/pdf/2106.16213.pdf https://arxiv.org/pdf/2106.16213.pdf [4]: https://arxiv.org/pdf/2301.10743.pdf https://arxiv.org/pdf/2301.10743.pdf [5]: https://arxiv.org/pdf/2303.04613.pdf https://arxiv.org/pdf/2303.04613.pdf
- HarHarVeryFunny 3y ago
  > The current generation of LLMs will never generalize on problems whose expected complexity is significantly harder than the training set. I think it could be done, using same method that has recently been successfully used to distill reasoning power of a large model into a smaller one... The trick would be to self-generate a fine tuning dataset by generating "think step by step" responses to challenging problems. By training on "step by step" output you teach the model how to reason, rather than just what is the desired output (which without step by step might be outside of it's capabilities).
- ticviking 3y ago
  It seems like you will have to actually instruct the model like a human pupil with explicit feedback on what is correct and why.
roughly 3y ago
An LLM is Always making things up - that’s what it Does. It is a corpus of word (token) probabilities and a function to generate probabilistic paths through them from a starting point. It is always hallucinating, it just often hallucinating things that also happen to exist.
- gfodor 3y ago
  You've constructed a very elegant frame here that says basically nothing at all.

by gfodor 3y ago

Not really, newer LLMs do a better job than older LLMs at not making things up, and there is a lot of research going into this issue, since it’s the most obvious problem with LLMs right now.

bmc7505 3y ago

The current generation of LLMs will never generalize on problems whose expected complexity is significantly harder than the training set. One way to think of why this is the case is because transformers are learning shortcuts to multistep reasoning tasks [1, 2], but do not learn to reason systematically and unsurprisingly fail to generalize when presented with more challenging tasks. The proper way to evaluate systematic reasoning is by training on a curriculum of problems of increasing average-case complexity. Think random k-SAT in the critical region, bisimulation games, or interpolation in certain logical theories. Although there is a growing evidence that Transformers and GNNs have foundational limits to expressivity [3, 4, 5], if the scaling crowd could show strong generalization results on these sorts of problems they would have the last laugh... [1]: https://arxiv.org/pdf/2210.10749.pdf https://arxiv.org/pdf/2210.10749.pdf [2]: https://arxiv.org/pdf/2305.18654.pdf https://arxiv.org/pdf/2305.18654.pdf [3]: https://arxiv.org/pdf/2106.16213.pdf https://arxiv.org/pdf/2106.16213.pdf [4]: https://arxiv.org/pdf/2301.10743.pdf https://arxiv.org/pdf/2301.10743.pdf [5]: https://arxiv.org/pdf/2303.04613.pdf https://arxiv.org/pdf/2303.04613.pdf

HarHarVeryFunny 3y ago
> The current generation of LLMs will never generalize on problems whose expected complexity is significantly harder than the training set. I think it could be done, using same method that has recently been successfully used to distill reasoning power of a large model into a smaller one... The trick would be to self-generate a fine tuning dataset by generating "think step by step" responses to challenging problems. By training on "step by step" output you teach the model how to reason, rather than just what is the desired output (which without step by step might be outside of it's capabilities).
ticviking 3y ago
It seems like you will have to actually instruct the model like a human pupil with explicit feedback on what is correct and why.

roughly 3y ago

An LLM is Always making things up - that’s what it Does. It is a corpus of word (token) probabilities and a function to generate probabilistic paths through them from a starting point. It is always hallucinating, it just often hallucinating things that also happen to exist.

gfodor 3y ago
You've constructed a very elegant frame here that says basically nothing at all.