3 ms·
LLMs reason to the extent they are allowed to. You could say that they are overfitting when it comes to reasoning. They weren't trained to reason to begin with,
by imtringued 2y ago
LLMs reason to the extent they are allowed to. You could say that they are overfitting when it comes to reasoning. They weren't trained to reason to begin with, so the bigger surprise is that they can do it within limits.
- andrewla 2y agoI agree about the surprise -- LLMs can do many surprising things that frankly are astonishing given their architecture. The fact that they can produce output that is difficult to distinguish from the output of an actor that we "know" can reason is pretty astonishing too. I don't agree with the idea of "the extent they are allowed to"; that's giving the creators of LLMs waaay more agency than they have in reality. These things have already escaped the bounds of what we thought they could do, I don't think we have a realistic way of constraining their behavior in a deterministic way (other than maybe cutting down the context length).