2 ms·
> And that's despite LLMs being wildly bad at, specifically, logic. I'd say "wildly bad" is wildly exaggerated given that GPT-4 exists, even if you define "log
by Sharlin 3y ago
> And that's despite LLMs being wildly bad at, specifically, logic.
I'd say "wildly bad" is wildly exaggerated given that GPT-4 exists, even if you define "logic" as "logical puzzles encoded in natural language" and not "the thing that allows an LLM to reason highly efficiently about what token comes next".
- ben_w 3y agoThe numbers in Table 1 of this paper are mostly worse than I remembered, but I would also note that humans would fail the "can they reason?" standard I quoted, given the grades the species gets on several of the tests: https://arxiv.org/pdf/2304.03439.pdf https://arxiv.org/pdf/2304.03439.pdf Table 2 likewise shows significant weaknesses in GPT-4, worse than I remember, except on MED which was better than I remember it being capable of. > even if you define "logic" as "logical puzzles encoded in natural language" and not "the thing that allows an LLM to reason highly efficiently about what token comes next". I think this is a reasonable thing to say of LLMs specifically[0], given I think most people would agree with the sentence "humans are bad at quantum mechanics" even though that's what our cell chemistry is implemented on. [0] Though not AI in general! One of my recently-developed bug-bears is the conflation of the limits of LLMs with AI even in technical circles. Human brains aren't just a language cortex, there's no reason to look at an isolated language model and assume an AI can't be wired into other things, yet this assumption is baked into quite a lot of talking points even on HN.