But that's arithmetic, not logical reasoning. That's not an LLM's purpose. Also I'm a human and would struggle quite a bit to multiply 1405 and 1421 in my head

But that's arithmetic, not logical reasoning. That's not an LLM's purpose. Also I'm a human and would struggle quite a bit to multiply 1405 and 1421 in my head, and nobody expects me to. I think when we test an AI we should use tests in which humans weren't already beaten by machines half a century ago.