If any LLM can solve a novel exercise that requires some form of reasoning, wouldn't that be evidence that LLMs are capable of at least some level of reasoning?

If any LLM can solve a novel exercise that requires some form of reasoning, wouldn't that be evidence that LLMs are capable of at least some level of reasoning? Not necessarily AGI or average human level reasoning, but just any form.