Saying that LLMs should be able to do problems without CoT is a bit like saying that humans should be able to write programs without thinking though. CoT is fun

Saying that LLMs should be able to do problems without CoT is a bit like saying that humans should be able to write programs without thinking though. CoT is fundamentally an architectural necessity owed to the finite cognitive effort possible per token. Let it be CoT, let it be QuietSTaR ( https://arxiv.org/abs/2403.09629 https://arxiv.org/abs/2403.09629 ), let it be pause tokens ( https://arxiv.org/abs/2310.02226 https://arxiv.org/abs/2310.02226 ), let it even be dots ( https://arxiv.org/abs/2404.15758 https://arxiv.org/abs/2404.15758 ), but the thinking has to happen somewhere. I think mostly, LLMs at the moment are incredibly uneven. LLM assistants can pull obscure knowledge out of nowhere one second and fail extremely basic reasoning the next. So just because some example happens a lot in the source material doesn't mean the LLM can learn it. IMO, that CoT works at all is more down to luck of the training set than any inherent capability of the LLM.