Ilya Sutskever says that one of his early insights is that neural networks can do most things that humans can do quickly. I think that applies here as well. So

Ilya Sutskever says that one of his early insights is that neural networks can do most things that humans can do quickly. I think that applies here as well. Some people can do math quickly in their head, but most people cant. They need pen and paper and if you give them pen and paper you get vastly better answers. I would conjecture that there is a prompt where the LLM is better able to explain its process, but it will require more "pen and paper thinking". It is true that humans generally introspect better than LLM's, but a lot of the time you will also see people producing garbage (they post rationalize). Human tendency to produce garbage introspection is usually tied to feelings about themselves, which the LLM doesn't have. Either way I find a lot of similarities between the two :)