I agree with you a LLM is perfectly capable of explaining its actions. However it cannot do so after the fact. If there's a reasoning trace it could extract a

I agree with you a LLM is perfectly capable of explaining its actions. However it cannot do so after the fact. If there's a reasoning trace it could extract a justification from it. But if there isn't, or if the reasoning trace makes no sense, then the LLM will just lie and make up reasons that sound about right.