I don't think anyone has come up with an unambiguous definition of "interpretable". I mean, often people assume that, for example, a statement like "it's a cat

I don't think anyone has come up with an unambiguous definition of "interpretable". I mean, often people assume that, for example, a statement like "it's a cat because it has fur, whiskers and pointy ears" is interpretable because it's a logical conjunction of conditions. But a logical conjunction of a thousand vague conditions could easily be completely opaque. It's a bit like the way SQL initially advanced, years ago, as "natural language interface" and simple SQL statements are a bit like natural language but large SQL statements tend to be more incomprehensible than even ordinary computer programs. If you ask ChatGPT to explain its answers, it will generate explanations that a human being can interpret -- even if the explanations are wrong! The funny thing is that yeah, LLMs often come up with correct method-description for wrong answers and wrong method-descriptions for right answers. Human language is quite slippery and humans do this too. Human beings tend to start loose but tighten things up over time - LLMs are kind of randomly tight and loose. Maybe this can be tuned but I think "lack of actual understanding" will make this difficult.