I like this point of view, and I'll even go a step further... If we're going to use LLMs as the basis for anything resembling general intelligence, it won't be

I like this point of view, and I'll even go a step further... If we're going to use LLMs as the basis for anything resembling general intelligence, it won't be through one-shot invocation of the model. It'll be through some kind of chain/tree/graph of thought where the model invokes itself recursively. In this scenario, we have an exact transcript of the model's thought process, exactly as it occurred, written for us in english. The model can't even have a private "thought", everything needs to be in the visible context. You can't get more interpretable than that.