I certainly agree on "very limited". (Transformers are linear-time complexity per token, as far as I understand, so from computational theory principles, there

I certainly agree on "very limited". (Transformers are linear-time complexity per token, as far as I understand, so from computational theory principles, there are things these things trivially will not be able to answer, as they just "can't take longer thinking about it" as you point out.) But "very limited" isn't "not at all", and I think the distinction matters, since the former can often be scaled more easily than the latter. And at the meta-level, one possible output token can always be "let me ramble on this a bit more before I provide my final answer". In any case, I think "they can't look back at their output", as presented in the article, is just the wrong way to look at it; even "they can (currently) not take longer time to look at their output" is also wrong, since they can just produce more output, which then in turn lets them "think more".