I agree about the surprise -- LLMs can do many surprising things that frankly are astonishing given their architecture. The fact that they can produce output th

I agree about the surprise -- LLMs can do many surprising things that frankly are astonishing given their architecture. The fact that they can produce output that is difficult to distinguish from the output of an actor that we "know" can reason is pretty astonishing too. I don't agree with the idea of "the extent they are allowed to"; that's giving the creators of LLMs waaay more agency than they have in reality. These things have already escaped the bounds of what we thought they could do, I don't think we have a realistic way of constraining their behavior in a deterministic way (other than maybe cutting down the context length).