Yeah this is a well known concept in formal languages. But the human programmed guard rails act this way since the more powerful human LLM can figure it out.

Yeah this is a well known concept in formal languages. But the human programmed guard rails act this way since the more powerful human LLM can figure it out. So for now we will still need humans! I don’t think anyone has put together the halting problem for LLMs directly yet though. You could imagine a halt token but any simulated LLM should be less powerful. Interesting thought experiment. Can chatgpt create an algorithm to solve the digits of pi and execute it? Might try this. Google has a paper about DNN architectures and the Chomsky hierarchy for generalizing to distribution shifts. This is interesting in that specific architectures should limit what a transformer LLM can do. I imagine this is an active research area.