I was exactly thinking of Rice's Theorem too. I don't know if OpenAI is equivalent in power to a Turing machine (or whether we can even formalize it in that way

I was exactly thinking of Rice's Theorem too. I don't know if OpenAI is equivalent in power to a Turing machine (or whether we can even formalize it in that way since it's non-deterministic), but if it is, it's effectively impossible to prevent it from exhibiting certain properties.