I don't think this is a valid argument. Humans could always relatively easy pretend to be a chatbot, just the inverse didn't work until now. In this case, the

I don't think this is a valid argument. Humans could always relatively easy pretend to be a chatbot, just the inverse didn't work until now. In this case, the knowledge limitations of the model are well-known, so if you wanted to fake this, your human operators would be instructed to play along and not reveal any information that the model could not know.