I don't have a good answer, but suspect it will involve a metacognition that can delineate internal and external knowledge. To further handwave this thought bu

I don't have a good answer, but suspect it will involve a metacognition that can delineate internal and external knowledge. To further handwave this thought bubble, you can attend to certain thought processes, and intuit the bounds to those processes. An example is if you ask a gifted young chess player, they would say "it felt right", or when asked how Ramanujan figured out a solution to math problems he would say it was divine revelation. In these cases, they know "they did it" but cannot explain how. As the child gets older and learns formal theory, they would give a mixture of logic and inspiration. Both parts can be attended to; the logic will be apparent and the intuition will be revealed, and have different metacognitive properties. If I were to take a first crack at this, I'd test GPT-AC for its awareness of bounds of its own cognition, not in the sense of "are you able to produce this answer" (because anything any human can come up with, it can answer), but to describe its internal process. Perhaps the language at its intellectual level will be far beyond humans but the language in its metacognitive level will remain primitive. I don't know though. If our only interface is human language, this is probably a lost cause. But I doubt I will feel guilt in pulling the power cord and maybe that makes me a callous committer of cybercide.