>> If the AI is evil it would argue the exact same thing in order to deceive us meatbags. But if it knows you'll find a proposition fishy, why would it waste t

>> If the AI is evil it would argue the exact same thing in order to deceive us meatbags. But if it knows you'll find a proposition fishy, why would it waste time following through that decision branch? I figure a conversation with the AI would avoid the "but-I-am-telling-the-truth" paradox altogether, in favor of a conversation that focuses on easily verifiable data. >> It would be easy for the AI to get one of the assumptions subtly wrong, to abuse a flaw in our proof verification software, and so on. I think the flaw abuse is unlikely given that the AI would not know how the verification system works and it only has one go at trying to crack/fool it (without any one catching on, at that). The misunderstanding of scope due to complexity is an interesting point. Three things come to mind: - paradigms: Thread safety is mind-boggling in procedural paradigms, but a non-issue in functional. - abstraction: the AI should be able to give you readable, modularized, unambiguous, testable code, rather than a monolithic rats' nest. - scope: if an AI can help me find good restaurants, that's a feature; if it can weigh human life, that's a bug :)