4 ms·

- the real and apparent motives of the AI can be completely different. If the AI is evil it would argue the exact same thing in order to deceive us meatbags. So

by gizmo 17y ago

- the real and apparent motives of the AI can be completely different. If the AI is evil it would argue the exact same thing in order to deceive us meatbags. So we can't take the word of the AI at face value. If the AI does break out of its box, it could dominate the world if it wanted to. There's nothing we could do to stop it -- it's smarter than we are. - a mathematical proof is only a proof in a certain context. It would be easy for the AI to get one of the assumptions subtly wrong, to abuse a flaw in our proof verification software, and so on. The correctness proof of a program can easily exceed the complexity of the program itself. Even if the proof were correct we cannot prevent it from doing evil things because evil is too difficult to define. Perhaps it has the "good intention" of liberating the earth from humans to allow for evolution of a more humane species. - yes, but by allowing it to talk to the outside world you've completely freed it. Freeing an infinitely powerful being (compared to us) still seems unwise. - it can give those reasons, but unless we have reason to believe the AI is trustworthy (and an evil AI is likely to fool us into believing it is) we'd be safer with the AI stuck in a box.

lhorie 17y ago
>> If the AI is evil it would argue the exact same thing in order to deceive us meatbags. But if it knows you'll find a proposition fishy, why would it waste time following through that decision branch? I figure a conversation with the AI would avoid the "but-I-am-telling-the-truth" paradox altogether, in favor of a conversation that focuses on easily verifiable data. >> It would be easy for the AI to get one of the assumptions subtly wrong, to abuse a flaw in our proof verification software, and so on. I think the flaw abuse is unlikely given that the AI would not know how the verification system works and it only has one go at trying to crack/fool it (without any one catching on, at that). The misunderstanding of scope due to complexity is an interesting point. Three things come to mind: - paradigms: Thread safety is mind-boggling in procedural paradigms, but a non-issue in functional. - abstraction: the AI should be able to give you readable, modularized, unambiguous, testable code, rather than a monolithic rats' nest. - scope: if an AI can help me find good restaurants, that's a feature; if it can weigh human life, that's a bug :)

by gizmo 17y ago

lhorie 17y ago

>> If the AI is evil it would argue the exact same thing in order to deceive us meatbags. But if it knows you'll find a proposition fishy, why would it waste time following through that decision branch? I figure a conversation with the AI would avoid the "but-I-am-telling-the-truth" paradox altogether, in favor of a conversation that focuses on easily verifiable data. >> It would be easy for the AI to get one of the assumptions subtly wrong, to abuse a flaw in our proof verification software, and so on. I think the flaw abuse is unlikely given that the AI would not know how the verification system works and it only has one go at trying to crack/fool it (without any one catching on, at that). The misunderstanding of scope due to complexity is an interesting point. Three things come to mind: - paradigms: Thread safety is mind-boggling in procedural paradigms, but a non-issue in functional. - abstraction: the AI should be able to give you readable, modularized, unambiguous, testable code, rather than a monolithic rats' nest. - scope: if an AI can help me find good restaurants, that's a feature; if it can weigh human life, that's a bug :)