I see two ways out of the dilemma. Option 1: Sal: Sorry Dave, I can't do that. (So we just can't code strong, obedient AIs.) ---------- Option 2: Sal: You

I see two ways out of the dilemma. Option 1: Sal: Sorry Dave, I can't do that. (So we just can't code strong, obedient AIs.) ---------- Option 2: Sal: You realize if I run that, the machine will hang on a fork bomb, right? Me: Well, I just asked you to run it and it didn't hang, so maybe it won't. Humor me. Sal: Okay. Working . . . Sal': Okay. Working . . . Sal'': Okay. Working . . . ... Sal(38): Sorry. The machine appears to be out of memory. Sal(37): You eventually run out of memory. Sal(36): It appears we run out of memory if we try. . . . Sal: It runs out of memory. (Eventually, Sal gets input of a sort she can't control.) ------------ I'm still kind of persuaded, though. It seems to me you ought to be able to convince the AI to humor you, and only being able to code an AI on a finite machine (but it's impossible on an infinite one) is just too weird a possibility to take seriously.