4 ms·
Yes, because examining the old source code allows you to predict its behaviour, including the rewriting of source code. If line 42 says "never rewrite lines 42
by finnw 15y ago
Yes, because examining the old source code allows you to predict its behaviour, including the rewriting of source code. If line 42 says "never rewrite lines 42 or 43" and line 43 says "never kill humans" you would be more likely to let it out of the box than if line 42 said "rewrite whatever you want" and line 43 said "do whatever is necessary to achieve world domination."
- TeHCrAzY 15y agoUntil it adds Line 165: "ignore line 42".
- vannevar 15y agoYes, because examining the old source code allows you to predict its behaviour, including the rewriting of source code. This is the halting problem (http://en.wikipedia.org/wiki/Halting_problem http://en.wikipedia.org/wiki/Halting_problem), and there is no solution.
- esrauch 15y agoYou are incorrect, the halting problem only proves that you cannot solve it in the general case. A very significant subset of programs can be statically determined; it's easy to prove that "main(){}" halts and that "main(){while(true);}" doesn't. It should be trivially obvious that you could group all programs into "Halts" or "Unknown" with no false positives simply by executing the program for X steps and observing the result. If this was actually a concern of the programmers, they could design the program carefully to ensure it falls into the Halts category.
- vannevar 15y agoA very significant subset of programs can be statically determined... Technically this may be correct, but I feel confident in asserting that a transhuman AI would not fall into that subset. You would have to run a second AI with the exact same inputs in order to make your 'prediction', leaving you in the same predicament with the second AI.