5 ms·
You're going to have to implement a value system into the AGI if it is a fully general AGI. Otherwise, it will just use its exceptional intelligence to defeat i
by someguyorother 6y ago
You're going to have to implement a value system into the AGI if it is a fully general AGI. Otherwise, it will just use its exceptional intelligence to defeat its own sensors, take control of its objective function, and essentially wirehead itself.
So an AGI without a value system is (mostly) harmless. If the AGI has a defined value system, it's a lot clearer what that value system is. No "whoops gonna destroy the world to make paperclips" will happen.
(A not-fully general AGI might not be smart enough to defeat its own sensors, yet still be smart enough to do lots of harm. The typical example is grey goo, which isn't smart at all, yet plenty harmful.)
- amcknight 6y agoAn AGI with a small goal that it can wirehead to max out is not harmless by default. It can always add 9s to its confidence that it isn't being tricked and then secure itself while doing so.