3 ms·

I have the intuition that, given a specific "dangerous" goal (i.e., paperclip maximization), AGI will have too much autonomy to stick to it and AI won't have en

• PeterisP 9y ago
  Can you elaborate what you mean by "AGI will have too much autonomy to stick to it" ? Any system that is capable decide that e.g. paperclip maximization is a bad goal must unavoidably have some scale of what constitutes better or worse goals... and that de facto means that whatever is at the "good" end of that scale will be the true goal of that system. But where does that scale come from? Especially given that this scale is absolutely arbitrary, there are no competing "innate drives" (like mammals have) that would make it difficult to stick with any arbitrary goal. We're not talking about giving some intelligence orders that might conflict with what it "really wants" - we're talking about configuring the ultimate desires of that system, this configuration will define the world states that it will find more or less "desirable" given complete autonomy.