>We simply need to make sure that the AGI doesn't optimize anything, but instead takes the saying "all things in moderation" to its heart. This may be easier s

>We simply need to make sure that the AGI doesn't optimize anything, but instead takes the saying "all things in moderation" to its heart. This may be easier said than done. This is a fairly common proposal actually. The term used is "satisficer". Instead of having the AI optimize a quantity, you instead have it satisfice it, I.e. have it attempt to get at least a particular value. But, iirc, the standard reply is that satisficers are not stable under self modification; a satisficer may self modify into an optimizer for some quantity, because this will also satisfice whatever it is a satisficer for. However, there are setups which have been shown to be stable under self modification, and don't optimize the expected value of a function. Iirc, one which is stable is one that maximizes a function applied to the expected value of the utility function (so, optimizes f(E(u))), where the function f is increasing up to a certain point, and after that point it is decreasing.