Intelligence definitely helps with gaining power. Humans aren’t very strong yet we have a lot of power thanks to our intelligence. You can set constraints just

Intelligence definitely helps with gaining power. Humans aren’t very strong yet we have a lot of power thanks to our intelligence. You can set constraints just fine. It’s simply a part of the goal: “do x without doing y”. It’s just really hard to find the right constraints, no simple one works. For example “if I approve, make paperclips” - so it gets more reward if you approve? What’s to stop it from manipulating you into thinking nothing is wrong so you always approve? “Efficiently make 100 paperclips.” I already linked a video on why capping the reward like that doesn’t work, but if you don’t want to watch it the gist is that for your suggestion it may just make a maximiser which is pretty guaranteed to make at least 100, and is pretty efficient because it’s not doing much work itself. Then the maximiser kills us all