3 ms·
The most intelligent person ever born could still die to a gun. In these discussions superintelligent AI can be more accurately described as "the genie" or "God
by seventhtiger 3y ago
The most intelligent person ever born could still die to a gun. In these discussions superintelligent AI can be more accurately described as "the genie" or "God". If you assume omniscience and omnipotence I guess nothing else matters. But intelligence is not equal to power, and never has.
Second, if you are able to set a goal then during this setting you can set many constraints, even fundamental ones. There is no reason the goal is more fundamental than the constraint. If I approve, make paperclips. Efficiently make 100 paperclips.
It's the duality of being able to set a rule but not being able to set a constraint that I find a strange concept. I lean towards the picture of not being able to set goals nor constraints at all.
- circuit10 3y agoIntelligence definitely helps with gaining power. Humans aren’t very strong yet we have a lot of power thanks to our intelligence. You can set constraints just fine. It’s simply a part of the goal: “do x without doing y”. It’s just really hard to find the right constraints, no simple one works. For example “if I approve, make paperclips” - so it gets more reward if you approve? What’s to stop it from manipulating you into thinking nothing is wrong so you always approve? “Efficiently make 100 paperclips.” I already linked a video on why capping the reward like that doesn’t work, but if you don’t want to watch it the gist is that for your suggestion it may just make a maximiser which is pretty guaranteed to make at least 100, and is pretty efficient because it’s not doing much work itself. Then the maximiser kills us all