4 ms·
"Motivated" is a weird word, it anthropomorphises the machine. It will do what it's programmed to do. It doesn't have a personality. Formally specifying what w
by jtolmar 11y ago
"Motivated" is a weird word, it anthropomorphises the machine. It will do what it's programmed to do. It doesn't have a personality.
Formally specifying what we actually wanted is why programmers have a job. And the difficulty of that is why we have bugs. So that's still scary. But it's not the machine's choice in any way
- mikeash 11y agoYes, I shouldn't anthropomorphize computers, they hate that. Anyway, yes, it will optimize whatever function it's told to optimize. So you'd better hope that whoever gives it that function 1) cares about people's wellbeing and 2) is good at expressing that in a way the computer can interpret. 1 is pretty obvious, 2 may be a bit less so. For example, a superintelligent computer commanded to maximize global average happiness might decide the best route would be to work towards killing the entire human population except for one individual, who is kept constantly drugged out of their mind.