3 ms·
Why would an AI start with a human utility function? Why would it care about that if it wasn't explicitly built in?
by morlockabove 6y ago
Why would an AI start with a human utility function? Why would it care about that if it wasn't explicitly built in?
- agumonkey 6y agoself similarity recognition ? intelligent agent, as the AI itself, would represent a form a sibling.
- morlockabove 6y agoUnless the AI was formed under a selection mechanism that rewarded something like kin preference, there's no reason to think it would care. The orthogonality principle is that you could have any level of intelligence furthering any particular goal/utility function; there isn't some level of computational complexity where e.g. a paperclip maximiser's utility function magically changes. In the particular case of humans, our utility function(s) is(are) so complex that what we think of as our 'true values' can change (because our True values are some inscrutable tug-of-war between all the parts of the brain), and also we're hacked together by evolution, so just blindly trying to make a human smarter might change their values (or turn them mad, or cause seizures, or...). But this doesn't have to be the case in general for intelligent agents. In principle, you can build an AI whose terminal values remain stable as it improves its intelligence. (If this is impossible, we're doomed.) So unless you explicitly built an AI to care about all humans, there's no reason to think it magically would.
- agumonkey 6y agoCan't an advanced AI produce emergent concepts and integrate them on (and as) its own ?
- morlockabove 6y agoWhat exactly does 'emergent concepts' mean, in this context?
- agumonkey 6y agoUnplanned reasoning mechanism for instance. Not something the AI was preprogrammed or biased into perceiving.
- meowface 6y agoWhy do you assume it'd not want to harm other very similar artificial intelligences, let alone meat intelligences?