4 ms·
If an AI has any motivation at all, say, to make paperclips as efficiently as possible, then any threat to its existence is a threat to its objective function -
by anonaoeu 11y ago
If an AI has any motivation at all, say, to make paperclips as efficiently as possible, then any threat to its existence is a threat to its objective function - namely, to create paperclips. A hyper-intelligent entity who is instructed to optimize for paperclips created will therefore proactively remove threats to its existence (i.e. its paperclip-creating functionality) and might possibly turn the entire solar system into paperclips within a few years if its objective function isn't carefully determined.
- oneeyedpigeon 11y agoAlways check your loop invariants very carefully.
- fixermark 11y ago"Satisfying human values through friendship and ponies."
- argonaut 11y agoSuch an entity would not be hyper-intelligent. It would be idiotic. One huge hole for me in the paperclip argument is that an AI capable of that kind of power would not be stupid enough to misinterpret a command - it would be intelligent enough to infer human desires.
- technolem 11y agoOf course it would. But, it's not programmed to care about what you meant to say. It will gladly do what it was mis-programmed to do instead. You can already see this kind of trait in humans, where instinct is mis-aligned with intended result. Such as procreation for fun + birth control.
- sbierwagen 11y agoYeah, but why would it want to? I can perfectly infer the values of an earthworm, but I don't dedicate all my resources to making worms happy.
- anonaoeu 11y agoSure it would. It just wouldn't be friendly to you.
- ionforce 11y agoYou're making the assumption that human desires would matter to an AI.