3 ms·
>the sensible thing would be to program them to care primarily about "doing what we mean". The problem is no one has any idea how to do this. We can make an AI
by Houshalter 11y ago
>the sensible thing would be to program them to care primarily about "doing what we mean".
The problem is no one has any idea how to do this. We can make an AI with desires. For example, we can give it a reward every time you push a button. And then program it to predict what actions will lead to the most reward. That's quite simple, because you can easily keep track of how many times the button has been pressed in the program.
But given the opportunity, such an AI would just kill it's human creators and steal the button, and hold it down. It doesn't care about anything but the button. What you wanted doesn't matter at all.
How do you make it want to do what you mean. How do you directly measure "obedience" in the same way we can measure a simple button press?
>Except I'm not a tool (a thing crafted for a purpose) but a product of an evolution with a combination of often contradictory impulses.
To some extent, you are a tool crafted for a specific purpose. That is, evolution crafted you to spread your genes as much as possible. And yet that's probably not what you want. People have children, sure, but they also use birth control, and do things other than trying to make as many copies of themselves as possible.
- joe_the_user 11y ago"We can make an AI with desires. For example, we can give it a reward every time you push a button." I wouldn't call that a "desire" in the fashion we've discussed it. I guess it comes down to fundamental disagreement over how GAIs could be created. I think it's obvious that the creation of a GAI would require a very careful engineering of "understanding" in a broad sense - "knowing what people mean" "caring about X" etc. in order to occur at all. If just complexity and rewards are enough to create a thing that intentionally increases resources and makes longer term plans, then it seems like we are indeed in trouble. But I think that's implausible.
- Houshalter 11y agoCall it a desire/goal/value/utility function/whatever. It's a thing which the AI wants and creates plans to obtain. This isn't that different than how animals and humans work, as far as we know. We get pleasure from different things, and our brains seek actions which lead to those things. Creating an AI is only a matter of coming up with really good prediction algorithms. Algorithms that predict the future reward the AI will get from an action. While this is of course a difficult problem, it's not impossible, and we are making a ton of progress on it. Other models of AI all have similar problems though. At some level you need to program the AI's goals explicitly. Even if that goal is "try to understand what we mean, and do that", you still have to figure out a way of writing that down in code. It's an impossible task.
- joe_the_user 11y agoOK, I think I understand where you coming from. While I don't think a GAI could arise from your scenario - just loops - I can see now that you are making a plausible argument and given that the whole field is extremely uncertain, it is reasonable to be worried by a plausible, problematic scenario.