3 ms·

> if we make machines that are smarter than us and have desires, they will influence the future to achieve their desires. If these desires conflict with our own

> if we make machines that are smarter than us and have desires, they will influence the future to achieve their desires. If these desires conflict with our own Aren't you anthropomorphising them here? Just because something might be intelligent, it doesn't follow that they automatically gain other attributes that we have, such as desires. Why would any AI have anything that could meaningfully be called "desires", except in the sense that it is designed to achieve its purpose, e.g. a self-driving car "desires" to get to its destination? > there is no reason to expect the first AIs to have goals that correspond to what we want. There's every reason to expect this. Look at the ways in which we use AI at the moment. It's to achieve our goals. We build self-driving cars to get from A to B. We build optimising query planners to make our databases run faster. We build software that designs antennae so we have better antennae. Nobody wants to build software that doesn't do what they want it to. You know what you call intelligent software that has a goal that doesn't correspond to what we want? Broken and useless. Who wants to build broken and useless software?

Rangi42 12y ago
Making something intelligent does not automatically give it desires. But as you said, there is every reason to expect that we would build AI to achieve our goals, which means making it (a) desire to achieve a goal we share and (b) intelligent enough to succeed. (This is not anthropomorphizing AI; there's no need to make it feel desire the way we do, it just needs to tend to reach a state where the goal is achieved.) The danger comes from our inability to accurately define our own desires. There is no doWhatIMean() function. A programmed goal can subtly differ from the one we want so that it passes the obvious test cases but breaks catastrophically in an unusual real-world situation. There's an anecdote[1] about a neural network trained to detect camouflaged tanks (I've also heard of it detecting Soviet versus NATO tanks); it ends up learning to detect objects with shadows, because the tanks were photographed on a sunny day and the non-tanks on an overcast day. Now imagine if the program had been prematurely instructed to attack the camouflaged tanks. You can call this "broken and useless software," but by the time we realize how it's broken it could be too late. [1]: http://www.j-paine.org/dobbs/neural_net_urban_legends.html http://www.j-paine.org/dobbs/neural_net_urban_legends.html

Rangi42 12y ago

Making something intelligent does not automatically give it desires. But as you said, there is every reason to expect that we would build AI to achieve our goals, which means making it (a) desire to achieve a goal we share and (b) intelligent enough to succeed. (This is not anthropomorphizing AI; there's no need to make it feel desire the way we do, it just needs to tend to reach a state where the goal is achieved.) The danger comes from our inability to accurately define our own desires. There is no doWhatIMean() function. A programmed goal can subtly differ from the one we want so that it passes the obvious test cases but breaks catastrophically in an unusual real-world situation. There's an anecdote[1] about a neural network trained to detect camouflaged tanks (I've also heard of it detecting Soviet versus NATO tanks); it ends up learning to detect objects with shadows, because the tanks were photographed on a sunny day and the non-tanks on an overcast day. Now imagine if the program had been prematurely instructed to attack the camouflaged tanks. You can call this "broken and useless software," but by the time we realize how it's broken it could be too late. [1]: http://www.j-paine.org/dobbs/neural_net_urban_legends.html http://www.j-paine.org/dobbs/neural_net_urban_legends.html