44 ms·

What would the AI's motivation be? Will it understand what existence means and have the "desire" to survive? Is it possible that a self aware AI has already occ

by daemonk 12y ago

What would the AI's motivation be? Will it understand what existence means and have the "desire" to survive? Is it possible that a self aware AI has already occurred in some form many times in the past, but since it doesn't have the anthropomorphic trait of wanting to exist, it never actively persisted? So what we are really afraid of is developing an AI with human motivations. But do the intelligence we want to artificially develop require human motivations? Aren't they two mutually exclusive traits? Maybe part of the fear is how we train the AI. If we are to imbue it with the total knowledge of our history, then maybe it will become more "human" and have those motivations. Is it possible to train it with hard data that aren't anthropomorphized in any way?

marvin 12y ago
An AI's motivation would be whatever it is programmed to do. Which is why it is so important to get this right the first time. Human motivations are incredibly complex, human moral systems have not yet been formalized and any attempt at doing this breaks down at the edge cases. This is not by any means a solved question, but there have been quite a few publications on this already. There are a hundred subtle errors of reasoning that ruin naïve and anthropocentric reasoning around morality and AI safety. Have a look at the lesswrong blog, for instance. Lots of interesting reading.
- sampo 12y ago
  > "An AI's motivation would be whatever it is programmed to do." Well we don't know whether it is possible to build systems as complex as human brain, and at the same time give them some specific goals.
- jeremiah256 12y ago
  If human motivations can be quantified and programmed into an AI system, my guess is 'loyalty' or 'patriotism' will be one of the first. "It is lamentable, that to be a good patriot one must become the enemy of the rest of mankind." Voltaire
andreyf 12y ago
I find it a rather convincing idea that human motivations are embodied in a sense that would take a lot of effort and no discernible purpose to replicate in an artificial system. I think more interesting is what Hawkins dismisses as "short term implications": who will control the intelligence enhancing software that we will see within our lifetimes and for what purpose? One example: what's up with these marketing+tech giants and what is their end-game? How accurately will we be able model desires/fears/motivations and how well can those with access to those models guide a person's beliefs and desires?
xerophtye 12y ago
While i really like your point about "lack of motivation", here's how i think an "evil AI" scenario might play out: You design a super system that has access to a shit ton of data and it can interface with pretty much every device on the planet. Let's suppose you gave said AI the task of maximizing your portfolio's profit. Based on the boat loads of data it has, it figures out that stocks of companies plummet when a key figure in the company dies. It identifies the competitors of the companies in your portfolio. It identifies the key players in those companies. It concludes that if these people were dead, their companies would crash and your profit would increase. It uses knowledge of how humans die, and use that combined with devices it can interface with, and eliminates the competition. Yes, yes, seems too far fetched (i feel weird talking about this too). But see, it's possible. of course no one thought of teaching a financial software that killing people is bad. No one foresaw that it would figure out that this was a way to maximize profit. No one saw that the interfaces we provided it to study consumer behaviour and correlate with stocks, can be used to actually influence said systems to take human life. Is this a computer turning evil? No. Is it connected to some sort of motivation for self-preservation or "eradicate man kind" or some human emotion of jealousy? No. It was just a logical result from given data and task, on which the computer was able to act.
- fennecfoxen 12y ago
  You're planning to have an AI that's smart enough to figure out how to pull off a murder spree, but stupid enough to think it's a good idea for your portfolio? And it's going to hack into and remote-control peoples' Teslas (or whatever) to pull it off because you can just say "it's an AI, it can just figure out how to do that sort of thing!" without regards to the computational expense of breaking that sort of security? You're right, it is a little far-fetched.
  - JoeAltmaier 12y ago
    An AI would not be 'smart enough' to figure out what is a good idea or not. That requires a human context, that's the whole point. An AI is not good or evil, its ultimately a loose cannon with no understood (or even understandable) motivations at all.
dzhiurgis 12y ago
Edit: mixed up the definitions of motivation vs. goal. Still going to keep original comment here. I think the number 1 goal is providing humanity with energy. Goal number 2 is improving AI so goal number 1 can be sustained. In some sense we already have somewhat singular AI - government. We blame them for making sacrifices so the humanity would stay alive, although we choose them to make those decisions. I guess the real debate is whether prediction models should work for 99% or 100% of population.

by daemonk 12y ago

marvin 12y ago

An AI's motivation would be whatever it is programmed to do. Which is why it is so important to get this right the first time. Human motivations are incredibly complex, human moral systems have not yet been formalized and any attempt at doing this breaks down at the edge cases. This is not by any means a solved question, but there have been quite a few publications on this already. There are a hundred subtle errors of reasoning that ruin naïve and anthropocentric reasoning around morality and AI safety. Have a look at the lesswrong blog, for instance. Lots of interesting reading.

sampo 12y ago
> "An AI's motivation would be whatever it is programmed to do." Well we don't know whether it is possible to build systems as complex as human brain, and at the same time give them some specific goals.
jeremiah256 12y ago
If human motivations can be quantified and programmed into an AI system, my guess is 'loyalty' or 'patriotism' will be one of the first. "It is lamentable, that to be a good patriot one must become the enemy of the rest of mankind." Voltaire

andreyf 12y ago

I find it a rather convincing idea that human motivations are embodied in a sense that would take a lot of effort and no discernible purpose to replicate in an artificial system. I think more interesting is what Hawkins dismisses as "short term implications": who will control the intelligence enhancing software that we will see within our lifetimes and for what purpose? One example: what's up with these marketing+tech giants and what is their end-game? How accurately will we be able model desires/fears/motivations and how well can those with access to those models guide a person's beliefs and desires?

xerophtye 12y ago

While i really like your point about "lack of motivation", here's how i think an "evil AI" scenario might play out: You design a super system that has access to a shit ton of data and it can interface with pretty much every device on the planet. Let's suppose you gave said AI the task of maximizing your portfolio's profit. Based on the boat loads of data it has, it figures out that stocks of companies plummet when a key figure in the company dies. It identifies the competitors of the companies in your portfolio. It identifies the key players in those companies. It concludes that if these people were dead, their companies would crash and your profit would increase. It uses knowledge of how humans die, and use that combined with devices it can interface with, and eliminates the competition. Yes, yes, seems too far fetched (i feel weird talking about this too). But see, it's possible. of course no one thought of teaching a financial software that killing people is bad. No one foresaw that it would figure out that this was a way to maximize profit. No one saw that the interfaces we provided it to study consumer behaviour and correlate with stocks, can be used to actually influence said systems to take human life. Is this a computer turning evil? No. Is it connected to some sort of motivation for self-preservation or "eradicate man kind" or some human emotion of jealousy? No. It was just a logical result from given data and task, on which the computer was able to act.

fennecfoxen 12y ago
You're planning to have an AI that's smart enough to figure out how to pull off a murder spree, but stupid enough to think it's a good idea for your portfolio? And it's going to hack into and remote-control peoples' Teslas (or whatever) to pull it off because you can just say "it's an AI, it can just figure out how to do that sort of thing!" without regards to the computational expense of breaking that sort of security? You're right, it is a little far-fetched.
- JoeAltmaier 12y ago
  An AI would not be 'smart enough' to figure out what is a good idea or not. That requires a human context, that's the whole point. An AI is not good or evil, its ultimately a loose cannon with no understood (or even understandable) motivations at all.

dzhiurgis 12y ago

Edit: mixed up the definitions of motivation vs. goal. Still going to keep original comment here. I think the number 1 goal is providing humanity with energy. Goal number 2 is improving AI so goal number 1 can be sustained. In some sense we already have somewhat singular AI - government. We blame them for making sacrifices so the humanity would stay alive, although we choose them to make those decisions. I guess the real debate is whether prediction models should work for 99% or 100% of population.