5 ms·

In short, because human intelligence evolved in a different environment than AI. We are the results of the need to survive and reproduce, and the majority of ou

by andreyf 11y ago

In short, because human intelligence evolved in a different environment than AI. We are the results of the need to survive and reproduce, and the majority of our "intelligence" is based on this foundation. Algorithms have no such darwinian pressures, and there is no point in developing AI replacements for them, since humans abound.

Moshe_Silnorin 11y ago
I thought much the same once. This paper convinced me otherwise: https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf https://selfawaresystems.files.wordpress.com/2008/01/ai_driv... I'd be interested in your take on it.
- andreyf 11y ago
  Only skimmed it, but seems like my thoughts would be similar to manish_gill's above [1]: the assumptions are rooted in pop science fantasy. More specifically, there are an infinite number of reasons for and against doing anything. Once we understand how humans weigh these reasons and choose between them and model them in an HGI system, we will be able to give it the values and morals that its creators choose. 1. https://news.ycombinator.com/item?id=10724108 https://news.ycombinator.com/item?id=10724108
feral 11y ago
You could make basically the same argument for why humans will never invent heavier-than-air flight - because avian flight 'evolved based on a need to survive and reproduce', and 'artificial flight doesn't have darwinian pressures'. But humans did make artificial flying machines. I don't mean to be unkind, but your argument doesn't have close to the level of certainly that we should bet our species on. No one knows where the roadblocks to AGI are, or what the timelines might be; but there is what seems like huge progress happening recently in an area which might eventually lead there. While no one knows the path, many intelligent people have thought about this without finding any theoretical roadblocks. Please don't publicly dismiss the concerns as 'science fiction' without a little more thought.
- andreyf 11y ago
  I'm not saying that HGI is impossible, but that the "risk scenarios" are complete fiction. In our journey to achieving HGI, we must first achieve an understanding of HGI ethics, motivations, and behavior. This will change the questions of risk into something that's tangible and addressable.
  - feral 11y ago
    On Ada Lovelace's 200 year birthday, we might remember her quote: "The Analytical Engine has no pretensions to originate anything. It can do whatever we know how to order it to perform." She was very smart, but she was wrong about that. >In our journey to achieving HGI, we must first achieve an understanding of HGI ethics, motivations, and behavior. You mean that it's impossible to make a human level general intelligence ('HGI', to use your term), without understanding those things? If so, I disagree with your assertion: it is not obvious that those things have to be done before a human level AGI could be made. It might be a good idea to do so, but its not clear that they are a necessary requirement. Some people would say this is why AGI is dangerous. Maybe there's a path to bootstrapping an AGI, where we build something from relatively simple bits we can understand, that then becomes very much more complex than we can understand. This happens all the time: for example, we get very complex classification behaviour from deep nets: [simple pieces] + [lots of data] doing something more complex than we have explicitly trained them to. If we were convinced that "In our journey to achieving HGI, we must first achieve an understanding of HGI ethics, motivations, and behavior" was a necessary thing to build before we could build a HGI, people would be less worried. But its not; and that's partially why people are worried. In fact, some of the people working on AI risk could be said to be basically racing to understand these things before other folk build an AGI.
    - joe_the_user 11y ago
      You mean that it's impossible to make a human level general intelligence ('HGI', to use your term), without understanding those things? It is entirely logical to argue that a high level intelligence could not be constructed without a broad and deep understanding of it's motivations and behaviors. That may or may not be ethics as such but if something we make on purpose, we'd need to understand it. And considering it's difficulties, making it on purpose seems necessary, contrary to movies etc.
robbensinger 11y ago
Stuart Russell (co-author of AI:MA, one of MIRI's research advisors) argues on http://edge.org/conversation/the-myth-of-ai#26015 http://edge.org/conversation/the-myth-of-ai#26015 that AI systems with "the ability to make high-quality decisions" (where "quality refers to the expected outcome utility of actions taken" and the utility function is represented in the system's programmed decision criteria) raises two problems: "1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down. "2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task." The first of those is what Bostrom calls "perverse instantiation" and Dietterich and Horvitz call the "Sorcerer's Apprentice" problem (http://cacm.acm.org/magazines/2015/10/192386-rise-of-concerns-about-ai/fulltext http://cacm.acm.org/magazines/2015/10/192386-rise-of-concern...). The second of these is what Bostrom calls "convergent instrumental goals" and Omohundro calls "basic AI drives." The first of these seems like a fairly obvious problem, if we think AI systems will ever be trusted with making important decisions. Human goals are complicated, and even a superintelligent system that can easily learn about our goals won't necessarily acquire the goals thereby. So solving the AI problem doesn't get us a solution to the goal specification problem for free. The second of these also has some intuitive force; https://intelligence.org/?p=12234 https://intelligence.org/?p=12234 shows Omohundro's idea can be stated formally, so it's not purely sci-fi. Averting the "Sorcerer's Apprentice" problem in full generality would mean averting this problem, since we'd then simply be able to give AI systems the right goals and let them go wild. Absent that, if AI systems become much more cognitively capable than humans, we'll probably need to actively work on some approach that violates Omohundro's assumptions (and the assumptions of the formalism above). Bostrom and MIRI both talk about a lot of interesting ideas along these lines.
- andreyf 11y ago
  What is "an AI system with the ability to make high-quality decisions"? Do automated derivative trading models count? Do systems which decide how much to bid on a RTB ad exchange count? The first problem is not new. We have a similar problem with some corporations, for example. "A sufficiently capable intelligent system" is as real as "sufficiently hostile aliens". It's hard to argue and reason about a fictional system with a assortment of properties picked by someone aiming to spreading fear.

by andreyf 11y ago

Moshe_Silnorin 11y ago

I thought much the same once. This paper convinced me otherwise: https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf https://selfawaresystems.files.wordpress.com/2008/01/ai_driv... I'd be interested in your take on it.

andreyf 11y ago
Only skimmed it, but seems like my thoughts would be similar to manish_gill's above [1]: the assumptions are rooted in pop science fantasy. More specifically, there are an infinite number of reasons for and against doing anything. Once we understand how humans weigh these reasons and choose between them and model them in an HGI system, we will be able to give it the values and morals that its creators choose. 1. https://news.ycombinator.com/item?id=10724108 https://news.ycombinator.com/item?id=10724108

feral 11y ago

You could make basically the same argument for why humans will never invent heavier-than-air flight - because avian flight 'evolved based on a need to survive and reproduce', and 'artificial flight doesn't have darwinian pressures'. But humans did make artificial flying machines. I don't mean to be unkind, but your argument doesn't have close to the level of certainly that we should bet our species on. No one knows where the roadblocks to AGI are, or what the timelines might be; but there is what seems like huge progress happening recently in an area which might eventually lead there. While no one knows the path, many intelligent people have thought about this without finding any theoretical roadblocks. Please don't publicly dismiss the concerns as 'science fiction' without a little more thought.

andreyf 11y ago
I'm not saying that HGI is impossible, but that the "risk scenarios" are complete fiction. In our journey to achieving HGI, we must first achieve an understanding of HGI ethics, motivations, and behavior. This will change the questions of risk into something that's tangible and addressable.
- feral 11y ago
  On Ada Lovelace's 200 year birthday, we might remember her quote: "The Analytical Engine has no pretensions to originate anything. It can do whatever we know how to order it to perform." She was very smart, but she was wrong about that. >In our journey to achieving HGI, we must first achieve an understanding of HGI ethics, motivations, and behavior. You mean that it's impossible to make a human level general intelligence ('HGI', to use your term), without understanding those things? If so, I disagree with your assertion: it is not obvious that those things have to be done before a human level AGI could be made. It might be a good idea to do so, but its not clear that they are a necessary requirement. Some people would say this is why AGI is dangerous. Maybe there's a path to bootstrapping an AGI, where we build something from relatively simple bits we can understand, that then becomes very much more complex than we can understand. This happens all the time: for example, we get very complex classification behaviour from deep nets: [simple pieces] + [lots of data] doing something more complex than we have explicitly trained them to. If we were convinced that "In our journey to achieving HGI, we must first achieve an understanding of HGI ethics, motivations, and behavior" was a necessary thing to build before we could build a HGI, people would be less worried. But its not; and that's partially why people are worried. In fact, some of the people working on AI risk could be said to be basically racing to understand these things before other folk build an AGI.
  - joe_the_user 11y ago
    You mean that it's impossible to make a human level general intelligence ('HGI', to use your term), without understanding those things? It is entirely logical to argue that a high level intelligence could not be constructed without a broad and deep understanding of it's motivations and behaviors. That may or may not be ethics as such but if something we make on purpose, we'd need to understand it. And considering it's difficulties, making it on purpose seems necessary, contrary to movies etc.

robbensinger 11y ago

Stuart Russell (co-author of AI:MA, one of MIRI's research advisors) argues on http://edge.org/conversation/the-myth-of-ai#26015 http://edge.org/conversation/the-myth-of-ai#26015 that AI systems with "the ability to make high-quality decisions" (where "quality refers to the expected outcome utility of actions taken" and the utility function is represented in the system's programmed decision criteria) raises two problems: "1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down. "2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task." The first of those is what Bostrom calls "perverse instantiation" and Dietterich and Horvitz call the "Sorcerer's Apprentice" problem (http://cacm.acm.org/magazines/2015/10/192386-rise-of-concerns-about-ai/fulltext http://cacm.acm.org/magazines/2015/10/192386-rise-of-concern...). The second of these is what Bostrom calls "convergent instrumental goals" and Omohundro calls "basic AI drives." The first of these seems like a fairly obvious problem, if we think AI systems will ever be trusted with making important decisions. Human goals are complicated, and even a superintelligent system that can easily learn about our goals won't necessarily acquire the goals thereby. So solving the AI problem doesn't get us a solution to the goal specification problem for free. The second of these also has some intuitive force; https://intelligence.org/?p=12234 https://intelligence.org/?p=12234 shows Omohundro's idea can be stated formally, so it's not purely sci-fi. Averting the "Sorcerer's Apprentice" problem in full generality would mean averting this problem, since we'd then simply be able to give AI systems the right goals and let them go wild. Absent that, if AI systems become much more cognitively capable than humans, we'll probably need to actively work on some approach that violates Omohundro's assumptions (and the assumptions of the formalism above). Bostrom and MIRI both talk about a lot of interesting ideas along these lines.

andreyf 11y ago
What is "an AI system with the ability to make high-quality decisions"? Do automated derivative trading models count? Do systems which decide how much to bid on a RTB ad exchange count? The first problem is not new. We have a similar problem with some corporations, for example. "A sufficiently capable intelligent system" is as real as "sufficiently hostile aliens". It's hard to argue and reason about a fictional system with a assortment of properties picked by someone aiming to spreading fear.