5 ms·

When you want to discuss "the myth of a superhuman AI", it's important to carefully separate the two categories of claims: 1. The claims by economists that AI-

When you want to discuss "the myth of a superhuman AI", it's important to carefully separate the two categories of claims: 1. The claims by economists that AI-- even if it's not "strong AI"-- will put lots of people out of a job with potentially severe societal/economic repercussions 2. The claims by the Bostrom/Yudkowsky/etc. crowd that an AI intelligence explosion will cause the extinction of humanity Without saying anything about the plausibility or lack thereof of either 1 or 2, I think we can all agree that they are very different claims and need to be analyzed separately. Right from the very first sentence the author seems to muddle the two, so I don't think there's much of cogent analysis in here.

mindcrime 10y ago
You make a good point, but it seemed pretty clear to me that the author was mostly discussing the (2) scenario as you labeled them. To me, his analysis was very cogent, considered within the broader framework of how we ultimately "just don't know" about some of this stuff.
deleted 10y ago
[deleted]
ffwd 10y ago
> 2. The claims by the Bostrom/Yudkowsky/etc. crowd that an AI intelligence explosion will cause the extinction of humanity Maybe someone can correct me if I'm wrong here, but I have a hard time understanding what /any/ "utility function" would be, that the super ai people talk about. It can't be a passive deep learning network that parses information and gives an output, it has to be some kind of complex perception / action loop of many neural nets and actuators in the real world that somehow leads to an intelligent self improving behavior? I guess you could make a deep learning controller for self driving cars say, and if an input to many cars is wrong, all the cars crash and create a big cascading mess of wrong input values, but that kind of accident is a far cry from an intelligent chain of events where every link in the chain is an intelligent decision but the ultimate goal is bad. And, do we even know any way to chain many deep learning networks together that accurately give correct output values, that we then can hook up to a controller to give a utility function, which can then lead to a cascade of intelligent decisions across domains?
- jononor 10y ago
  Check for instance the 'Paperclip maximizer' thought experiment, https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer
  - goatlover 10y ago
    The Paperclip maximizer assumes that you could have an AI smart enough to turn the world into paperclips, but not smart enough to understand what the human meant by "make me some paperclips". My guess is that paperclipping the world takes magnitudes more intelligence than understanding what a human means when saying something ambiguous.
    - Smaug123 10y ago
      Not necessarily. It could be smart enough to know what you meant (and indeed to deduce the entire history of your thoughts!) and simply not care, because you told it to care about something else. For example, maybe you inadvertently programmed it to "be totally literally 100% certain that you've completed the task I told you to do". Then from its perspective, there's always a tiny, tiny chance that its sensors are being fooled or malfunctioning, so it can't be literally 100% certain that it's ever made a paperclip successfully, so by your programming, it should keep making more paperclips. This is independent of whether or not you wanted it to do that: it's a result of what you programmed it to do and what you told it to do.
      - goatlover 10y ago
        If a human being is smart enough to realize that you don't want to paperclip the entire world, then surely a super AI would be smart enough, by definition. Part of intelligence is knowing when to ignore all those tiny chances. Human intelligence is less brittle than artificial because we can handle ambiguity and know when it's ridiculous to continue a task, just because there might be some small chance that it's not finished. We also know that paperclipping the world will get in the way of other goals, like going to the show or making money.
zeteo 10y ago
> Right from the very first sentence the author seems to muddle the two, so I don't think there's much of cogent analysis in here. If you read past the first sentence, it's quite obvious that the article is mostly concerned with claims of your type 2). Also I'm not sure why it would be so important to carefully separate the two categories of claims - they both rely on the assumption that AI can perform a wide variety of cognitive tasks better and cheaper than humans, which is addressed at length and cogently in the article. Also I believe that the economists you mention are concerned more about robotics than about AI per se, so I'm not even sure why you've brought in that point.

mindcrime 10y ago

You make a good point, but it seemed pretty clear to me that the author was mostly discussing the (2) scenario as you labeled them. To me, his analysis was very cogent, considered within the broader framework of how we ultimately "just don't know" about some of this stuff.

deleted 10y ago

[deleted]

ffwd 10y ago

> 2. The claims by the Bostrom/Yudkowsky/etc. crowd that an AI intelligence explosion will cause the extinction of humanity Maybe someone can correct me if I'm wrong here, but I have a hard time understanding what /any/ "utility function" would be, that the super ai people talk about. It can't be a passive deep learning network that parses information and gives an output, it has to be some kind of complex perception / action loop of many neural nets and actuators in the real world that somehow leads to an intelligent self improving behavior? I guess you could make a deep learning controller for self driving cars say, and if an input to many cars is wrong, all the cars crash and create a big cascading mess of wrong input values, but that kind of accident is a far cry from an intelligent chain of events where every link in the chain is an intelligent decision but the ultimate goal is bad. And, do we even know any way to chain many deep learning networks together that accurately give correct output values, that we then can hook up to a controller to give a utility function, which can then lead to a cascade of intelligent decisions across domains?

jononor 10y ago
Check for instance the 'Paperclip maximizer' thought experiment, https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer
- goatlover 10y ago
  The Paperclip maximizer assumes that you could have an AI smart enough to turn the world into paperclips, but not smart enough to understand what the human meant by "make me some paperclips". My guess is that paperclipping the world takes magnitudes more intelligence than understanding what a human means when saying something ambiguous.
  - Smaug123 10y ago
    Not necessarily. It could be smart enough to know what you meant (and indeed to deduce the entire history of your thoughts!) and simply not care, because you told it to care about something else. For example, maybe you inadvertently programmed it to "be totally literally 100% certain that you've completed the task I told you to do". Then from its perspective, there's always a tiny, tiny chance that its sensors are being fooled or malfunctioning, so it can't be literally 100% certain that it's ever made a paperclip successfully, so by your programming, it should keep making more paperclips. This is independent of whether or not you wanted it to do that: it's a result of what you programmed it to do and what you told it to do.
    - goatlover 10y ago
      If a human being is smart enough to realize that you don't want to paperclip the entire world, then surely a super AI would be smart enough, by definition. Part of intelligence is knowing when to ignore all those tiny chances. Human intelligence is less brittle than artificial because we can handle ambiguity and know when it's ridiculous to continue a task, just because there might be some small chance that it's not finished. We also know that paperclipping the world will get in the way of other goals, like going to the show or making money.

zeteo 10y ago

> Right from the very first sentence the author seems to muddle the two, so I don't think there's much of cogent analysis in here. If you read past the first sentence, it's quite obvious that the article is mostly concerned with claims of your type 2). Also I'm not sure why it would be so important to carefully separate the two categories of claims - they both rely on the assumption that AI can perform a wide variety of cognitive tasks better and cheaper than humans, which is addressed at length and cogently in the article. Also I believe that the economists you mention are concerned more about robotics than about AI per se, so I'm not even sure why you've brought in that point.