9 ms·

The author hasn't taken the time to read the arguments for AI risk. First, citing Kurzweil as a proponent of AI risk is very odd. Kurzweil thinks AI is the bees

The author hasn't taken the time to read the arguments for AI risk. First, citing Kurzweil as a proponent of AI risk is very odd. Kurzweil thinks AI is the bees knees and doesn't see much risk in it AFAICT. Second, neither Bostrom, MIRI, nor the Future of Life institute claim Moore's law will continue indefinitely or in any way endorse Kurzweil's work. As for his other comments on AI risk, here's what I wrote last year on a similar thread: Nobody is afraid of today's AI algorithms. But if we make machines that are smarter than us and have desires, they will influence the future to achieve their desires. If these desires conflict with our own, things will not end well for the dumber party. As we really have no idea what we, collectively, think of as a moral terminal goal, and less so how to formalize this, there is no reason to expect the first AIs to have goals that correspond to what we want. If AIs self-replicate in a competitive ecology, what would be selected for would be agents millions of times more intelligent than us who use their intellects only to make more copies of themselves - using all available resources including those we need to survive. I'd recommend this summary of the arguments: https://medium.com/@LyleCantor/russell-bostrom-and-the-risk-of-ai-45f69c9ee204#.shqn6fqa4 https://medium.com/@LyleCantor/russell-bostrom-and-the-risk-...

SolaceQuantum 11y ago
This also assumes an AI would have an inherent desire to propagate abd continue itself. There's no evidence of that, methinks.
- maaku 11y ago
  That's a derivable sub-goal for just about any top-level goal. If you don't exist or you don't have sufficient power, you can't make sure your top-level goals get met and stay met.
  - varjag 11y ago
    That's not a superintelligence, or even human-level intelligence though. Superintelligence will read the Bostrom book or this thread and understand the arguments, or indeed predict the consequences of its actions even from very limited data. Now what kind of decision it would arrive to is a very open question. Point is, superintelligence is more than a really really big A* algorithm.
    - cousin_it 11y ago
      Imagine someone develops a really big A* algorithm with many clever optimizations, gives it a goal of choosing actions that lead to maximum paperclips, and sets it loose upon the world. Let's call it Clippy. I hope it's clear that Clippy will read the Bostrom book and use the information to maximize paperclips, rather than do something it wasn't programmed to do like "help humanity". I'm not interested in arguing whether Clippy is truly "intelligent", let's leave that to the dictionary makers. But there's no law of nature saying you can't make Clippy, or that it won't be dangerous once made. All the evidence we have right now says that it's possible. The big problem right now is that many organizations are working to develop the moral equivalent of Clippy, they are making visible progress, they are in an arms race with each other so we shouldn't hope for restraint, and we have absolutely no idea how to make it safe.
      - varjag 11y ago
        > All the evidence we have right now says that it's possible. Speaking about A* and heuristic programming specifically, we have a pretty big corpus of evidence it's not possible. More than that, many would doubt if an intelligent system with a rigid agenda constrained by design is possible at all. The scenario you present is essentially the grey goo argument, and it does not require intelligence.
    - maaku 11y ago
      I believe you are anthropomorphizing too much. On what basis would it update its goals? What is the reason it would do so, and why does that reason make sense in the context of it's prior goals? More likely I think it mift learn from Bostrom's book that it has reason to fear human intervention.
      - varjag 11y ago
        And assuming it would fear anything is not anthropomorphizing at all? :) > On what basis would it update its goals? What is the reason it would do so, and why does that reason make sense in the context of it's prior goals? That's a very open ended question as well, given that the systems we discuss are not yet conceived. My point was that calling a superintelligence what by description is pretty dumb automation is a bit ridiculous.
  - SolaceQuantum 11y ago
    I'm not talking about subgoals, because the ultimate goal is the task which must be careful in scope obviously. What I'm concerned about is the assumption that an ai can't avoid situations it needs to propagate itself(its not a direct line to goal) and then simply not stop once a task is completed.
    - Smaug123 11y ago
      But propagating itself quite probably is a direct line to achieving its goal faster.
- Smaug123 11y ago
  If we're really careful about how we program the AI, then we might be able to make it shut down after doing some limited-scope task. However, if its task is not extremely limited in scope, then it has strong incentives to continue itself so that it can complete its task. An idea of how things could potentially go wrong if you're not ever so careful in this way is the following thought experiment: > You told the AI to make exactly 100 paperclips. But the AI is made of (and knows that it is made of) human-made components, which are notoriously fairly-to-mostly reliable but not 100% reliable. Say they're 99.999% reliable. The AI's sensors are currently giving it the answer of "I have made 100 paperclips". Can it trust them? It really really wants to have made 100 paperclips, and it's not completely sure of having made 100 paperclips, so it decides to verify whether it has made 100 paperclips. Boom, suddenly you have unbounded behaviour: "verify a probability". If the AI is not extremely carefully designed, unbounded behaviour will lead to the AI acquiring resources and perhaps attempting to improve itself, in order to better fulfill its task. Look how readily humans attempt to learn new things and get better at what we do, even though our own goal systems are extremely fuzzy: we don't pursue one task with single-minded devotion like we hope an AI would. (If our AI's goal systems are fuzzy, then we won't understand why it's doing what it's doing, and then it's probably curtains for humanity.)
CM30 11y ago
Don't these arguments kind of assume a few things though? Like, 'strong' AI being implemented into things that really don't need it? I mean, the paperclip example in the Medium article... did that factory ever need AI involved? Couldn't a simple computer program no more advanced than those of today do the exact same job... without the risk of all that 'thinking and making copies of itself' stuff? Maybe I'm being naive here, but you don't need strong AI in everything. And they also kind of assume that there's only one AI at a time (what if two factories are run by AIs designed to build different things, and they both end up in this sort of situation at the same time?), that everything is connected to the internet in some way, that humanity couldn't simply wipe out anything that poses a threat (and no, an AI in a factory environment wouldn't get access to anything that would allow for nuclear/chemical/biological weapons). Just seems like a lot of this speculation is based around a society that's making tons of careless mistakes one after another. But maybe I'm missing something here.
- cousin_it 11y ago
  Don't focus on one scenario leading to disaster. There are many scenarios pointing roughly in the same direction. Here's the argument in its shortest form: The current situation, where unmodified humans are the smartest creatures around and call the shots, is unstable. It could go several different ways depending on the desires and quirks of the first superhuman intelligence to appear. It could happen via math-based AI, nature-imitating AI, self-improving mind uploads, biological intelligence amplification, or other means. We're making fast progress on all these fronts, so I'd be surprised if at least one of those didn't happen within the next century, possibly much sooner. Since it's an arms race between many competing organizations, it's unrealistic to expect that all superhuman intelligences will be kept powerless. So one way or another, it will happen. How do we ensure that humanity and its values survive the transition?
  - click170 11y ago
    > How do we ensure that humanity and its values survive the transition? I think a better question is do we deserve to.
Chathamization 11y ago
> First, citing Kurzweil as a proponent of AI risk is very odd. The article doesn't do that. It says: > The leading theorist and cheerleader for mankind’s imminent disappearance into insignificance or worse is Ray Kurzweil, who extrapolates the exponential growth in technological capability characterised by Moore’s law to a point in the mid 2040s—the Singularity—where AI will be self-perpetuating and no longer reliant on human intellect. Kurzweil is being brought up as someone who thinks that some sort of superintelligence will soon surpass humans, which is true. The "cheerleader" comment is to indicate that, as you put it, "Kurzweil thinks AI is the bees knees."

SolaceQuantum 11y ago

This also assumes an AI would have an inherent desire to propagate abd continue itself. There's no evidence of that, methinks.

maaku 11y ago
That's a derivable sub-goal for just about any top-level goal. If you don't exist or you don't have sufficient power, you can't make sure your top-level goals get met and stay met.
- varjag 11y ago
  That's not a superintelligence, or even human-level intelligence though. Superintelligence will read the Bostrom book or this thread and understand the arguments, or indeed predict the consequences of its actions even from very limited data. Now what kind of decision it would arrive to is a very open question. Point is, superintelligence is more than a really really big A* algorithm.
  - cousin_it 11y ago
    Imagine someone develops a really big A* algorithm with many clever optimizations, gives it a goal of choosing actions that lead to maximum paperclips, and sets it loose upon the world. Let's call it Clippy. I hope it's clear that Clippy will read the Bostrom book and use the information to maximize paperclips, rather than do something it wasn't programmed to do like "help humanity". I'm not interested in arguing whether Clippy is truly "intelligent", let's leave that to the dictionary makers. But there's no law of nature saying you can't make Clippy, or that it won't be dangerous once made. All the evidence we have right now says that it's possible. The big problem right now is that many organizations are working to develop the moral equivalent of Clippy, they are making visible progress, they are in an arms race with each other so we shouldn't hope for restraint, and we have absolutely no idea how to make it safe.
    - varjag 11y ago
      > All the evidence we have right now says that it's possible. Speaking about A* and heuristic programming specifically, we have a pretty big corpus of evidence it's not possible. More than that, many would doubt if an intelligent system with a rigid agenda constrained by design is possible at all. The scenario you present is essentially the grey goo argument, and it does not require intelligence.
  - maaku 11y ago
    I believe you are anthropomorphizing too much. On what basis would it update its goals? What is the reason it would do so, and why does that reason make sense in the context of it's prior goals? More likely I think it mift learn from Bostrom's book that it has reason to fear human intervention.
    - varjag 11y ago
      And assuming it would fear anything is not anthropomorphizing at all? :) > On what basis would it update its goals? What is the reason it would do so, and why does that reason make sense in the context of it's prior goals? That's a very open ended question as well, given that the systems we discuss are not yet conceived. My point was that calling a superintelligence what by description is pretty dumb automation is a bit ridiculous.
- SolaceQuantum 11y ago
  I'm not talking about subgoals, because the ultimate goal is the task which must be careful in scope obviously. What I'm concerned about is the assumption that an ai can't avoid situations it needs to propagate itself(its not a direct line to goal) and then simply not stop once a task is completed.
  - Smaug123 11y ago
    But propagating itself quite probably is a direct line to achieving its goal faster.
Smaug123 11y ago
If we're really careful about how we program the AI, then we might be able to make it shut down after doing some limited-scope task. However, if its task is not extremely limited in scope, then it has strong incentives to continue itself so that it can complete its task. An idea of how things could potentially go wrong if you're not ever so careful in this way is the following thought experiment: > You told the AI to make exactly 100 paperclips. But the AI is made of (and knows that it is made of) human-made components, which are notoriously fairly-to-mostly reliable but not 100% reliable. Say they're 99.999% reliable. The AI's sensors are currently giving it the answer of "I have made 100 paperclips". Can it trust them? It really really wants to have made 100 paperclips, and it's not completely sure of having made 100 paperclips, so it decides to verify whether it has made 100 paperclips. Boom, suddenly you have unbounded behaviour: "verify a probability". If the AI is not extremely carefully designed, unbounded behaviour will lead to the AI acquiring resources and perhaps attempting to improve itself, in order to better fulfill its task. Look how readily humans attempt to learn new things and get better at what we do, even though our own goal systems are extremely fuzzy: we don't pursue one task with single-minded devotion like we hope an AI would. (If our AI's goal systems are fuzzy, then we won't understand why it's doing what it's doing, and then it's probably curtains for humanity.)

CM30 11y ago

Don't these arguments kind of assume a few things though? Like, 'strong' AI being implemented into things that really don't need it? I mean, the paperclip example in the Medium article... did that factory ever need AI involved? Couldn't a simple computer program no more advanced than those of today do the exact same job... without the risk of all that 'thinking and making copies of itself' stuff? Maybe I'm being naive here, but you don't need strong AI in everything. And they also kind of assume that there's only one AI at a time (what if two factories are run by AIs designed to build different things, and they both end up in this sort of situation at the same time?), that everything is connected to the internet in some way, that humanity couldn't simply wipe out anything that poses a threat (and no, an AI in a factory environment wouldn't get access to anything that would allow for nuclear/chemical/biological weapons). Just seems like a lot of this speculation is based around a society that's making tons of careless mistakes one after another. But maybe I'm missing something here.

cousin_it 11y ago
Don't focus on one scenario leading to disaster. There are many scenarios pointing roughly in the same direction. Here's the argument in its shortest form: The current situation, where unmodified humans are the smartest creatures around and call the shots, is unstable. It could go several different ways depending on the desires and quirks of the first superhuman intelligence to appear. It could happen via math-based AI, nature-imitating AI, self-improving mind uploads, biological intelligence amplification, or other means. We're making fast progress on all these fronts, so I'd be surprised if at least one of those didn't happen within the next century, possibly much sooner. Since it's an arms race between many competing organizations, it's unrealistic to expect that all superhuman intelligences will be kept powerless. So one way or another, it will happen. How do we ensure that humanity and its values survive the transition?
- click170 11y ago
  > How do we ensure that humanity and its values survive the transition? I think a better question is do we deserve to.

Chathamization 11y ago

> First, citing Kurzweil as a proponent of AI risk is very odd. The article doesn't do that. It says: > The leading theorist and cheerleader for mankind’s imminent disappearance into insignificance or worse is Ray Kurzweil, who extrapolates the exponential growth in technological capability characterised by Moore’s law to a point in the mid 2040s—the Singularity—where AI will be self-perpetuating and no longer reliant on human intellect. Kurzweil is being brought up as someone who thinks that some sort of superintelligence will soon surpass humans, which is true. The "cheerleader" comment is to indicate that, as you put it, "Kurzweil thinks AI is the bees knees."