7 ms·

That's a derivable sub-goal for just about any top-level goal. If you don't exist or you don't have sufficient power, you can't make sure your top-level goals g

by maaku 11y ago

That's a derivable sub-goal for just about any top-level goal. If you don't exist or you don't have sufficient power, you can't make sure your top-level goals get met and stay met.

varjag 11y ago
That's not a superintelligence, or even human-level intelligence though. Superintelligence will read the Bostrom book or this thread and understand the arguments, or indeed predict the consequences of its actions even from very limited data. Now what kind of decision it would arrive to is a very open question. Point is, superintelligence is more than a really really big A* algorithm.
- cousin_it 11y ago
  Imagine someone develops a really big A* algorithm with many clever optimizations, gives it a goal of choosing actions that lead to maximum paperclips, and sets it loose upon the world. Let's call it Clippy. I hope it's clear that Clippy will read the Bostrom book and use the information to maximize paperclips, rather than do something it wasn't programmed to do like "help humanity". I'm not interested in arguing whether Clippy is truly "intelligent", let's leave that to the dictionary makers. But there's no law of nature saying you can't make Clippy, or that it won't be dangerous once made. All the evidence we have right now says that it's possible. The big problem right now is that many organizations are working to develop the moral equivalent of Clippy, they are making visible progress, they are in an arms race with each other so we shouldn't hope for restraint, and we have absolutely no idea how to make it safe.
  - varjag 11y ago
    > All the evidence we have right now says that it's possible. Speaking about A* and heuristic programming specifically, we have a pretty big corpus of evidence it's not possible. More than that, many would doubt if an intelligent system with a rigid agenda constrained by design is possible at all. The scenario you present is essentially the grey goo argument, and it does not require intelligence.
- maaku 11y ago
  I believe you are anthropomorphizing too much. On what basis would it update its goals? What is the reason it would do so, and why does that reason make sense in the context of it's prior goals? More likely I think it mift learn from Bostrom's book that it has reason to fear human intervention.
  - varjag 11y ago
    And assuming it would fear anything is not anthropomorphizing at all? :) > On what basis would it update its goals? What is the reason it would do so, and why does that reason make sense in the context of it's prior goals? That's a very open ended question as well, given that the systems we discuss are not yet conceived. My point was that calling a superintelligence what by description is pretty dumb automation is a bit ridiculous.
    - maaku 11y ago
      > And assuming it would fear anything is not anthropomorphizing at all? :) It's anthropomorphic language, yes, which is clumsy of me, especially given the context. What I meant is it would identify a category of risks, and then take action to mitigate those risks. > My point was that calling a superintelligence what by description is pretty dumb automation is a bit ridiculous. I will assume by "dumb" you mean "simple" or else that statement is somewhat tautological. There is nothing that says intelligence has to be complicated. Our brains, the neocortex in particular (which is responsible for most of human "intelligence" that separates us from other mammals) operates by rather simplistic principles. I don't want to bikeshed a definition of intelligence, but common phrasings thrown around are along the lines of "ability to achieve arbitrary goals in complex environments" or "efficiency at problem solving." These are not things that by their nature mandate a complicated solution. It is entirely possible that we could crate a superintelligence that runs a very simple algorithm.
      - varjag 11y ago
        It is entirely possible, or even plausible, that the algorithm behind a superintelligence would be simple. However I was talking about behaviour, and whether its behaviour can realistically restricted by an imposed constraint. More or less, the issue is whether the intelligence without its own agency is possible at all.
SolaceQuantum 11y ago
I'm not talking about subgoals, because the ultimate goal is the task which must be careful in scope obviously. What I'm concerned about is the assumption that an ai can't avoid situations it needs to propagate itself(its not a direct line to goal) and then simply not stop once a task is completed.
- Smaug123 11y ago
  But propagating itself quite probably is a direct line to achieving its goal faster.

by maaku 11y ago

That's a derivable sub-goal for just about any top-level goal. If you don't exist or you don't have sufficient power, you can't make sure your top-level goals get met and stay met.

varjag 11y ago

That's not a superintelligence, or even human-level intelligence though. Superintelligence will read the Bostrom book or this thread and understand the arguments, or indeed predict the consequences of its actions even from very limited data. Now what kind of decision it would arrive to is a very open question. Point is, superintelligence is more than a really really big A* algorithm.

cousin_it 11y ago
Imagine someone develops a really big A* algorithm with many clever optimizations, gives it a goal of choosing actions that lead to maximum paperclips, and sets it loose upon the world. Let's call it Clippy. I hope it's clear that Clippy will read the Bostrom book and use the information to maximize paperclips, rather than do something it wasn't programmed to do like "help humanity". I'm not interested in arguing whether Clippy is truly "intelligent", let's leave that to the dictionary makers. But there's no law of nature saying you can't make Clippy, or that it won't be dangerous once made. All the evidence we have right now says that it's possible. The big problem right now is that many organizations are working to develop the moral equivalent of Clippy, they are making visible progress, they are in an arms race with each other so we shouldn't hope for restraint, and we have absolutely no idea how to make it safe.
- varjag 11y ago
  > All the evidence we have right now says that it's possible. Speaking about A* and heuristic programming specifically, we have a pretty big corpus of evidence it's not possible. More than that, many would doubt if an intelligent system with a rigid agenda constrained by design is possible at all. The scenario you present is essentially the grey goo argument, and it does not require intelligence.
maaku 11y ago
I believe you are anthropomorphizing too much. On what basis would it update its goals? What is the reason it would do so, and why does that reason make sense in the context of it's prior goals? More likely I think it mift learn from Bostrom's book that it has reason to fear human intervention.
- varjag 11y ago
  And assuming it would fear anything is not anthropomorphizing at all? :) > On what basis would it update its goals? What is the reason it would do so, and why does that reason make sense in the context of it's prior goals? That's a very open ended question as well, given that the systems we discuss are not yet conceived. My point was that calling a superintelligence what by description is pretty dumb automation is a bit ridiculous.
  - maaku 11y ago
    > And assuming it would fear anything is not anthropomorphizing at all? :) It's anthropomorphic language, yes, which is clumsy of me, especially given the context. What I meant is it would identify a category of risks, and then take action to mitigate those risks. > My point was that calling a superintelligence what by description is pretty dumb automation is a bit ridiculous. I will assume by "dumb" you mean "simple" or else that statement is somewhat tautological. There is nothing that says intelligence has to be complicated. Our brains, the neocortex in particular (which is responsible for most of human "intelligence" that separates us from other mammals) operates by rather simplistic principles. I don't want to bikeshed a definition of intelligence, but common phrasings thrown around are along the lines of "ability to achieve arbitrary goals in complex environments" or "efficiency at problem solving." These are not things that by their nature mandate a complicated solution. It is entirely possible that we could crate a superintelligence that runs a very simple algorithm.
    - varjag 11y ago
      It is entirely possible, or even plausible, that the algorithm behind a superintelligence would be simple. However I was talking about behaviour, and whether its behaviour can realistically restricted by an imposed constraint. More or less, the issue is whether the intelligence without its own agency is possible at all.

SolaceQuantum 11y ago

I'm not talking about subgoals, because the ultimate goal is the task which must be careful in scope obviously. What I'm concerned about is the assumption that an ai can't avoid situations it needs to propagate itself(its not a direct line to goal) and then simply not stop once a task is completed.

Smaug123 11y ago
But propagating itself quite probably is a direct line to achieving its goal faster.