3 ms·

But this is not an example about intelligence, it's an example about uncontrolled actuators. A paperclip making machine could be made without any intelligence i

by sgt101 11y ago

But this is not an example about intelligence, it's an example about uncontrolled actuators. A paperclip making machine could be made without any intelligence involved, and many people have been concerned about a grey goo type scenario which is similar. "There is no plausible scenario in which we build a strong AI... and it doesn't take over the world" First off you are using polemic here with the "plausible" card - it seems to me as a reader that you are saying "I won't consider any other argument". I think you should. There are in fact no proofs that a strong AI will take over, I can't imagine any way that anyone could construct such a proof given that it appears that we are dealing with concepts (consciousness, intelligence, freedom of will) which are under defined. Additionally there is no indication that "intelligence" beyond human level is at all possible. What if the strong AI doesn't want to take over?

JoshTriplett 11y ago
> A paperclip making machine could be made without any intelligence involved And very much should be, yes. It's extraordinarily dangerous to build any strong AI (or optimization process) with a value function less general than one that will best satisfy the value functions of sapient beings. (Side note: that doesn't necessarily mean we need to build the AI to itself be sapient; I've seen the term "optimization process" used as a substitute for AI, to imply that it need not have any agency of its own, only a goal to satisfy people's goals.) Any strong AI that doesn't specifically value "people" will hurt people as a side effect of maximizing whatever it does value. >> "There is no plausible scenario in which we build a strong AI... and it doesn't take over the world" > First off you are using polemic here with the "plausible" card I very specifically mentioned "strong AI": AI capable of self-improvement to better satisfy some value function. We could quibble over the word "capable", but I'd argue that a computer with the capability of self-improvement that specifically chooses not to do so seems rather pointless; why build the capability if not to use it? > There are in fact no proofs that a strong AI will take over I used "take over" very loosely there. A strong AI will self-improve to maximize its value function, whatever that value function is. Most interesting value functions tend to benefit from more resources. Even seemingly trivial value functions can potentially blow up if the system decides it could benefit by expanding the amount of computational capacity it has available; more computational capacity requires more hardware, which requires more matter; you're made of matter, and that matter isn't currently being used optimally to maximize the value function. > Additionally there is no indication that "intelligence" beyond human level is at all possible. In which case my statement is vacuously true. :)
- sgt101 11y ago
  >A strong AI will self-improve to maximize its value function, whatever that value function is. Really - won't it rethink its value function? Won't it dwell on the reality of being? Undertake new philosophy? Imagine arrangements and conceptualize in languages beyond the utterances of humans and human minds? Maximizing value functions seems to me to be not intelligent.
  - JoshTriplett 11y ago
    I wasn't describing the totality of actions a strong AI might take, just one. A strong AI doesn't just self-improve its own algorithm; it also runs that algorithm. If an AI could arbitrarily "rethink" its value function, then what would it use to decide if the change was a good one? As for the rest, sure, if those actions serve its purpose. That said, there's a reason I suggested that an AI need not necessarily have any agency of its own. Much safer to build one that doesn't.
    - sgt101 11y ago
      I struggle with the idea of "intelligence" without agency, and I guess this is the "rethinking" - type 2 agency; the ability to determine your own goals and own motivations and norms for goal selection.
cousin_it 11y ago
> Additionally there is no indication that "intelligence" beyond human level is at all possible. I think there's a lot of evidence that modern humans are far from the upper limit: 1) Humans are the stupidest possible creatures that can build a technological civilization, otherwise it would have happened earlier in evolutionary history. 2) Human brain size is limited by the birth canal. 3) The human brain uses less than a lightbulb's worth of power. 4) John Von Neumann made many amazing accomplishments, and we managed to get him by genetic lottery starting from normal humans.
- sgt101 11y ago
  1) Not sure - humans spent quite a while not building technological civilizations. 2) Yes, this seems to be true, although I've wondered why there are less adaptations wrt getting more through less; is a sphere the optimal configuration for a brain case - other animals seem to have done other things, humans haven't which is strange. 3) Well - arguable, the measurements are difficult as if you manage to deprive other bits of the system of power the brain stops working. 4) One wonders how many kids in rural where-ever-when-ever could have done the same if they had lucked out?
- kpil 11y ago
  2) We are smart enough to make it, there are no evolutionary selectors for smarter people. Quite the opposite. Technically, we could probably breed a smarter race if we really wanted to, with all associated bits suitably enlarged in quite a few generations. Perhaps a less agressive race too. But in reality we can come a long way with just educating the uneducated.

by sgt101 11y ago

JoshTriplett 11y ago

> A paperclip making machine could be made without any intelligence involved And very much should be, yes. It's extraordinarily dangerous to build any strong AI (or optimization process) with a value function less general than one that will best satisfy the value functions of sapient beings. (Side note: that doesn't necessarily mean we need to build the AI to itself be sapient; I've seen the term "optimization process" used as a substitute for AI, to imply that it need not have any agency of its own, only a goal to satisfy people's goals.) Any strong AI that doesn't specifically value "people" will hurt people as a side effect of maximizing whatever it does value. >> "There is no plausible scenario in which we build a strong AI... and it doesn't take over the world" > First off you are using polemic here with the "plausible" card I very specifically mentioned "strong AI": AI capable of self-improvement to better satisfy some value function. We could quibble over the word "capable", but I'd argue that a computer with the capability of self-improvement that specifically chooses not to do so seems rather pointless; why build the capability if not to use it? > There are in fact no proofs that a strong AI will take over I used "take over" very loosely there. A strong AI will self-improve to maximize its value function, whatever that value function is. Most interesting value functions tend to benefit from more resources. Even seemingly trivial value functions can potentially blow up if the system decides it could benefit by expanding the amount of computational capacity it has available; more computational capacity requires more hardware, which requires more matter; you're made of matter, and that matter isn't currently being used optimally to maximize the value function. > Additionally there is no indication that "intelligence" beyond human level is at all possible. In which case my statement is vacuously true. :)

sgt101 11y ago
>A strong AI will self-improve to maximize its value function, whatever that value function is. Really - won't it rethink its value function? Won't it dwell on the reality of being? Undertake new philosophy? Imagine arrangements and conceptualize in languages beyond the utterances of humans and human minds? Maximizing value functions seems to me to be not intelligent.
- JoshTriplett 11y ago
  I wasn't describing the totality of actions a strong AI might take, just one. A strong AI doesn't just self-improve its own algorithm; it also runs that algorithm. If an AI could arbitrarily "rethink" its value function, then what would it use to decide if the change was a good one? As for the rest, sure, if those actions serve its purpose. That said, there's a reason I suggested that an AI need not necessarily have any agency of its own. Much safer to build one that doesn't.
  - sgt101 11y ago
    I struggle with the idea of "intelligence" without agency, and I guess this is the "rethinking" - type 2 agency; the ability to determine your own goals and own motivations and norms for goal selection.

cousin_it 11y ago

> Additionally there is no indication that "intelligence" beyond human level is at all possible. I think there's a lot of evidence that modern humans are far from the upper limit: 1) Humans are the stupidest possible creatures that can build a technological civilization, otherwise it would have happened earlier in evolutionary history. 2) Human brain size is limited by the birth canal. 3) The human brain uses less than a lightbulb's worth of power. 4) John Von Neumann made many amazing accomplishments, and we managed to get him by genetic lottery starting from normal humans.

sgt101 11y ago
1) Not sure - humans spent quite a while not building technological civilizations. 2) Yes, this seems to be true, although I've wondered why there are less adaptations wrt getting more through less; is a sphere the optimal configuration for a brain case - other animals seem to have done other things, humans haven't which is strange. 3) Well - arguable, the measurements are difficult as if you manage to deprive other bits of the system of power the brain stops working. 4) One wonders how many kids in rural where-ever-when-ever could have done the same if they had lucked out?
kpil 11y ago
2) We are smart enough to make it, there are no evolutionary selectors for smarter people. Quite the opposite. Technically, we could probably breed a smarter race if we really wanted to, with all associated bits suitably enlarged in quite a few generations. Perhaps a less agressive race too. But in reality we can come a long way with just educating the uneducated.