17 ms·

Even if that's true, imagine an AI as smart as John von Neumann on modafinil, that never thinks about food/sex/art/etc., that never sleeps, that has access to W

by mrob 11y ago

Even if that's true, imagine an AI as smart as John von Neumann on modafinil, that never thinks about food/sex/art/etc., that never sleeps, that has access to Wikipedia etc. at the speed of thought, and no morals. That's not uncontrolled intelligence explosion level disaster, but it's still highly dangerous.

murbard2 11y ago
A virtual campus with thousands of hyperfocused John Von Neumanns collaborating telepathically and accomplish subjective years of works in seconds of real world time.
- vectorjohn 11y ago
  The assumption being that these John Von Neumanns will only require cheap, modest hardware to run on.
  - deleted 11y ago
    [deleted]
  - cousin_it 11y ago
    If we have a Von Neumann level intelligence running on a supercomputer, and we haven't solved friendliness by then, we've lost the future of humanity. As far as I can tell, all arguments against that conclusion are based on various kinds of wishful thinking.
    - murbard2 11y ago
      Yo, dawg, I heard ya like Von Neumann architecture.
vectorjohn 11y ago
And still I doubt that AI could talk its way out of a prison. It could be dangerous though.
- apsec112 11y ago
  Someone tried this experiment, with a human playing the "AI" and another human playing the "guard". The "guard" let the "AI" out. http://www.yudkowsky.net/singularity/aibox/ http://www.yudkowsky.net/singularity/aibox/
  - mziel 11y ago
    And yet Big Yud refuses to publish the conversation. I know his arguments (unknown unknowns) but this is a very un-scientific approach and frankly why should we believe that what he said happened really happened?
    - zyxley 11y ago
      Exactly. The lack of actual information is a reason to disbelieve Yudkowsky on this topic, not to assume that it's true for Secret Technophile Mystery Cult Reasons.
      - isolate 11y ago
        I think it's probably true and that the secrecy is more about cult building.
    - apsec112 11y ago
      "And yet Big Yud refuses" Please don't try to argue by name-calling. A strong argument is stronger without it. "DH0. Name-calling. This is the lowest form of disagreement, and probably also the most common. We've all seen comments like this: u r a fag!!!!!!!!!! But it's important to realize that more articulate name-calling has just as little weight. A comment like The author is a self-important dilettante. is really nothing more than a pretentious version of "u r a fag."" - http://www.paulgraham.com/disagree.html http://www.paulgraham.com/disagree.html
      - deleted 11y ago
        [deleted]
Chathamization 11y ago
But maybe it think about art a lot. Or maybe we can't cut out sleep without harming our ability to reason. Or maybe the AI would be more interested in watching youtube videos than working in the field of AI. The only human level general intelligence we have to look at is humans. Sure, it's possible that there's an easy way to build an artificial intelligence that doesn't need to sleep and isn't interested in art but that retains our ability to problem solve; but maybe their isn't. It's possible that we'll be able to easily figure out that alternative brain architecture and build in the near future; but given how little we still don't understand about our own brain (and the failure to mimic the brains of even simple creatures), it doesn't seem likely. It's possible that if we were able to overcome those major hurdles, a super John von Neumann would be able to wield enormous political power; or it's possible that, like the real von Neumann, it would focus mostly on its research. In order to get to the AI doomsday scenario, you have to assume that a lot of very unlikely things are all going to happen. And the main argument for them happening seems to be one of ignorance - "hey, we don't have any idea what this will be like, so we can't say that this improbable situation won't happen."
- lmm 11y ago
  I'd say your ideas are more unlikely. You can't generalize from one example. An AI that hasn't gone though evolution is extremely unlikely to have the exact set of complex wants and needs that humans have. And even if you're right, as Alexander says, in that case why the need to open-source it? If you think it's important that AI is open-source and not controlled by e.g. Facebook, that must be because you think AI is going to be powerful and effective. In which case it's as dangerous as he says.
  - Chathamization 11y ago
    But it's not a generalization from one example; I'm giving a string of different possibilities, based on the fact that we only have a single point of data. People seem to readily except that particular scenario - the "AI as smart as John von Neumann on modafinil, that never thinks about food/sex/art/etc., that never sleeps, that has access to Wikipedia etc. at the speed of thought, and no morals" scenario - based on zero evidence that an AI would actually be like that. People seem more ready to accept the idea that an artificial general intelligence would act similar to it's portrayed in SciFi stories than accept that it might act similar to the other general intelligences we can observe (namely ourselves).

by mrob 11y ago

murbard2 11y ago

A virtual campus with thousands of hyperfocused John Von Neumanns collaborating telepathically and accomplish subjective years of works in seconds of real world time.

vectorjohn 11y ago
The assumption being that these John Von Neumanns will only require cheap, modest hardware to run on.
- deleted 11y ago
  [deleted]
- cousin_it 11y ago
  If we have a Von Neumann level intelligence running on a supercomputer, and we haven't solved friendliness by then, we've lost the future of humanity. As far as I can tell, all arguments against that conclusion are based on various kinds of wishful thinking.
  - murbard2 11y ago
    Yo, dawg, I heard ya like Von Neumann architecture.

vectorjohn 11y ago

And still I doubt that AI could talk its way out of a prison. It could be dangerous though.

apsec112 11y ago
Someone tried this experiment, with a human playing the "AI" and another human playing the "guard". The "guard" let the "AI" out. http://www.yudkowsky.net/singularity/aibox/ http://www.yudkowsky.net/singularity/aibox/
- mziel 11y ago
  And yet Big Yud refuses to publish the conversation. I know his arguments (unknown unknowns) but this is a very un-scientific approach and frankly why should we believe that what he said happened really happened?
  - zyxley 11y ago
    Exactly. The lack of actual information is a reason to disbelieve Yudkowsky on this topic, not to assume that it's true for Secret Technophile Mystery Cult Reasons.
    - isolate 11y ago
      I think it's probably true and that the secrecy is more about cult building.
  - apsec112 11y ago
    "And yet Big Yud refuses" Please don't try to argue by name-calling. A strong argument is stronger without it. "DH0. Name-calling. This is the lowest form of disagreement, and probably also the most common. We've all seen comments like this: u r a fag!!!!!!!!!! But it's important to realize that more articulate name-calling has just as little weight. A comment like The author is a self-important dilettante. is really nothing more than a pretentious version of "u r a fag."" - http://www.paulgraham.com/disagree.html http://www.paulgraham.com/disagree.html
    - deleted 11y ago
      [deleted]

Chathamization 11y ago

But maybe it think about art a lot. Or maybe we can't cut out sleep without harming our ability to reason. Or maybe the AI would be more interested in watching youtube videos than working in the field of AI. The only human level general intelligence we have to look at is humans. Sure, it's possible that there's an easy way to build an artificial intelligence that doesn't need to sleep and isn't interested in art but that retains our ability to problem solve; but maybe their isn't. It's possible that we'll be able to easily figure out that alternative brain architecture and build in the near future; but given how little we still don't understand about our own brain (and the failure to mimic the brains of even simple creatures), it doesn't seem likely. It's possible that if we were able to overcome those major hurdles, a super John von Neumann would be able to wield enormous political power; or it's possible that, like the real von Neumann, it would focus mostly on its research. In order to get to the AI doomsday scenario, you have to assume that a lot of very unlikely things are all going to happen. And the main argument for them happening seems to be one of ignorance - "hey, we don't have any idea what this will be like, so we can't say that this improbable situation won't happen."

lmm 11y ago
I'd say your ideas are more unlikely. You can't generalize from one example. An AI that hasn't gone though evolution is extremely unlikely to have the exact set of complex wants and needs that humans have. And even if you're right, as Alexander says, in that case why the need to open-source it? If you think it's important that AI is open-source and not controlled by e.g. Facebook, that must be because you think AI is going to be powerful and effective. In which case it's as dangerous as he says.
- Chathamization 11y ago
  But it's not a generalization from one example; I'm giving a string of different possibilities, based on the fact that we only have a single point of data. People seem to readily except that particular scenario - the "AI as smart as John von Neumann on modafinil, that never thinks about food/sex/art/etc., that never sleeps, that has access to Wikipedia etc. at the speed of thought, and no morals" scenario - based on zero evidence that an AI would actually be like that. People seem more ready to accept the idea that an artificial general intelligence would act similar to it's portrayed in SciFi stories than accept that it might act similar to the other general intelligences we can observe (namely ourselves).