5 ms·

You say things like "google scares me", and "kurzweil ... isn't exactly comforting". Is this a bad thing?

by CodeCube 14y ago

karamazov 14y ago
It's obviously bad from his perspective.
- taligent 14y ago
  That may just be the aluminum from his tinfoil hat seeping into his brain.
- ericb 14y ago
  I think his perspective is entirely reasonable. Take one small example: http://wiki.lesswrong.com/wiki/Paperclip_maximizer http://wiki.lesswrong.com/wiki/Paperclip_maximizer
  - andrewflnr 14y ago
    That assumes that the maximizer has the basic abilities and tools to modify its intelligence and/or launch a paper-clip manufacturing campaign. If you just hook up its outputs to a web server, it's going to take a mighty impressive AI to destroy the world.
    - Jach 14y ago
      Oh the AI-in-a-box argument... Ever read this, by chance? http://yudkowsky.net/singularity/aibox http://yudkowsky.net/singularity/aibox
      - ericb 14y ago
        Yes, and the results are not encouraging. Frankly, even a 0.05% rate of box-release per incident would terrify me. Expanded over a long enough timeframe and enough attempts, 0.05% (or any easily type-able number) becomes 100% release rate.
    - TeMPOraL 14y ago
      > If you just hook up its outputs to a web server, it's going to take a mighty impressive AI to destroy the world. All it would take is for the AI to just be smarter than you. Human psychology is hackable too. Cf. AI box experiment.
    - ericb 14y ago
      Intelligence creates tools--that is its nature. Modifying intelligence is easy--just fork a process--now you have two minds working on the problem. I'm human and I could hack into other machines to gain their processing power. Even if you don't put it on the web, it could probably talk you or someone in the lab to do so by promising stock market riches, or to cure the janitor's kid of cancer. All of these things would be behaviors that further the paperclip goal.
      - andrewflnr 14y ago
        You can't directly modify your intelligence. You can only do so indirectly using tools made possible by the physics in which you are embedded. Why assume that the framework around the AI is more permissive? Forking a process, initially anyway, only means that two identical minds are working on the problem at half-speed. And I was, in fact, considering "AI in a box"-style scenarios. My point remains: it would be a very impressive AI that could break out. Presumably it at least needs some significant basic resources, memory at least, and CPU to do it in reasonable time, to do deep human-hacking. How can it convince us to give them to it if it doesn't have them to begin with? If we're assuming we can control its goals at all (if we can't we're already doomed by virtue of trying), will it even occur to the thing to act beyond giving direct instructions to humans? Will it have enough concept of self to conceive of escaping, to say "I think therefore I am"? Can it even realize what kinds of computational resources it needs and how to ask for them? There may be good answers to these questions, but I don't believe they're trivial.

You say things like "google scares me", and "kurzweil ... isn't exactly comforting". Is this a bad thing?

karamazov 14y ago

It's obviously bad from his perspective.

taligent 14y ago
That may just be the aluminum from his tinfoil hat seeping into his brain.
ericb 14y ago
I think his perspective is entirely reasonable. Take one small example: http://wiki.lesswrong.com/wiki/Paperclip_maximizer http://wiki.lesswrong.com/wiki/Paperclip_maximizer
- andrewflnr 14y ago
  That assumes that the maximizer has the basic abilities and tools to modify its intelligence and/or launch a paper-clip manufacturing campaign. If you just hook up its outputs to a web server, it's going to take a mighty impressive AI to destroy the world.
  - Jach 14y ago
    Oh the AI-in-a-box argument... Ever read this, by chance? http://yudkowsky.net/singularity/aibox http://yudkowsky.net/singularity/aibox
    - ericb 14y ago
      Yes, and the results are not encouraging. Frankly, even a 0.05% rate of box-release per incident would terrify me. Expanded over a long enough timeframe and enough attempts, 0.05% (or any easily type-able number) becomes 100% release rate.
  - TeMPOraL 14y ago
    > If you just hook up its outputs to a web server, it's going to take a mighty impressive AI to destroy the world. All it would take is for the AI to just be smarter than you. Human psychology is hackable too. Cf. AI box experiment.
  - ericb 14y ago
    Intelligence creates tools--that is its nature. Modifying intelligence is easy--just fork a process--now you have two minds working on the problem. I'm human and I could hack into other machines to gain their processing power. Even if you don't put it on the web, it could probably talk you or someone in the lab to do so by promising stock market riches, or to cure the janitor's kid of cancer. All of these things would be behaviors that further the paperclip goal.
    - andrewflnr 14y ago
      You can't directly modify your intelligence. You can only do so indirectly using tools made possible by the physics in which you are embedded. Why assume that the framework around the AI is more permissive? Forking a process, initially anyway, only means that two identical minds are working on the problem at half-speed. And I was, in fact, considering "AI in a box"-style scenarios. My point remains: it would be a very impressive AI that could break out. Presumably it at least needs some significant basic resources, memory at least, and CPU to do it in reasonable time, to do deep human-hacking. How can it convince us to give them to it if it doesn't have them to begin with? If we're assuming we can control its goals at all (if we can't we're already doomed by virtue of trying), will it even occur to the thing to act beyond giving direct instructions to humans? Will it have enough concept of self to conceive of escaping, to say "I think therefore I am"? Can it even realize what kinds of computational resources it needs and how to ask for them? There may be good answers to these questions, but I don't believe they're trivial.