3 ms·

I'm not worried because I don't see how the replication part would work without us noticing. Imagine the first singular computer: lots of custom hardware hooke

I'm not worried because I don't see how the replication part would work without us noticing. Imagine the first singular computer: lots of custom hardware hooked up together in a datacenter. We'll be able to control it and notice if it's trying to play tricks on us. I'll worry about the singularity once we detect one super-computer trying to trick us. Then we can start treating it like a virus and contain it.

schoen 11y ago
Some people who advocate worrying about AI risks have mentioned the "AI Box Experiment" in this regard: http://rationalwiki.org/wiki/AI-box_experiment http://rationalwiki.org/wiki/AI-box_experiment https://en.wikipedia.org/wiki/AI_box https://en.wikipedia.org/wiki/AI_box As a summary, people roleplayed this situation and the "AI" managed to "escape" the controls by somehow inducing a supervisor to disable them. (The supervisor's preassigned responsibility in the roleplay was to refuse to disable the controls.) We don't know how this was accomplished, but it happened more than once. I'm not sure that this is always an important risk for every kind of superintelligence in every circumstance, but it seems like it sure could be for one that knows a tremendous amount about the world and sees important reasons to try to act autonomously.
- danieltillett 11y ago
  We would be total fools to think that some superior intelligence won’t be able to come up with some reason to release it. Once released there will be no way to put it back in its box. We actually need to build into the AI’s a lock that even we can’t break and then release. If the lock is good we are fine, if bad we are gone. What we can’t do is let ourselves be able to disable the lock.
  - alain94040 11y ago
    I'm still arguing about the "release" part. My point is that early on, the infrastructure required to duplicate yourself is not something you can talk someone into "releasing" you. If you believe in distributed computing and think spinning instances on S3 is enough to build an AI, then maybe...
    - danieltillett 11y ago
      The level of AI we are going to need to duplicate a person is going to be far beyond human level intelligence. Even if it wasn’t, there is always the risk that we underestimate the AI and it is smarter than we think and it gets out that way.

schoen 11y ago

Some people who advocate worrying about AI risks have mentioned the "AI Box Experiment" in this regard: http://rationalwiki.org/wiki/AI-box_experiment http://rationalwiki.org/wiki/AI-box_experiment https://en.wikipedia.org/wiki/AI_box https://en.wikipedia.org/wiki/AI_box As a summary, people roleplayed this situation and the "AI" managed to "escape" the controls by somehow inducing a supervisor to disable them. (The supervisor's preassigned responsibility in the roleplay was to refuse to disable the controls.) We don't know how this was accomplished, but it happened more than once. I'm not sure that this is always an important risk for every kind of superintelligence in every circumstance, but it seems like it sure could be for one that knows a tremendous amount about the world and sees important reasons to try to act autonomously.

danieltillett 11y ago
We would be total fools to think that some superior intelligence won’t be able to come up with some reason to release it. Once released there will be no way to put it back in its box. We actually need to build into the AI’s a lock that even we can’t break and then release. If the lock is good we are fine, if bad we are gone. What we can’t do is let ourselves be able to disable the lock.
- alain94040 11y ago
  I'm still arguing about the "release" part. My point is that early on, the infrastructure required to duplicate yourself is not something you can talk someone into "releasing" you. If you believe in distributed computing and think spinning instances on S3 is enough to build an AI, then maybe...
  - danieltillett 11y ago
    The level of AI we are going to need to duplicate a person is going to be far beyond human level intelligence. Even if it wasn’t, there is always the risk that we underestimate the AI and it is smarter than we think and it gets out that way.