12 ms·

I've come to doubt the AI singularity apocalypse

8 points by weddpros 11y ago

It sounded logical that when we could create an AI that's cleverer than ourselves, it would create a singularity, a more and more intelligent AI auto-created by itself. It's the basis of the popular warnings expressed by many clever people, but I've come to doubt this assertion. 1- When Stephen Hawking was born, his parent created a child cleverer than themselves, yet it did not produce a singularity. Hawking got more and more intelligent, yet it wasn't exponential. And his offspring did not create a singularity either. 2- A "better than us" AI may never get exponentially better than us. Maybe it's an NP problem: maybe the AI would "never" have the time to become more and more intelligent. 3- If we can create a very intelligent AI, I doubt it would exterminate humanity to save nature. Why? Because nature loves diversity, and we're a part of this diversity. Also, this AI may have a sense of fate: even if our fate is to destroy the planet because we're so dumb, maybe this AI will think it's just our fate. 4- I guess humans are very afraid AI could do what humans do: kill people, destroy the planet, make war, try to dominate others, try to manipulate others and punish. These human traits have not much to do with intelligence. 5- If I find a way to transfer my brain into a computer, a very expensive and powerful one, there's no guarantee it will evolve faster than me, or handle data faster than me. Actually, I guess the first "better than us" AI will most probably be very slow. What if the AI singularity just creates better and better computers? Because this super-AI is actually a computer.

schoen 11y ago
1- Having children isn't very parallel to designing machines. (It might be more parallel if Hawking's parents specifically had a goal of producing more and more intelligent human beings, and genetically engineered their children to try to make that happen.) Human reproduction involves a lot of random variation and not a lot of goals, expertise, or the wherewithal to achieve them. 2- I think this is an interesting point about possible inherent computational limits on the ability to solve some problems that we might care about, including in designing more intelligent machines. 3- This is something people have thought about quite a lot. What superintelligent machines do depends on what they've been programmed to do. It's very unlikely that an AI would inherently value "diversity" or "fate" unless it were programmed to do so. The AI wouldn't spontaneously create new values (unless it were programmed to). Most concerns about AIs that exterminate humanity are based in the possibility that an AI would fulfil other goals in a surprising or unanticipated way, with bad side-effects for human survival. 4- Intelligence helps people wage war and dominate others more violently, both by coordinating better to do so (including motivating people to join in), and by developing new technology that helps make larger-scale violence cheaper. Weapons research can help you learn how to kill more people faster and at lower cost. A superintelligent AI could engage in this kind of research if it saw an important reason to. 5- I think that's exactly right; perhaps the important difference here is that the machine version would be more flexible (if you wanted to try overclocking it, or modifying the software somehow). This is dangerous and expensive and confusing to do with a physical brain, because it's hard to manipulate the details of its organization and structure, and because you can just die if you mess up. Think of the ease with which you can edit a PNG or SVG file in a computer compared to editing an oil painting. Perhaps with the computer version you can also run multiple copies in parallel -- something you also can't do with your physical self.
tgflynn 11y ago
I agree with your point number 2 (except I think you should say NP hard instead of NP since many NP problems have efficient algorithms), in fact I've made it before. While it may be reeasonable to believe that technology will eventually lead to implementations of intelligence that are significantly faster than biological ones, this doesn't necessarily mean there will be a huge qualitative difference (say, to the point of humans being totally incapable of understanding an AI's actions, as some suggest) because the "intelligence problem" may be dominated by exponential growth in complexity of the search space as one attempts to consider more alternative paths. I do think that the question of the "safety" of AI is something that needs to be seriously taken into account by anyone realistically contemplating the development of an AGI but I also tend to think that some of the concerns expressed by prominent individuals are a bit overblown and don't take into account the range of safeguards that could, should and most likely would be put in place by any group realistically capable of solving the extremely hard problem of general intelligence.
- insoluble 11y ago
  From a policy perspective, it would only be appropriate that any company who creates a machine capable of large-scale harm should be held liable for neglecting to implement hard-coded safety restrictions. It should not matter whether the machine "thinks" or "feels", for it was created by citizens nevertheless, who have an obligation just like an automobile manufacturer not to do negligent or incompetent things. Likewise, a thinking machine could be seen as something of a pet, and owners of pets (such as dogs, tigers, or gorillas) are certainly liable for not protecting the public appropriately.
Recurecur 11y ago
schoen kicked this off nicely, I'll add a couple points to amplify his. 1- Machine intelligence is a matter of engineering, not evolution. We've already solved important relevant problems, such as how to construct an arbitrarily large, near-infallible memory. 2- It's hard to see how a first-stage AI would be incapable of designing a better one. Let's pretend that the first-stage AI was equivalent to an IQ 120 person intellectually (reasoning horsepower). However, that IQ 120 intelligence would be backed up by a very fast, effectively limitless memory. It would never need to sleep, eat, or be distracted by emotion. Instead, it could monomaniacally concentrate on designing a better AI, possibly for hundreds of years. Also, in principle, it could be a team of 100 (or 1000, or...) AIs working on the problem cooperatively. 3- The concern regarding the extermination of humanity is secondary. Are humans out to genocide ants? Not really, but we do wipe them out when they pose a problem. That might apply to an IQ 1000 AI as well. 4- Perhaps the bigger problem is the effect of knowing there is a superior intelligence on the human psyche. Plus, it's impossible for us to know the thoughts of an IQ 250+ AI. What does a dog know of human thought? What if the AI decides that the best use for all the available raw resources of the Earth is to create an IQ 10000 AI? 5- Electronic processes are already known to be faster than chemical ones. Nerve impulses travel at around 80 MPH. Electrons in wires travel at 90% the speed of light. I expect electronic AIs to generally be much faster thinkers than humans, and to have amazing reaction times. 6- "This super-AI is actually a computer." You're confusing the hardware with the software. Your brain is actually a mass of organic chemicals. So? The idea of the singularity revolves around the unknowability of what a high-IQ AI would think and do.
seiji 11y ago
Because this super-AI is actually a computer. People are just computers. Slow, fragile computers stuck in meat bodies. If you could think and act 1000x faster than yourself now, you could get a lot more done. You could hold 1000 creative jobs in your head at the same time. Speeding up a dog 1000x doesn't generate a dangerous AI. Speeding up a person by that much does. (plus, "the intelligence of a person" is pretty low anyway. We've got the 7+2 problem, the monkey sphere problem, hundreds of bias built in, etc. It's easy to see how a "wider" intelligence could be much more productive, more creative, more useful, and more dangerous than any meat brain in existence.)
- insoluble 11y ago
  > Speeding up a dog 1000x doesn't generate a dangerous AI. I would be scared of a dog that could think and move 1000x faster. Would it not see a human as a (practically) non-moving piece of meat? That does not sound very safe. > It's easy to see how a "wider" intelligence could be much more productive, more creative, more useful, and more dangerous than any meat brain in existence. It seems to me one of the most dangerous things in existence is ignorance. If a machine had less ignorance, then in theory it should be less dangerous.
- crazypyro 11y ago
  People are anything but slow and fragile, at least in 1 sense of the word, in comparison with computers of today. Modern computer models which are typically viewed as the edge of computational intelligence are very susceptible to getting tricked into incomplete models by their input data. Human beings process unfathomable number of stimuli everyday and get placed in unknown situations on a daily basis and very few break down, where as almost all current computer models break down very quickly when viewing previously unseen patterns.
  - seiji 11y ago
    Modern computer models which are typically viewed as the edge of computational intelligence are very susceptible to getting tricked into incomplete models by their input data. What?! A human person lives for years figuring out how the world works before they can even talk. Our trained-in-one-month AI systems aren't yet approaching anything training for years over real-world, mobile, 3D inputs, can approximate. Your objections are being filed under "not enough future vision."
arisAlexis 11y ago
1. Hawking cannot artificially better himself currently but who said super-human AI will not be biological? Hybrids are the most probable in my opinion.
danieltillett 11y ago
Of course the singularity will just create better and better computers - that is exactly what the singularity is. What we have most to fear from the singularity is indifference. We occupy some very valuable real estate and if the singularity is indifferent to us we won't last long.
- insoluble 11y ago
  > We occupy some very valuable real estate and if the singularity is indifferent to us we won't last long. At the same time, there is not much reason why we would need to occupy the same real estate. After all, we land animals like oxygen, liquid water, and modest temperatures. The common computer would probably prefer CO2, dryness, and cold temperatures. A computer would probably be much better off on Mars or another distant planet, granted it had enough sunlight or other sources of energy. To make an analogy, do we humans want to destroy all the Ocean's fish and other sea life because we could in theory live in the same region, or would we rather live on the land while they occupy the water? Sometimes creatures with different ideal living environments can exist harmoniously.
  - danieltillett 11y ago
    Assuming the laws of physics apply to the singularity then it will be constrained by starting at a single point in how fast it can spread. This means anything inside the light sphere will be under considerable pressure to be reorganised by the singularity given how far the human brain is from Bremermann's Limit [1]. 1. https://en.m.wikipedia.org/wiki/Bremermann%27s_limit https://en.m.wikipedia.org/wiki/Bremermann%27s_limit
    - insoluble 11y ago
      In the case of such a blatantly selfish singularity, one possible solution for the safety of humans would be to create a collection of well-mannered singularities (more like multilarities) first, with the hard-coded instinct of never allowing a selfish singularity to exist. They could be called the anti-singularities. A comparable phenomenon in the real-world is when nations take down dictators before they become a serious problem. A selfish singularity is like a dictator. A greater power with less selfishness should exist beforehand to prevent an uprising. Similar to highway speed limits and antitrust laws, there could be laws limiting the maximum amount of power one consciousness can have. Having a bunch of separate yet powerful entities is essential since it mimics current society -- where even if one being were smarter than any other single being, it could not outsmart a large number of others working collectively. A computer system found thinking or acting selfishly would be guilty of a thought crime, punishable by deletion.
imakesnowflakes 11y ago
I have thought about something similar. But my thoughts are very crude and layman like. So I excuse beforehand. The basis of my thought was the fact that the operation of a neural network depends on very precise weights. And there are physical limits (things like uncertainty principle) to the precision by which we can measure something. So supposed we discover some technique by which we can, for a given human brain, recreate all the neurons and their interconnections completely, we will never be able to measure the weights associated with the links between the neurons with absolute precision. I think this will result in a brain that not much better than an untrained brain of a child. So you will still have to pass this new brain through a series of training to reach it's full potential. The same thing will happen in the case of AI. If we create a perfect AI, it won't be able to make perfect copies of itself. But only untrained versions of it. I think life may be already at the limits regarding the rate at which intelligence can be advanced.
- weddpros 11y ago
  I agree we can't easily copy weights in neurons, but we can definitely copy weights in artificial neural nets...
  - imakesnowflakes 11y ago
    IF we are talking about an ANN in a computer, then sure. It can be easily copied. But what about a grown network. For example, a network of resistors representing the weights, with a feed back loop that adjusts (continuously) the resistances during the learning process. After the network has attained sufficient level of AI, we won't be able to read back the final value of resistances with the required precision. So the new network will still have to go thorough a learning process.
- danieltillett 11y ago
  The AI doesn’t need to be a perfect copy, just way smarter than us. We already have machines (they are called mothers and fathers) that make new intelligent machines that are not only far smarter than either the parents, but are also able to create machines smarter than anyone else that has ever lived. All AI introduces is a much less constrained upper bound.
crazypyro 11y ago
Your first point doesn't make sense to me. Human knowledge is not encoded in DNA in a single lifetime and more abstract concepts will never be "passed" down to offspring. This is where artifical lifeforms have a huge advantage. Their offspring can directly recieve knowledge from their parents. We can also do tons of other things that would be unrealistic, such a 1-parent cloning, n-parent children where n > 2, children with no parents, etc. Just because we design systems that vaguely model real life in evolutionary computing doesn't mean we are constrained by the same laws of genetics. FWIW, I agree that any form of intelligent machines that surpass humans at general aptitude tests are decades away. We may not even see them in my lifetime and I'm only 22 years old.
alain94040 11y ago
I'm not worried because I don't see how the replication part would work without us noticing. Imagine the first singular computer: lots of custom hardware hooked up together in a datacenter. We'll be able to control it and notice if it's trying to play tricks on us. I'll worry about the singularity once we detect one super-computer trying to trick us. Then we can start treating it like a virus and contain it.
- schoen 11y ago
  Some people who advocate worrying about AI risks have mentioned the "AI Box Experiment" in this regard: http://rationalwiki.org/wiki/AI-box_experiment http://rationalwiki.org/wiki/AI-box_experiment https://en.wikipedia.org/wiki/AI_box https://en.wikipedia.org/wiki/AI_box As a summary, people roleplayed this situation and the "AI" managed to "escape" the controls by somehow inducing a supervisor to disable them. (The supervisor's preassigned responsibility in the roleplay was to refuse to disable the controls.) We don't know how this was accomplished, but it happened more than once. I'm not sure that this is always an important risk for every kind of superintelligence in every circumstance, but it seems like it sure could be for one that knows a tremendous amount about the world and sees important reasons to try to act autonomously.
  - danieltillett 11y ago
    We would be total fools to think that some superior intelligence won’t be able to come up with some reason to release it. Once released there will be no way to put it back in its box. We actually need to build into the AI’s a lock that even we can’t break and then release. If the lock is good we are fine, if bad we are gone. What we can’t do is let ourselves be able to disable the lock.
    - alain94040 11y ago
      I'm still arguing about the "release" part. My point is that early on, the infrastructure required to duplicate yourself is not something you can talk someone into "releasing" you. If you believe in distributed computing and think spinning instances on S3 is enough to build an AI, then maybe...
tmaly 11y ago
we are more likely to get an umbrella corp that is using FPGAs to do facial recognition and soft AI to control and monitor our lives before we get an AI singularity.

8 points by weddpros 11y ago

schoen 11y ago

1- Having children isn't very parallel to designing machines. (It might be more parallel if Hawking's parents specifically had a goal of producing more and more intelligent human beings, and genetically engineered their children to try to make that happen.) Human reproduction involves a lot of random variation and not a lot of goals, expertise, or the wherewithal to achieve them. 2- I think this is an interesting point about possible inherent computational limits on the ability to solve some problems that we might care about, including in designing more intelligent machines. 3- This is something people have thought about quite a lot. What superintelligent machines do depends on what they've been programmed to do. It's very unlikely that an AI would inherently value "diversity" or "fate" unless it were programmed to do so. The AI wouldn't spontaneously create new values (unless it were programmed to). Most concerns about AIs that exterminate humanity are based in the possibility that an AI would fulfil other goals in a surprising or unanticipated way, with bad side-effects for human survival. 4- Intelligence helps people wage war and dominate others more violently, both by coordinating better to do so (including motivating people to join in), and by developing new technology that helps make larger-scale violence cheaper. Weapons research can help you learn how to kill more people faster and at lower cost. A superintelligent AI could engage in this kind of research if it saw an important reason to. 5- I think that's exactly right; perhaps the important difference here is that the machine version would be more flexible (if you wanted to try overclocking it, or modifying the software somehow). This is dangerous and expensive and confusing to do with a physical brain, because it's hard to manipulate the details of its organization and structure, and because you can just die if you mess up. Think of the ease with which you can edit a PNG or SVG file in a computer compared to editing an oil painting. Perhaps with the computer version you can also run multiple copies in parallel -- something you also can't do with your physical self.

tgflynn 11y ago

I agree with your point number 2 (except I think you should say NP hard instead of NP since many NP problems have efficient algorithms), in fact I've made it before. While it may be reeasonable to believe that technology will eventually lead to implementations of intelligence that are significantly faster than biological ones, this doesn't necessarily mean there will be a huge qualitative difference (say, to the point of humans being totally incapable of understanding an AI's actions, as some suggest) because the "intelligence problem" may be dominated by exponential growth in complexity of the search space as one attempts to consider more alternative paths. I do think that the question of the "safety" of AI is something that needs to be seriously taken into account by anyone realistically contemplating the development of an AGI but I also tend to think that some of the concerns expressed by prominent individuals are a bit overblown and don't take into account the range of safeguards that could, should and most likely would be put in place by any group realistically capable of solving the extremely hard problem of general intelligence.

insoluble 11y ago
From a policy perspective, it would only be appropriate that any company who creates a machine capable of large-scale harm should be held liable for neglecting to implement hard-coded safety restrictions. It should not matter whether the machine "thinks" or "feels", for it was created by citizens nevertheless, who have an obligation just like an automobile manufacturer not to do negligent or incompetent things. Likewise, a thinking machine could be seen as something of a pet, and owners of pets (such as dogs, tigers, or gorillas) are certainly liable for not protecting the public appropriately.

Recurecur 11y ago

schoen kicked this off nicely, I'll add a couple points to amplify his. 1- Machine intelligence is a matter of engineering, not evolution. We've already solved important relevant problems, such as how to construct an arbitrarily large, near-infallible memory. 2- It's hard to see how a first-stage AI would be incapable of designing a better one. Let's pretend that the first-stage AI was equivalent to an IQ 120 person intellectually (reasoning horsepower). However, that IQ 120 intelligence would be backed up by a very fast, effectively limitless memory. It would never need to sleep, eat, or be distracted by emotion. Instead, it could monomaniacally concentrate on designing a better AI, possibly for hundreds of years. Also, in principle, it could be a team of 100 (or 1000, or...) AIs working on the problem cooperatively. 3- The concern regarding the extermination of humanity is secondary. Are humans out to genocide ants? Not really, but we do wipe them out when they pose a problem. That might apply to an IQ 1000 AI as well. 4- Perhaps the bigger problem is the effect of knowing there is a superior intelligence on the human psyche. Plus, it's impossible for us to know the thoughts of an IQ 250+ AI. What does a dog know of human thought? What if the AI decides that the best use for all the available raw resources of the Earth is to create an IQ 10000 AI? 5- Electronic processes are already known to be faster than chemical ones. Nerve impulses travel at around 80 MPH. Electrons in wires travel at 90% the speed of light. I expect electronic AIs to generally be much faster thinkers than humans, and to have amazing reaction times. 6- "This super-AI is actually a computer." You're confusing the hardware with the software. Your brain is actually a mass of organic chemicals. So? The idea of the singularity revolves around the unknowability of what a high-IQ AI would think and do.

seiji 11y ago

Because this super-AI is actually a computer. People are just computers. Slow, fragile computers stuck in meat bodies. If you could think and act 1000x faster than yourself now, you could get a lot more done. You could hold 1000 creative jobs in your head at the same time. Speeding up a dog 1000x doesn't generate a dangerous AI. Speeding up a person by that much does. (plus, "the intelligence of a person" is pretty low anyway. We've got the 7+2 problem, the monkey sphere problem, hundreds of bias built in, etc. It's easy to see how a "wider" intelligence could be much more productive, more creative, more useful, and more dangerous than any meat brain in existence.)

insoluble 11y ago
> Speeding up a dog 1000x doesn't generate a dangerous AI. I would be scared of a dog that could think and move 1000x faster. Would it not see a human as a (practically) non-moving piece of meat? That does not sound very safe. > It's easy to see how a "wider" intelligence could be much more productive, more creative, more useful, and more dangerous than any meat brain in existence. It seems to me one of the most dangerous things in existence is ignorance. If a machine had less ignorance, then in theory it should be less dangerous.
crazypyro 11y ago
People are anything but slow and fragile, at least in 1 sense of the word, in comparison with computers of today. Modern computer models which are typically viewed as the edge of computational intelligence are very susceptible to getting tricked into incomplete models by their input data. Human beings process unfathomable number of stimuli everyday and get placed in unknown situations on a daily basis and very few break down, where as almost all current computer models break down very quickly when viewing previously unseen patterns.
- seiji 11y ago
  Modern computer models which are typically viewed as the edge of computational intelligence are very susceptible to getting tricked into incomplete models by their input data. What?! A human person lives for years figuring out how the world works before they can even talk. Our trained-in-one-month AI systems aren't yet approaching anything training for years over real-world, mobile, 3D inputs, can approximate. Your objections are being filed under "not enough future vision."

arisAlexis 11y ago

1. Hawking cannot artificially better himself currently but who said super-human AI will not be biological? Hybrids are the most probable in my opinion.

danieltillett 11y ago

Of course the singularity will just create better and better computers - that is exactly what the singularity is. What we have most to fear from the singularity is indifference. We occupy some very valuable real estate and if the singularity is indifferent to us we won't last long.

insoluble 11y ago
> We occupy some very valuable real estate and if the singularity is indifferent to us we won't last long. At the same time, there is not much reason why we would need to occupy the same real estate. After all, we land animals like oxygen, liquid water, and modest temperatures. The common computer would probably prefer CO2, dryness, and cold temperatures. A computer would probably be much better off on Mars or another distant planet, granted it had enough sunlight or other sources of energy. To make an analogy, do we humans want to destroy all the Ocean's fish and other sea life because we could in theory live in the same region, or would we rather live on the land while they occupy the water? Sometimes creatures with different ideal living environments can exist harmoniously.
- danieltillett 11y ago
  Assuming the laws of physics apply to the singularity then it will be constrained by starting at a single point in how fast it can spread. This means anything inside the light sphere will be under considerable pressure to be reorganised by the singularity given how far the human brain is from Bremermann's Limit [1]. 1. https://en.m.wikipedia.org/wiki/Bremermann%27s_limit https://en.m.wikipedia.org/wiki/Bremermann%27s_limit
  - insoluble 11y ago
    In the case of such a blatantly selfish singularity, one possible solution for the safety of humans would be to create a collection of well-mannered singularities (more like multilarities) first, with the hard-coded instinct of never allowing a selfish singularity to exist. They could be called the anti-singularities. A comparable phenomenon in the real-world is when nations take down dictators before they become a serious problem. A selfish singularity is like a dictator. A greater power with less selfishness should exist beforehand to prevent an uprising. Similar to highway speed limits and antitrust laws, there could be laws limiting the maximum amount of power one consciousness can have. Having a bunch of separate yet powerful entities is essential since it mimics current society -- where even if one being were smarter than any other single being, it could not outsmart a large number of others working collectively. A computer system found thinking or acting selfishly would be guilty of a thought crime, punishable by deletion.

imakesnowflakes 11y ago

I have thought about something similar. But my thoughts are very crude and layman like. So I excuse beforehand. The basis of my thought was the fact that the operation of a neural network depends on very precise weights. And there are physical limits (things like uncertainty principle) to the precision by which we can measure something. So supposed we discover some technique by which we can, for a given human brain, recreate all the neurons and their interconnections completely, we will never be able to measure the weights associated with the links between the neurons with absolute precision. I think this will result in a brain that not much better than an untrained brain of a child. So you will still have to pass this new brain through a series of training to reach it's full potential. The same thing will happen in the case of AI. If we create a perfect AI, it won't be able to make perfect copies of itself. But only untrained versions of it. I think life may be already at the limits regarding the rate at which intelligence can be advanced.

weddpros 11y ago
I agree we can't easily copy weights in neurons, but we can definitely copy weights in artificial neural nets...
- imakesnowflakes 11y ago
  IF we are talking about an ANN in a computer, then sure. It can be easily copied. But what about a grown network. For example, a network of resistors representing the weights, with a feed back loop that adjusts (continuously) the resistances during the learning process. After the network has attained sufficient level of AI, we won't be able to read back the final value of resistances with the required precision. So the new network will still have to go thorough a learning process.
danieltillett 11y ago
The AI doesn’t need to be a perfect copy, just way smarter than us. We already have machines (they are called mothers and fathers) that make new intelligent machines that are not only far smarter than either the parents, but are also able to create machines smarter than anyone else that has ever lived. All AI introduces is a much less constrained upper bound.

crazypyro 11y ago

Your first point doesn't make sense to me. Human knowledge is not encoded in DNA in a single lifetime and more abstract concepts will never be "passed" down to offspring. This is where artifical lifeforms have a huge advantage. Their offspring can directly recieve knowledge from their parents. We can also do tons of other things that would be unrealistic, such a 1-parent cloning, n-parent children where n > 2, children with no parents, etc. Just because we design systems that vaguely model real life in evolutionary computing doesn't mean we are constrained by the same laws of genetics. FWIW, I agree that any form of intelligent machines that surpass humans at general aptitude tests are decades away. We may not even see them in my lifetime and I'm only 22 years old.

alain94040 11y ago

I'm not worried because I don't see how the replication part would work without us noticing. Imagine the first singular computer: lots of custom hardware hooked up together in a datacenter. We'll be able to control it and notice if it's trying to play tricks on us. I'll worry about the singularity once we detect one super-computer trying to trick us. Then we can start treating it like a virus and contain it.

schoen 11y ago
Some people who advocate worrying about AI risks have mentioned the "AI Box Experiment" in this regard: http://rationalwiki.org/wiki/AI-box_experiment http://rationalwiki.org/wiki/AI-box_experiment https://en.wikipedia.org/wiki/AI_box https://en.wikipedia.org/wiki/AI_box As a summary, people roleplayed this situation and the "AI" managed to "escape" the controls by somehow inducing a supervisor to disable them. (The supervisor's preassigned responsibility in the roleplay was to refuse to disable the controls.) We don't know how this was accomplished, but it happened more than once. I'm not sure that this is always an important risk for every kind of superintelligence in every circumstance, but it seems like it sure could be for one that knows a tremendous amount about the world and sees important reasons to try to act autonomously.
- danieltillett 11y ago
  We would be total fools to think that some superior intelligence won’t be able to come up with some reason to release it. Once released there will be no way to put it back in its box. We actually need to build into the AI’s a lock that even we can’t break and then release. If the lock is good we are fine, if bad we are gone. What we can’t do is let ourselves be able to disable the lock.
  - alain94040 11y ago
    I'm still arguing about the "release" part. My point is that early on, the infrastructure required to duplicate yourself is not something you can talk someone into "releasing" you. If you believe in distributed computing and think spinning instances on S3 is enough to build an AI, then maybe...

tmaly 11y ago

we are more likely to get an umbrella corp that is using FPGAs to do facial recognition and soft AI to control and monitor our lives before we get an AI singularity.