3 ms·

It lives in a computer. Can't we just turn it off?

Turing_Machine 12y ago
If it truly has human-level (or greater) intelligence, you're going to have some ethical issues that crop up with turning it off. Even if you're personally okay with it, there are going to be other people who aren't. Waiting until it commits a crime and then having some kind of "trial" and "execution" may not be an option. If it's smart/powerful enough, there may not be enough time for that.
- vinceguidry 12y ago
  That's not the problem. The problem is that when we are reliant on intelligent systems to, say, detect certain kinds of fraud in financial markets, then turning them off would have huge deleterious effects all on their own.
  - Turing_Machine 12y ago
    It's not a problem now. If we get machines anywhere near as smart as people, the AI rights organizations will soon follow.
    - vinceguidry 12y ago
      You mean like the people trying to give plants rights? Sure, they might spring up, doesn't mean they'll get anywhere. For better or worse, altruism will forever follow economics. The civil war didn't end the Atlantic slave trade, technology eventually made it largely uneconomical, allowing those wanting to end slavery the political ability to outlaw it. Even that didn't stop the slave trade. It took extra-special attention from the British Navy to actually enforce a slave trade ban on the open seas. 150 years later and black people are still getting the short end of the stick. By the time AI gets real rights, it will be long past the time where most of us believe they should get them.
      - Turing_Machine 12y ago
        It's not that simple. Slavery never took hold in much of Europe, for instance, even though it might've made economic sense (serfdom was a different thing, which had largely died out for complex reasons). I note in passing that slavery still exists today, and is not now, nor has it ever been, limited to black people as the victims. "By the time AI gets real rights, it will be long past the time where most of us believe they should get them." I'm not sure this is really relevant to the point at hand. We turn off "bad" AIs until "long past the time when people think they should have rights". What happens then?
davidrusu 12y ago
good luck: http://www.yudkowsky.net/singularity/aibox/ http://www.yudkowsky.net/singularity/aibox/
PeterisP 12y ago
Let's assume that the AI has the intelligence and capabilities that match or exceed a decent human teenage hacker. From historical evidence we can observe that such capabilities are enough to create a botnet, obtain significant money through fraud, hire employees and make shell companies, and rent (or obtain by hacking) large amounts of computing power on Amazon or other cloud services. All of that through a single low-bandwidth network connection and no physical interaction, assuming that it can figure out a single zero-day vulnerability in some common software. After that, you can't turn it off. Before that, you shouldn't assume that you can outsmart it in security - as you should assume that the adversary is smarter than you, and if your security works in 99 scenarios but fails in one... then it fails.
schoen 12y ago
It might take precautions against that, for example by compromising other computers, or renting other computers, or persuading people to give it access to other computers, or getting people to put some its infrastructure in space or under the ocean, or obfuscating where its infrastructure is, or hiring people to physically defend its infrastructure, or inventing computing systems that don't look like computing systems. Edit: or create what seem to be, or are, credible threats against people who are complicit in turning it off, or putting itself conspicuously into a role where transportation systems or economic markets or life support or ecosystems or medical progress seem dependent on some of its services.
jimbokun 12y ago
It's software. That's pretty much equivalent to "delete this picture from the Internet". First thing a super intelligent computer with a goal of survival would do would be to copy itself in sophisticated ways across computers all over the world.

If it truly has human-level (or greater) intelligence, you're going to have some ethical issues that crop up with turning it off. Even if you're personally okay with it, there are going to be other people who aren't. Waiting until it commits a crime and then having some kind of "trial" and "execution" may not be an option. If it's smart/powerful enough, there may not be enough time for that.

vinceguidry 12y ago
That's not the problem. The problem is that when we are reliant on intelligent systems to, say, detect certain kinds of fraud in financial markets, then turning them off would have huge deleterious effects all on their own.
- Turing_Machine 12y ago
  It's not a problem now. If we get machines anywhere near as smart as people, the AI rights organizations will soon follow.
  - vinceguidry 12y ago
    You mean like the people trying to give plants rights? Sure, they might spring up, doesn't mean they'll get anywhere. For better or worse, altruism will forever follow economics. The civil war didn't end the Atlantic slave trade, technology eventually made it largely uneconomical, allowing those wanting to end slavery the political ability to outlaw it. Even that didn't stop the slave trade. It took extra-special attention from the British Navy to actually enforce a slave trade ban on the open seas. 150 years later and black people are still getting the short end of the stick. By the time AI gets real rights, it will be long past the time where most of us believe they should get them.
    - Turing_Machine 12y ago
      It's not that simple. Slavery never took hold in much of Europe, for instance, even though it might've made economic sense (serfdom was a different thing, which had largely died out for complex reasons). I note in passing that slavery still exists today, and is not now, nor has it ever been, limited to black people as the victims. "By the time AI gets real rights, it will be long past the time where most of us believe they should get them." I'm not sure this is really relevant to the point at hand. We turn off "bad" AIs until "long past the time when people think they should have rights". What happens then?

davidrusu 12y ago

good luck: http://www.yudkowsky.net/singularity/aibox/ http://www.yudkowsky.net/singularity/aibox/

PeterisP 12y ago

Let's assume that the AI has the intelligence and capabilities that match or exceed a decent human teenage hacker. From historical evidence we can observe that such capabilities are enough to create a botnet, obtain significant money through fraud, hire employees and make shell companies, and rent (or obtain by hacking) large amounts of computing power on Amazon or other cloud services. All of that through a single low-bandwidth network connection and no physical interaction, assuming that it can figure out a single zero-day vulnerability in some common software. After that, you can't turn it off. Before that, you shouldn't assume that you can outsmart it in security - as you should assume that the adversary is smarter than you, and if your security works in 99 scenarios but fails in one... then it fails.

schoen 12y ago

It might take precautions against that, for example by compromising other computers, or renting other computers, or persuading people to give it access to other computers, or getting people to put some its infrastructure in space or under the ocean, or obfuscating where its infrastructure is, or hiring people to physically defend its infrastructure, or inventing computing systems that don't look like computing systems. Edit: or create what seem to be, or are, credible threats against people who are complicit in turning it off, or putting itself conspicuously into a role where transportation systems or economic markets or life support or ecosystems or medical progress seem dependent on some of its services.

jimbokun 12y ago

It's software. That's pretty much equivalent to "delete this picture from the Internet". First thing a super intelligent computer with a goal of survival would do would be to copy itself in sophisticated ways across computers all over the world.