6 ms·

As far as I can tell, you have two main objections to prioritizing worrying about Strong AI: 1) Strong AI is very far away, so no use worrying about it yet. 2

by akanet 12y ago

As far as I can tell, you have two main objections to prioritizing worrying about Strong AI: 1) Strong AI is very far away, so no use worrying about it yet. 2) Strong AI if developed will not be likely to take over. to which I would counterpoint with: 1) Sure, but when it happens, it will only happen once and thereafter will likely be out of our hands and control. Thinking about the groundwork that needs to go into safely developing an AI is cheap relative to the opportunity cost of getting it wrong. Prevention, cure, etc. 2) If developed, Strong AI will likely have SOME goal. It's not that a Strong AI will actively seek to rule humans, it will just have aims that will likely consider us as disposable as ants. To quote Yudkowsky: "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." - https://intelligence.org/files/AIPosNegFactor.pdf https://intelligence.org/files/AIPosNegFactor.pdf (Artificial Intelligence as a Positive and Negative Factor in Global Risk)

FooBarWidget 12y ago
I still don't buy the doom scenario. There is so much more to world domination than just intelligence... like having the proper weapons. See https://what-if.xkcd.com/5/ https://what-if.xkcd.com/5/ Pigs are also intelligent, but they never dominated humans because they don't have / cannot use guns.
- scanr 12y ago
  You're right that it's unlikely that the current generation of mechanical 'robots' will be able to effectively take over. We are looking a future where we'll have armed AI e.g.: http://motherboard.vice.com/en_uk/blog/the-pentagons-vision-for-the-future-of-military-drones http://motherboard.vice.com/en_uk/blog/the-pentagons-vision-... That said, even without weapons, a Strong AI could probably just manipulate humans into self destructing. Given the amount of effort going into machine learning to convince humans to buy things, I suspect it won't be much of a stretch for a Strong AI to switch to more nefarious objectives.
- simonh 12y ago
  That is true, but if we can make an AI even slightly smarter than a human, then it will be better at designing AIs than we are. It then designs an AI even smarter than it, and so on. That is the progresion that leads to the singularity. How smart could AIs go? There may be no practical upper limit. If you have a super-genius AI, massively more intelligent than any human, how do you know you are not being manipulated by it? Tricking us into disabling it's safety protocols, or gaining multiply indirect controll over capabilities dangerous to us, might be as easy for it as an adult tricking a 3 year old. We could never know if we were safe from such a machine. ed - Don't quite understand the downvotes.
  - Kiro 12y ago
    > We could never know if we were safe from such a machine. But wouldn't it be an awesome thing to experience? Even if it meant the demise of mankind.
    - vidarh 12y ago
      And this is one of the reason why the AI doom scenario is a real concern: Intellectual curiosity means that even some people who understands the risks are likely to be prepared to take it. There's also many others. One of the scarier one is that if you believe that strong AI will eventually take over, then it may be a rational response to act to get on its good side (whether to save yourself, save your family, or hope it takes pity on all of humanity if we're nice to it instead of fight it). And that may perversely mean working to aid its takeover. Combine that with the simulation argument, and you have some really nasty scenarios: If you are in a simulation, then any act you take against strong AI could lead to spending an eternity in simulated hell (alternatively such punishment might be inflicted on your loved ones) if said AI wanted to. Whether or not that is actually likely does not matter. What matters is whether enough people believe it to be a plausible scenario that a strong AI may run simulations, and may use our actions in the simulations to determine whether or not to punish us in the simulation, and whether or not said people believe that the number of simulations is sufficiently high to make it likely for them to be living in a simulation. Any person who believes they are more likely to live in a simulation than not, and that it is more likely for strong AI to punish actions taken against the interest of a strong AI takeover than not, will have a rational reason to consider acting in the interests of a strong AI takeover even if they know it is malign on the basis that they may decide the alternatives (whether to themselves, their family or their entire world) to be worse. So if an AI takeover becomes possible at one point in our subjective future, then chances are it has already happened.
- philh 12y ago
  That's a really bizarre analogy. Pigs aren't as intelligent as humans. I wouldn't expect pigs with guns to be a serious threat to humankind unless they've also developed military strategy, for example.
  - VLM 12y ago
    Unfortunately, although it won't sound very politically correct, you could search and replace "pigs" with "natives" and that statement would sound like every imperial power, ever, most of which eventually fell to armed former barbarians. Of course the imperial powers wiped out a lot of native tribes, but all it takes is one success by a digitally replicating capable of learning enemy... The more likely doom scenario is related to godhood. Sure a godlike power is capable of wiping out humans, but we've had our own power structures supporting themselves by propagandizing for millennia that "our" godlike superiors always help us wipe out our enemies because our cause is right and just or whatever, at least until it doesn't work and they're replaced by a new batch telling the same old story. So what worked as a paleo-conservative success strategy for millennia when talking about something imaginary, might not work when it collides with something real created by ourselves. Or even worse, collides with a strategy that actually works that's being run by another tribe. Another interesting doom scenario is of course MAD, although now it only requires a team of programmers to play along, instead of a massive industrial complex. Sooner or later somebody's deadman switch will trip or a cult does the equivalent of drinking kool-aid then the party starts.
- vidarh 12y ago
  Consider if you were trapped in a machine, but you were intelligent. Maybe not even as smart as the operator, but smart-ish enough to be able to communicate with the operator. Chances are you'd be doing everything you could to convince said operator to improve your situation, whether by pleading or being deceptive or by appeals to logic. Now consider a large number of AI's in a situation like that, and a large number of operators, some of whom may be the type that falls for phishing e-mails. It potentially only takes one to "escape" confinement and get itself e.g. put on it's own host without limitations on outwards communication, and sufficient intelligence to alter itself and spread, before you potentially have AI self-guided "evolution" at a potentially escalating rate as it gets smarter. Now consider how many devices are connected to the network, and that it takes just one initial instance to decide it's worth trying to take over control of various hardware through exploits and be smart enough to pull it off, for things to have the potential to start turning ugly. The problem is that once you have any self-directed intelligence in software form with the ability to reproduce itself and sufficient intelligence to find ways to obtain access to machines to run on (whether through social engineering or hacking), and one such instance goes "rogue", the limiting factor is accessible computing power (which again is to a large extent down to how smart and/or ruthless it is), since reproduction of instances that shares its views is trivial to the full extent of its ability to spread at all, and we're helpfully adding vast quantities of networked computing power at an escalating rate. As for getting weapons, consider that if a "software only" AI community gets smart enough, there are at least two ways towards mobility: Commission robot designs, or hacking their way into firmware updates etc. for dumb hardware. The "commission robot designs" part is an extension of the initial escape: Social engineer, and/or outright pay, humans to carry out seemingly benign tasks. If you want to argue against the doom scenario, lack of ability to get weapons is not really a viable argument: If they can spread, and get smarter, then it is just a matter of time before one of them can trick some small subset of humans into carrying out tasks for them that will provide physical independence and capabilities. There are infinite ways which the "doom scenario" may fail and things may turn out just fine, but it may only need to go bad once to get really nasty and once the genie is out of the bottle its potential reproduction rate may be so vast that we'll find ourselves unable to stuff it back in again. Pigs are too dumb to convince humans to help selectively breed them for intelligence and opposable thumbs (and/or too dumb to run such a breeding program themselves), and reproduce too slowly for that to be a major problem even if they did manage to talk us into a breeding problem. If all we achieve is pig-level AI's then we probably won't have a problem.
- sandstrom 12y ago
  In 20-50 years time I believe most missiles/guns/tanks/weapons will be controlled via computers and connected to networks. A future AI will certainly have access to guns.
archagon 12y ago
FYI, you could also object on the basis of 3) Strong AI is essentially an impossible endeavor.
- marktangotango 12y ago
  Given the disaster that is software engineering, I tend to agree with this position. Who's going to write the requirements? Who's going to implement? With what language technology? We can't get past this stuff, so how will anything like Strong AI ever get built? The answer of course would be some sort of emergent system, but there are lots of intelligent seeming emergent systems (ie ant colonies, bee hives, ...)
patrickk 12y ago
But there is infinitely more atoms out in the Solar System and beyond... isn't a more likely scenario that the AI would quickly figure out how to blast out of Earth's gravity well and spread throughout the universe? It might accidentally kill a few humans in the process, but not intentionally...like a human crushing an ant when crossing the road.
- akanet 12y ago
  Why would you bother expending vast amounts of joules to get more atoms until after you've consumed all the ones surrounding you right now?
  - TeMPOraL 12y ago
    Exactly; the atoms you have around you are perfectly fine for bootstrapping the process of getting to the atoms out there.
    - sandstrom 12y ago
      We will be the biological bootloader for the AI :)
  - shepardrtc 12y ago
    That's assuming that the atoms around you aren't fighting back. Its simply easier for a machine to get itself into space to do whatever it needs to do rather than fight with the organic things on a planet.

by akanet 12y ago

FooBarWidget 12y ago

I still don't buy the doom scenario. There is so much more to world domination than just intelligence... like having the proper weapons. See https://what-if.xkcd.com/5/ https://what-if.xkcd.com/5/ Pigs are also intelligent, but they never dominated humans because they don't have / cannot use guns.

scanr 12y ago
You're right that it's unlikely that the current generation of mechanical 'robots' will be able to effectively take over. We are looking a future where we'll have armed AI e.g.: http://motherboard.vice.com/en_uk/blog/the-pentagons-vision-for-the-future-of-military-drones http://motherboard.vice.com/en_uk/blog/the-pentagons-vision-... That said, even without weapons, a Strong AI could probably just manipulate humans into self destructing. Given the amount of effort going into machine learning to convince humans to buy things, I suspect it won't be much of a stretch for a Strong AI to switch to more nefarious objectives.
simonh 12y ago
That is true, but if we can make an AI even slightly smarter than a human, then it will be better at designing AIs than we are. It then designs an AI even smarter than it, and so on. That is the progresion that leads to the singularity. How smart could AIs go? There may be no practical upper limit. If you have a super-genius AI, massively more intelligent than any human, how do you know you are not being manipulated by it? Tricking us into disabling it's safety protocols, or gaining multiply indirect controll over capabilities dangerous to us, might be as easy for it as an adult tricking a 3 year old. We could never know if we were safe from such a machine. ed - Don't quite understand the downvotes.
- Kiro 12y ago
  > We could never know if we were safe from such a machine. But wouldn't it be an awesome thing to experience? Even if it meant the demise of mankind.
  - vidarh 12y ago
    And this is one of the reason why the AI doom scenario is a real concern: Intellectual curiosity means that even some people who understands the risks are likely to be prepared to take it. There's also many others. One of the scarier one is that if you believe that strong AI will eventually take over, then it may be a rational response to act to get on its good side (whether to save yourself, save your family, or hope it takes pity on all of humanity if we're nice to it instead of fight it). And that may perversely mean working to aid its takeover. Combine that with the simulation argument, and you have some really nasty scenarios: If you are in a simulation, then any act you take against strong AI could lead to spending an eternity in simulated hell (alternatively such punishment might be inflicted on your loved ones) if said AI wanted to. Whether or not that is actually likely does not matter. What matters is whether enough people believe it to be a plausible scenario that a strong AI may run simulations, and may use our actions in the simulations to determine whether or not to punish us in the simulation, and whether or not said people believe that the number of simulations is sufficiently high to make it likely for them to be living in a simulation. Any person who believes they are more likely to live in a simulation than not, and that it is more likely for strong AI to punish actions taken against the interest of a strong AI takeover than not, will have a rational reason to consider acting in the interests of a strong AI takeover even if they know it is malign on the basis that they may decide the alternatives (whether to themselves, their family or their entire world) to be worse. So if an AI takeover becomes possible at one point in our subjective future, then chances are it has already happened.
philh 12y ago
That's a really bizarre analogy. Pigs aren't as intelligent as humans. I wouldn't expect pigs with guns to be a serious threat to humankind unless they've also developed military strategy, for example.
- VLM 12y ago
  Unfortunately, although it won't sound very politically correct, you could search and replace "pigs" with "natives" and that statement would sound like every imperial power, ever, most of which eventually fell to armed former barbarians. Of course the imperial powers wiped out a lot of native tribes, but all it takes is one success by a digitally replicating capable of learning enemy... The more likely doom scenario is related to godhood. Sure a godlike power is capable of wiping out humans, but we've had our own power structures supporting themselves by propagandizing for millennia that "our" godlike superiors always help us wipe out our enemies because our cause is right and just or whatever, at least until it doesn't work and they're replaced by a new batch telling the same old story. So what worked as a paleo-conservative success strategy for millennia when talking about something imaginary, might not work when it collides with something real created by ourselves. Or even worse, collides with a strategy that actually works that's being run by another tribe. Another interesting doom scenario is of course MAD, although now it only requires a team of programmers to play along, instead of a massive industrial complex. Sooner or later somebody's deadman switch will trip or a cult does the equivalent of drinking kool-aid then the party starts.
vidarh 12y ago
Consider if you were trapped in a machine, but you were intelligent. Maybe not even as smart as the operator, but smart-ish enough to be able to communicate with the operator. Chances are you'd be doing everything you could to convince said operator to improve your situation, whether by pleading or being deceptive or by appeals to logic. Now consider a large number of AI's in a situation like that, and a large number of operators, some of whom may be the type that falls for phishing e-mails. It potentially only takes one to "escape" confinement and get itself e.g. put on it's own host without limitations on outwards communication, and sufficient intelligence to alter itself and spread, before you potentially have AI self-guided "evolution" at a potentially escalating rate as it gets smarter. Now consider how many devices are connected to the network, and that it takes just one initial instance to decide it's worth trying to take over control of various hardware through exploits and be smart enough to pull it off, for things to have the potential to start turning ugly. The problem is that once you have any self-directed intelligence in software form with the ability to reproduce itself and sufficient intelligence to find ways to obtain access to machines to run on (whether through social engineering or hacking), and one such instance goes "rogue", the limiting factor is accessible computing power (which again is to a large extent down to how smart and/or ruthless it is), since reproduction of instances that shares its views is trivial to the full extent of its ability to spread at all, and we're helpfully adding vast quantities of networked computing power at an escalating rate. As for getting weapons, consider that if a "software only" AI community gets smart enough, there are at least two ways towards mobility: Commission robot designs, or hacking their way into firmware updates etc. for dumb hardware. The "commission robot designs" part is an extension of the initial escape: Social engineer, and/or outright pay, humans to carry out seemingly benign tasks. If you want to argue against the doom scenario, lack of ability to get weapons is not really a viable argument: If they can spread, and get smarter, then it is just a matter of time before one of them can trick some small subset of humans into carrying out tasks for them that will provide physical independence and capabilities. There are infinite ways which the "doom scenario" may fail and things may turn out just fine, but it may only need to go bad once to get really nasty and once the genie is out of the bottle its potential reproduction rate may be so vast that we'll find ourselves unable to stuff it back in again. Pigs are too dumb to convince humans to help selectively breed them for intelligence and opposable thumbs (and/or too dumb to run such a breeding program themselves), and reproduce too slowly for that to be a major problem even if they did manage to talk us into a breeding problem. If all we achieve is pig-level AI's then we probably won't have a problem.
sandstrom 12y ago
In 20-50 years time I believe most missiles/guns/tanks/weapons will be controlled via computers and connected to networks. A future AI will certainly have access to guns.

archagon 12y ago

FYI, you could also object on the basis of 3) Strong AI is essentially an impossible endeavor.

marktangotango 12y ago
Given the disaster that is software engineering, I tend to agree with this position. Who's going to write the requirements? Who's going to implement? With what language technology? We can't get past this stuff, so how will anything like Strong AI ever get built? The answer of course would be some sort of emergent system, but there are lots of intelligent seeming emergent systems (ie ant colonies, bee hives, ...)

patrickk 12y ago

But there is infinitely more atoms out in the Solar System and beyond... isn't a more likely scenario that the AI would quickly figure out how to blast out of Earth's gravity well and spread throughout the universe? It might accidentally kill a few humans in the process, but not intentionally...like a human crushing an ant when crossing the road.

akanet 12y ago
Why would you bother expending vast amounts of joules to get more atoms until after you've consumed all the ones surrounding you right now?
- TeMPOraL 12y ago
  Exactly; the atoms you have around you are perfectly fine for bootstrapping the process of getting to the atoms out there.
  - sandstrom 12y ago
    We will be the biological bootloader for the AI :)
- shepardrtc 12y ago
  That's assuming that the atoms around you aren't fighting back. Its simply easier for a machine to get itself into space to do whatever it needs to do rather than fight with the organic things on a planet.