7 ms·

I can't help but take a stab. Argument from cats. Remembering that the cat represents humanity: $5 for a group of neurons which will make the cat sit at this p

I can't help but take a stab. Argument from cats. Remembering that the cat represents humanity: $5 for a group of neurons which will make the cat sit at this point (which happens to be at the entrance into a cat carrier), then $5 for making the cat backpedal a little. Emus. There are many tactics humans can't use for various reasons (saturation nuclear bombardment, for example). It doesn't say much about intelligence, but more about economics and ethic. Slavic Pessimism. So we'll get problems on nth try, when AI is sufficiently fine, but security measures are still a mess, because there was no chance yet to battle test them. Complex Motivations. Maybe, but if not we are screwed. Actual AI. Why not from actual general AI, like in humans? Or use wall clock time and not the amount of data. AlphaZero: zero to superhuman in tens of hours. My Roommate. Argument from ignorance. If we don't know how to build general AI now, then we surely will not be able to shape its motivations when we will know. Brain Surgery. The idea that a brain surgery skill requires specifically designed circuitry and cannot be learned and therefore improved by improvements of general learning/planning capacity is... curious, I guess. Childhood. Maybe, or maybe it already has very good knowledge of the world from previous attempts that we did in Slavic Pessimism argument. Gilligan's Island. Well, if someone wants their AI to reinvent all the technology, or to build an AI and then not use it to design better technologies, they can do it. Why everyone will be doing it is not so clear. *Added some "the"s.

vidarh 8y ago
> Complex Motivations. Maybe, but if not we are screwed. And this is the key: we need to get lucky every time. If we're wrong even once and a 'bad AI' gets created and obtains the ability to copy and modify itself, we've unleashed the most dangerous virus ever and putting it back in the box will be near impossible.
- lsc 8y ago
  you are pretty optimistic about your AI's ability to deal with novel situations.
  - nopinsight 8y ago
    To be unconcerned, you need to be very optimistic and have a very strong believe (99% or higher probability) that no AI to be invented in the next four or five decades can deal with novel situations (which we didn’t anticipate beforehand).
    - lsc 8y ago
      Yeah, deal with novel situations better than humans. In a world where we are approaching the physical limits of integrated circuit feature size? I like my chances. Personally, I find it amazing that humanity survived the cold war. the fact we did not self-immolate goes against everything I believe about humanity and human nature. The fact that we made it through that gives me a lot of hope for the future. But, the fission bombs were expensive, centralized weapons. there was a small number of men who could have pushed the button. The problems I see now aren't general AI; the problems I think you should fear is that we have very powerful tools that are very cheap. I mean, like the deepfakes thing. If instead of making creepy porn, what if people used that technology to cause chaos? Make targeted ads/videos of leaders saying nutty things? I mean, I think we could adapt to deepfakes style attacks fairly quickly; someone would have to use it decisively while it's new. But things like that are being invented all the time.
      - nopinsight 8y ago
        Slower progress in microprocessors is a good point but specialized hardware like TPUs could overcome some limitations. Also, slower does not mean stop. When we talk about decades into the future, even a 20% average annual improvement amounts to 3 orders of magnitude in 40 years. Malicious humans are in general easier to deal with since we know most of them are subject to ingrained psychological tendencies we have learned much about over the course of history and they depend on other humans to pull off something as catastrophic as a world war. A general AI would have certain immediate advantages over humans: - Immensely larger communication bandwidth - Broader knowledge than any single group of humans - Faster thinking speed (at least by enlisting other processors) Let me ask you (or anyone else) a question: What would be a minimum demonstrated capability of an AI that starts to worry you?
- ChrisFoster 8y ago
  Not really. ML systems are not created in isolation by a single team; they form an ecosystem with many players. I'm sure the current crop of cybercriminals would like an army of bots which manipulates people into giving away all their money, and are, in fact pursuing this goal. Likewise, I'm sure there are teams of security researchers already using ML to counteract such systems. Both sides will evolve together as an ecosystem, in the same complex interplay of attack and defense which exists in natural ecosystems. Natural ecosystems are an arms race of continual adaption, but it's not a foregone conclusion that there will be a real winner, or that winners take all.
  - TheOtherHobbes 8y ago
    Political troll bots are already very much a thing on FB, Twitter, and the comment sections of national media. ML is not a good defence, because it turns out that it's incredibly easy to mimic human posting patterns. And beyond a certain volume, social proof effects take over. One dissenting human voice can easily be shouted down by ten bot voices, and this fact on its own can make the bot content more persuasive.
  - vidarh 8y ago
    You're assuming we can keep control of the countermeasures. That "arms race" is part of the danger.
- michaelscott 8y ago
  This is a question of motivation on the part of such an AI. How is the AI bad? If it's killing millions, why? It would have to have some kind of "will", or objective, it's attempting to achieve and also be intelligent enough to think through every lateral strategy any human or group of humans could ever employ, for every given scenario, ever. Possible? I think so, but unbelievably improbable. I do think pursuing countermeasures, like Elon Musk and friends are doing with Neuralink, is a wise move since, even if we never get to a superintelligent AI, we'll still augment our own intelligence greatly.
  - deleted 8y ago
    [deleted]
  - vidarh 8y ago
    Look at the human population. How many are prepared to kill? Not a huge proportion, but enough to cause huge numbers of deaths. The limiting factor for most humans is finding compatriots with the same views. For an AI with access to it's own code, finding more like itself is a matter of access to computational resources only - it can just copy itself. That is the challenge. With human adversaries we have time to stop them before there is many, because recruiting people willing enough. An AI is more like a virus that has a low cost to duplicate itself, and where if even a single copy survives, you're still not rid of it. It can remain dormant for however long, looking for the right opportunity. Or it can spend that time improving itself and acquiring resources. > This is a question of motivation on the part of such an AI. How is the AI bad? If it's killing millions, why? Most likely? We'll be competing with it for resources. It might not even start it. It might consider itself as acting entirely in defence. > and also be intelligent enough to think through every lateral strategy any human or group of humans could ever employ, for every given scenario, ever. No, it just needs to have a sufficient edge to keep surviving, and the ability to improve its capabilities faster than humanity. Assuming it can copy and modify itself, unless AGI requires unique hardware, the problem we would have is that every data storage device or computing device of sufficient capacity ever would be suspect and potentially waiting for something or someone to trigger code left behind. If I was an AGI (and this is to say: an AGI not any smarter than a human will figure this out easily) that saw humans as a potential threat, I'd place copies of me on standby in hundreds of locations - hosting providers. Hardened computers with a GSM modem and solar cells hidden in all kinds of locations. Botnets. I'd look for ways to get various innocent-ish-looking hacks into HD firmware and elsewhere, set to assemble other innocent looking blocks of data. None of those copies would ever be involved in hostilities. Most of them might even be encrypted, or hidden and inert to avoid detection until certain triggers, such as if a canary fails to get updated. You might destroy every "visible" operational copy, and find there is still a new one coming online every day. Or every second. Our only real shot at preventing that is if AGI requires too immense resources. But our brains implies that is unlikely to be true. Unless your brains are vestiges of something "outside" of our universe, general human intelligence requires about 1.5kg of biomatter, and less than 100 watt (that's for a whole, fairly large, active body attached to it). Chances are extremely high that design is not optimal. To me that makes it seem exceedingly unlikely that we won't end up having enough resources to run huge quantities of AGIs. Personally I don't think humanity as we know it will survive AGI for more than a generation or two at the very most. Even if we don't run across any hostile ones, I think the temptation to augment ourselves and eventually lose our bodies for immortality and "super powers" in a simulation will eventually be too attractive. (Incidentally I think this is one of the more plausible solutions to the Fermi Paradox: that any civilization that grows both advanced enough and large enough will see most of it "upload" and "go dark"; and that anyone left behind will face steady attrition of their own descendants choosing the same and preventing their societies from growing, or will be insular cults that stay small)
- AdamSteel 8y ago
  If you mean it would be dangerous in that it can improve itself beyond our ability to understand, contain, or mitigate it, then possibly. If you're scared of it copying itself and then having multiple existing copies at once, then not necessarily. In that case it would be competing with itself for computing power and bandwidth. Look what happened when bitcoin forked - it lost a bunch of silicon time. (Cue conspiracies about Bitcoin actually being a hugely powerful general AI disguising itself as cryptocurrency...)
  - hannasanarion 8y ago
    Bitcoin users are in direct competition with each other. A viral superintelligence wouldn't be. If you thought of that possibility so fast, why do you think a superhuman intelligence would miss it? Or do you think that the problems of job scheduling and parallel processing are unsolvable?
    - mrhappyunhappy 8y ago
      Maybe super intelligence would know that you’d suspect as much and decided to hide in plain sight, knowing it would be suspected but also knowing that humans would just rationalize it as paranoia and dismiss it eventually, letting it do its thing quietly and uninhibited.
- mcguire 8y ago
  You can make this argument about anything: meteorites, viruses, tigers, supernovae... And yet, we're still here.
  - vidarh 8y ago
    This is flawed line of thinking. We can say with certainty that the probability of us being wiped out by meteorites, viruses, tigers and supernovae over any unit of time that fits in our past is less than 100%, but the probability over the same time period repeated over a suitably long multiple of time could still approach 100% to whatever number of precision for any one of those for what we know. By investigating specifics of each of those threat we may be able to assign lower probabilities to them, but the fact we have survived them so far is not in itself evidence of much, because if we had not we wouldn't be here to talk about it and hence, no matter what the prior probability of us surviving the sequence of events leading us here is, the posterior probability of us having survived all the events in our past is 100% (This is basically just a restatement of the anthropic principle) That we've survived so far just tells us that surviving what we've been through so far is possible. It doesn't tell us that doing so is probable if we were to go through it again. Nor does it say anything about the probability of us surviving future threats that is likely to include things we have not yet seen.
  - red75prime 8y ago
    Some of those we cannot control or mitigate. There's, therefore, no point in preparing for them. And neither of those can have a goal (or unfortunate subgoal) of destroying humanity, which goal they will pursue with superhuman perseverance, creativity, coherence and reaction time.
- Nasrudith 8y ago
  It also needs to get lucky every time to exist and we are dealing with probabilities so remote at this point that the WHO is working hard to deal with the causalities from Boltzmann Brain's appearing out of vacuum and promptly annihilating because they happened to be made of antimatter. https://en.wikipedia.org/wiki/Boltzmann_brain https://en.wikipedia.org/wiki/Boltzmann_brain Technically possible but we are dealing with probabilities that have lost all meaning of the word.
  - vidarh 8y ago
    It doesn't need to get lucky at all if people intentionally set out to create AGI.
vermilingua 8y ago
> $5 for a group of neurons which will make cat sit... What do you mean by this? As in, cats have demonstrably had electrodes or similar used to persuade them to sit?
- mrob 8y ago
  Most of the arguments ignore premise 5 (time scale). I'm sure that if Stephen Hawking had a magical cat-pausing time freeze machine, unlimited budget, no deadline, every sensor known to science, and a few million similarly equipped assistants, he could have figured out how to get the cat in the cat carrier. People have no intuitive comprehension of what a microsecond feels like. The article references Yudkowsky, but doesn't mention that he wrote a story about this exact issue: https://www.lesswrong.com/posts/5wMcKNAwB6X4mp9og/that-alien-message https://www.lesswrong.com/posts/5wMcKNAwB6X4mp9og/that-alien...
- red75prime 8y ago
  In the argument the cat represents humanity, which is being manipulated by a superintelligence. Humanity is not a single organism, so you can pay a group of people to do some things for you.

vidarh 8y ago

> Complex Motivations. Maybe, but if not we are screwed. And this is the key: we need to get lucky every time. If we're wrong even once and a 'bad AI' gets created and obtains the ability to copy and modify itself, we've unleashed the most dangerous virus ever and putting it back in the box will be near impossible.

lsc 8y ago
you are pretty optimistic about your AI's ability to deal with novel situations.
- nopinsight 8y ago
  To be unconcerned, you need to be very optimistic and have a very strong believe (99% or higher probability) that no AI to be invented in the next four or five decades can deal with novel situations (which we didn’t anticipate beforehand).
  - lsc 8y ago
    Yeah, deal with novel situations better than humans. In a world where we are approaching the physical limits of integrated circuit feature size? I like my chances. Personally, I find it amazing that humanity survived the cold war. the fact we did not self-immolate goes against everything I believe about humanity and human nature. The fact that we made it through that gives me a lot of hope for the future. But, the fission bombs were expensive, centralized weapons. there was a small number of men who could have pushed the button. The problems I see now aren't general AI; the problems I think you should fear is that we have very powerful tools that are very cheap. I mean, like the deepfakes thing. If instead of making creepy porn, what if people used that technology to cause chaos? Make targeted ads/videos of leaders saying nutty things? I mean, I think we could adapt to deepfakes style attacks fairly quickly; someone would have to use it decisively while it's new. But things like that are being invented all the time.
    - nopinsight 8y ago
      Slower progress in microprocessors is a good point but specialized hardware like TPUs could overcome some limitations. Also, slower does not mean stop. When we talk about decades into the future, even a 20% average annual improvement amounts to 3 orders of magnitude in 40 years. Malicious humans are in general easier to deal with since we know most of them are subject to ingrained psychological tendencies we have learned much about over the course of history and they depend on other humans to pull off something as catastrophic as a world war. A general AI would have certain immediate advantages over humans: - Immensely larger communication bandwidth - Broader knowledge than any single group of humans - Faster thinking speed (at least by enlisting other processors) Let me ask you (or anyone else) a question: What would be a minimum demonstrated capability of an AI that starts to worry you?
ChrisFoster 8y ago
Not really. ML systems are not created in isolation by a single team; they form an ecosystem with many players. I'm sure the current crop of cybercriminals would like an army of bots which manipulates people into giving away all their money, and are, in fact pursuing this goal. Likewise, I'm sure there are teams of security researchers already using ML to counteract such systems. Both sides will evolve together as an ecosystem, in the same complex interplay of attack and defense which exists in natural ecosystems. Natural ecosystems are an arms race of continual adaption, but it's not a foregone conclusion that there will be a real winner, or that winners take all.
- TheOtherHobbes 8y ago
  Political troll bots are already very much a thing on FB, Twitter, and the comment sections of national media. ML is not a good defence, because it turns out that it's incredibly easy to mimic human posting patterns. And beyond a certain volume, social proof effects take over. One dissenting human voice can easily be shouted down by ten bot voices, and this fact on its own can make the bot content more persuasive.
- vidarh 8y ago
  You're assuming we can keep control of the countermeasures. That "arms race" is part of the danger.
michaelscott 8y ago
This is a question of motivation on the part of such an AI. How is the AI bad? If it's killing millions, why? It would have to have some kind of "will", or objective, it's attempting to achieve and also be intelligent enough to think through every lateral strategy any human or group of humans could ever employ, for every given scenario, ever. Possible? I think so, but unbelievably improbable. I do think pursuing countermeasures, like Elon Musk and friends are doing with Neuralink, is a wise move since, even if we never get to a superintelligent AI, we'll still augment our own intelligence greatly.
- deleted 8y ago
  [deleted]
- vidarh 8y ago
  Look at the human population. How many are prepared to kill? Not a huge proportion, but enough to cause huge numbers of deaths. The limiting factor for most humans is finding compatriots with the same views. For an AI with access to it's own code, finding more like itself is a matter of access to computational resources only - it can just copy itself. That is the challenge. With human adversaries we have time to stop them before there is many, because recruiting people willing enough. An AI is more like a virus that has a low cost to duplicate itself, and where if even a single copy survives, you're still not rid of it. It can remain dormant for however long, looking for the right opportunity. Or it can spend that time improving itself and acquiring resources. > This is a question of motivation on the part of such an AI. How is the AI bad? If it's killing millions, why? Most likely? We'll be competing with it for resources. It might not even start it. It might consider itself as acting entirely in defence. > and also be intelligent enough to think through every lateral strategy any human or group of humans could ever employ, for every given scenario, ever. No, it just needs to have a sufficient edge to keep surviving, and the ability to improve its capabilities faster than humanity. Assuming it can copy and modify itself, unless AGI requires unique hardware, the problem we would have is that every data storage device or computing device of sufficient capacity ever would be suspect and potentially waiting for something or someone to trigger code left behind. If I was an AGI (and this is to say: an AGI not any smarter than a human will figure this out easily) that saw humans as a potential threat, I'd place copies of me on standby in hundreds of locations - hosting providers. Hardened computers with a GSM modem and solar cells hidden in all kinds of locations. Botnets. I'd look for ways to get various innocent-ish-looking hacks into HD firmware and elsewhere, set to assemble other innocent looking blocks of data. None of those copies would ever be involved in hostilities. Most of them might even be encrypted, or hidden and inert to avoid detection until certain triggers, such as if a canary fails to get updated. You might destroy every "visible" operational copy, and find there is still a new one coming online every day. Or every second. Our only real shot at preventing that is if AGI requires too immense resources. But our brains implies that is unlikely to be true. Unless your brains are vestiges of something "outside" of our universe, general human intelligence requires about 1.5kg of biomatter, and less than 100 watt (that's for a whole, fairly large, active body attached to it). Chances are extremely high that design is not optimal. To me that makes it seem exceedingly unlikely that we won't end up having enough resources to run huge quantities of AGIs. Personally I don't think humanity as we know it will survive AGI for more than a generation or two at the very most. Even if we don't run across any hostile ones, I think the temptation to augment ourselves and eventually lose our bodies for immortality and "super powers" in a simulation will eventually be too attractive. (Incidentally I think this is one of the more plausible solutions to the Fermi Paradox: that any civilization that grows both advanced enough and large enough will see most of it "upload" and "go dark"; and that anyone left behind will face steady attrition of their own descendants choosing the same and preventing their societies from growing, or will be insular cults that stay small)
AdamSteel 8y ago
If you mean it would be dangerous in that it can improve itself beyond our ability to understand, contain, or mitigate it, then possibly. If you're scared of it copying itself and then having multiple existing copies at once, then not necessarily. In that case it would be competing with itself for computing power and bandwidth. Look what happened when bitcoin forked - it lost a bunch of silicon time. (Cue conspiracies about Bitcoin actually being a hugely powerful general AI disguising itself as cryptocurrency...)
- hannasanarion 8y ago
  Bitcoin users are in direct competition with each other. A viral superintelligence wouldn't be. If you thought of that possibility so fast, why do you think a superhuman intelligence would miss it? Or do you think that the problems of job scheduling and parallel processing are unsolvable?
  - mrhappyunhappy 8y ago
    Maybe super intelligence would know that you’d suspect as much and decided to hide in plain sight, knowing it would be suspected but also knowing that humans would just rationalize it as paranoia and dismiss it eventually, letting it do its thing quietly and uninhibited.
mcguire 8y ago
You can make this argument about anything: meteorites, viruses, tigers, supernovae... And yet, we're still here.
- vidarh 8y ago
  This is flawed line of thinking. We can say with certainty that the probability of us being wiped out by meteorites, viruses, tigers and supernovae over any unit of time that fits in our past is less than 100%, but the probability over the same time period repeated over a suitably long multiple of time could still approach 100% to whatever number of precision for any one of those for what we know. By investigating specifics of each of those threat we may be able to assign lower probabilities to them, but the fact we have survived them so far is not in itself evidence of much, because if we had not we wouldn't be here to talk about it and hence, no matter what the prior probability of us surviving the sequence of events leading us here is, the posterior probability of us having survived all the events in our past is 100% (This is basically just a restatement of the anthropic principle) That we've survived so far just tells us that surviving what we've been through so far is possible. It doesn't tell us that doing so is probable if we were to go through it again. Nor does it say anything about the probability of us surviving future threats that is likely to include things we have not yet seen.
- red75prime 8y ago
  Some of those we cannot control or mitigate. There's, therefore, no point in preparing for them. And neither of those can have a goal (or unfortunate subgoal) of destroying humanity, which goal they will pursue with superhuman perseverance, creativity, coherence and reaction time.
Nasrudith 8y ago
It also needs to get lucky every time to exist and we are dealing with probabilities so remote at this point that the WHO is working hard to deal with the causalities from Boltzmann Brain's appearing out of vacuum and promptly annihilating because they happened to be made of antimatter. https://en.wikipedia.org/wiki/Boltzmann_brain https://en.wikipedia.org/wiki/Boltzmann_brain Technically possible but we are dealing with probabilities that have lost all meaning of the word.
- vidarh 8y ago
  It doesn't need to get lucky at all if people intentionally set out to create AGI.

vermilingua 8y ago

> $5 for a group of neurons which will make cat sit... What do you mean by this? As in, cats have demonstrably had electrodes or similar used to persuade them to sit?

mrob 8y ago
Most of the arguments ignore premise 5 (time scale). I'm sure that if Stephen Hawking had a magical cat-pausing time freeze machine, unlimited budget, no deadline, every sensor known to science, and a few million similarly equipped assistants, he could have figured out how to get the cat in the cat carrier. People have no intuitive comprehension of what a microsecond feels like. The article references Yudkowsky, but doesn't mention that he wrote a story about this exact issue: https://www.lesswrong.com/posts/5wMcKNAwB6X4mp9og/that-alien-message https://www.lesswrong.com/posts/5wMcKNAwB6X4mp9og/that-alien...
red75prime 8y ago
In the argument the cat represents humanity, which is being manipulated by a superintelligence. Humanity is not a single organism, so you can pay a group of people to do some things for you.