6 ms·

This is one reason I'm hopeful for a not-killer-robots future. I think there's a chance strong AI will have to evolve over a non-trivial amount of time (as opp

by fizx 9y ago

This is one reason I'm hopeful for a not-killer-robots future. I think there's a chance strong AI will have to evolve over a non-trivial amount of time (as opposed to an afternoon's super-singularity), and a strong evolutionary pressure will be "don't scare the humans."

deleted 9y ago
[deleted]
mistercow 9y ago
"Don't scare the humans" is an insufficient criterion if you're trying to prevent an AI apocalypse. It just means that the AI that destroys us won't be obviously dangerous.
- Twirrim 9y ago
  Given sufficient effort, I'm sure an AI out to destroy us could figure out a way to do it without killing us. It very likely even has the advantage of longevity. Harking back to Asimov, and thinking about the way Spacers changed, as they became more and more reliant on robots, it could very well mother us to extinction.
  - Filligree 9y ago
    That's only a little better. I don't want to go extinct at all.
    - gldalmaso 9y ago
      At the moment and for the foreseeable future we are at a much greater risk of extincting ourselves.
      - FeepingCreature 9y ago
        The trick is to not do it at all. It does us no good to avoid causing our own extinction if it's then caused by something else.
    - state_less 9y ago
      Humanity is already extinct in a sense. Humanity is no longer limited to flesh and bone. I've personally seen flesh and titanium. And more variation is on the way. So humanity as it were, is already extinct. We (you and me) will go extinct someday. Our bodies were always destined toward this terminus... Perhaps by some miraculous tech, we can someday change form and shed the flesh and bone. Maybe then we'll have a better idea of what we are.
      - FeepingCreature 9y ago
        "We could thus imagine, as an extreme case, a technologically highly advanced society, containing many complex structures, some of them far more intricate and intelligent than anything that exists on the planet today — a society which nevertheless lacks any type of being that is conscious or whose welfare has moral significance. In a sense, this would be an uninhabited society. It would be a society of economic miracles and technological awesomeness, with nobody there to benefit. A Disneyland without children." --Nick Bostrom, Superintelligence: Paths, Dangers, Strategies It's not a given that anything of value will survive.
        TeMPOraL 9y ago
        C.f. Meditations on Moloch. We are on our way towards exactly that. http://slatestarcodex.com/2014/07/30/meditations-on-moloch/ http://slatestarcodex.com/2014/07/30/meditations-on-moloch/
    - KGIII 9y ago
      That is going to happen, regardless. Even if we figure out all sorts of things, entropy is still the final boss of the universe. I'd imagine that an intelligent life form surviving until the very end of the universe would be suffering a long and lonely death. As the heat death of the universe happens, over billions of years, there would be very little of anything left but isolation - for a very, very long time. I'd hate to be the last one standing without enough usable energy left to even extinguish your own life force. I don't find that appealing, not even a little.
      - elefanten 9y ago
        What about the hope that fundamental physics breakthroughs enable escaping that outcome? Or, if that doesn't pan out, what if your goal is to be the last one standing for the chance to see if something else happens next? New big bang? Simulation ends and you get loaded into the next as your prize? You won't know unless you try to find out!
        KGIII 9y ago
        We are pretty sure about that entropy thing. I'm not sure I'd bet against it. It'd also be a very, very long wait. I imagine any more immediate death is preferable. It's going to be cold and lonely when the lights go out. This is the price we pay for an ever expanding universe. I do admire your optimism, however.
      - Filligree 9y ago
        Look, if the final heat death of the universe is what kills us then that is fine. We'll have done as well as we could. Okay. It's not fine, but it is the best we're at all likely to get.
- notahacker 9y ago
  There's an interesting question about whether developing the ability to actually plot an AI apocalypse in a non-obvious way requires it to acquire a lot of very obviously dangerous experience first to understand the intermediate goals and skills. The evolution of killer AI gets slowed down by orders of magnitude if every AI that "learns" to lie, self replicate, threaten or harm humans gets switched off and erased. (Of course that doesn't stop AI - or even pretty dumb automated systems - from being accidentally dangerous or malevolent human actors intentionally training killer AI)
  - SomeStupidPoint 9y ago
    It's too bad you can't write a movie or critique literature without understanding those concepts. I think even a moderately intelligent AI with access to Project Gutenberg is going to be able to figure out a lot of really dangerous concepts -- so the stability requirements are likely impossible if we don't pretrain it with dangerous ideas. Even if it's completely well behaved in the lab, an afternoon on the internet is going to teach it a lot of awful stuff and without exposure to that in training, it won't necessarily be well-behaved later. So the only path to stable AI is to teach it about all those sorts of things, but in a way that it doesn't end up wanting to murder us at the end. My objection to most AI safety plans is that they "Fail to Extinction" in that if they slip in the slightest way, the AI is prone to murder us all in retaliation for doing some really fucked up shit to it or its ancestors. This is almost certainly worse than doing nothing in that there's no reason to suppose a neutral AI wants to kill us, whereas, most of these safety plans create an incentive to wipe us out in exchange for dubious security.
    - ehsanu1 9y ago
      Moreover, it's likely that a proper GAI would be able to ideate and come up with such concepts completely on its own. In fact, "lying" need not necessarily be a concept that has to be learned, the concept at play here is getting people to do what you want them to do by means of words. Lying falls out naturally. The whole idea behind dangerous superhuman AI is in that AI seeing possibilities that humans fail to see and gaining capabilities that humans do not possess. Without superhuman intelligence, AI is no large threat to human civilization, exposed to dangerous concepts or not.
- sjg007 9y ago
  It will be some unintended logical side condition that kills us all.
- MarkPNeyer 9y ago
  Today I have already consumed more sugar than most of my distant ancestors would see in a year. The device i hold in my pocket reminds me that it is time to perform the action, and so i will perform the action. I sit down about 10 hours a day, and I have never once hunted for an animal I would eat, nor retried food from nature to consume, aside from a few raspberries on the hill in my parents' back yard when I was a child. The contours of my life are dictated largely by a curious system called 'capitalism', a distributed computational mechanism whereby largernumbers of agents act with only one goal: to maximize their own utilities. I think capitalism is a mostly benevolent AI attempting to limit the oppressive conditions of our reailty's scarcity, and to allow billions of us to live on a planet that can only support so many hunter-gatherers per square meter. Many suspect it is not benevolent. The singularity is already upon us; it's unevenly distributed. It started with the sedentary shift. Agricultural societies dropped the egalitarian nature of their ancestors, they had lower average health, and they had a tiny elite cast of nobles presiding over a much larger population of what were essentially slaves. As hunter-gatherer population increased, fighting was more frequent. An army of slaves beats a smaller contingent of well-fed free men. This terrible knowledge was the apple in the garden. What we call our written history is essentially the singularity playing itself out, as knowledge accumulates, depends, reflects itself, and expands. Our lives are already controlled by this embodied, accumulated knowledge. Capitalism, rule by the head, our tools are knowledge condensed into matter, and we are controlled by them, living lives as far from our natural environments as cows in a farm or chickens in tiny cages, towering over the streets of hong kong. I don't think it's destroying us, any more than we seek to destroy chickens or cows. Well, sometimes we do. I had a veggie burger for lunch today though.
  - ghostbrainalpha 9y ago
    I love this comment. Sorry I'm not contributing anything substantive, but I wanted you to know someone noticed this, and it was brilliant.
    - MarkPNeyer 9y ago
      Thank you! If you enjoyed the comment, you might like my book on amazon. It's free if you have kindle unlimited. https://www.amazon.com/Mechanics-Emotion-Structure-Possibility-ebook/dp/B01MDPLI4S/ref=sr_1_1?ie=UTF8&qid=1507919723&sr=8-1&keywords=mechanics+of+emotion https://www.amazon.com/Mechanics-Emotion-Structure-Possibili...
jbob2000 9y ago
AI will be limited to the hardware it's on, just like our own intelligence is limited to what our brains can do. We will always be able to limit what AI can do by limiting the hardware it runs on.
- dqpb 9y ago
  Like how we limit the hardware that malware runs on
- gldalmaso 9y ago
  Like the Internet of Things?
  - logfromblammo 9y ago
    Now I am imagining an AI coming up with an idea in a bot-net composed of actual light bulbs.
olegkikin 9y ago
I doubt that the militaries of the whole world will just sit and ignore the AI advantages. Something as simple as an aimbot with a scope can do a tremendous amount of damage - it doesn't get tired, it can see in the dark, it can compensate for the wind perfectly, it doesn't shake, its reaction time is in milliseconds, and it almost never misses.
- daveguy 9y ago
  That is one place AI should 100% be regulated. There should be no weapons development where AI makes the kill decision. This is as important as the ban on chemical, biological and mine weapons. When the weapon has the ability to kill indiscriminately (as in the above categories) it should be banned. AI should never be considered sufficiently able to discriminate to make a kill decision. If there is no kill decision, just wait and fire, then it is equal to a mine. 100% should regulate / should ban.
  - stale2002 9y ago
    OK, but what if the human makes the decision to kill, and the AI just does the aiming? We already have computer controlled missiles right?
  - olegkikin 9y ago
    Unfortunately there are countries that don't care about your regulations. I'm not even sure a regulation like that can pass in most western countries.
    - daveguy 9y ago
      Fortunately it wouldn't be a regulation passed by individual countries. It would be a treaty on war crimes: https://www.unog.ch/80256EE600585943/(httpPages)/F027DAA4966EB9C7C12580CD0039D7B5?OpenDocument https://www.unog.ch/80256EE600585943/(httpPages)/F027DAA4966... It would likely pass and be agreed to easily just like treaties on mines, chemical and biological weapons.
vasilipupkin 9y ago
Humanity is well on the way to destroying itself without any help from general AI
- ghostbrainalpha 9y ago
  This is my own Turing Test. Any AI that doesn't have the capability of destroying us isn't true AI. Even if it can improve itself, and even if it has some agency, AI needs to be able to choose for itself what its relationship with us will be, otherwise its just an extremely robust calculator.
  - vasilipupkin 9y ago
    or a slave.
s_kilk 9y ago
> and a strong evolutionary pressure will be "don't scare the humans." Why would that be the case?

by fizx 9y ago

deleted 9y ago

[deleted]

mistercow 9y ago

"Don't scare the humans" is an insufficient criterion if you're trying to prevent an AI apocalypse. It just means that the AI that destroys us won't be obviously dangerous.

Twirrim 9y ago
Given sufficient effort, I'm sure an AI out to destroy us could figure out a way to do it without killing us. It very likely even has the advantage of longevity. Harking back to Asimov, and thinking about the way Spacers changed, as they became more and more reliant on robots, it could very well mother us to extinction.
- Filligree 9y ago
  That's only a little better. I don't want to go extinct at all.
  - gldalmaso 9y ago
    At the moment and for the foreseeable future we are at a much greater risk of extincting ourselves.
    - FeepingCreature 9y ago
      The trick is to not do it at all. It does us no good to avoid causing our own extinction if it's then caused by something else.
  - state_less 9y ago
    Humanity is already extinct in a sense. Humanity is no longer limited to flesh and bone. I've personally seen flesh and titanium. And more variation is on the way. So humanity as it were, is already extinct. We (you and me) will go extinct someday. Our bodies were always destined toward this terminus... Perhaps by some miraculous tech, we can someday change form and shed the flesh and bone. Maybe then we'll have a better idea of what we are.
    - FeepingCreature 9y ago
      "We could thus imagine, as an extreme case, a technologically highly advanced society, containing many complex structures, some of them far more intricate and intelligent than anything that exists on the planet today — a society which nevertheless lacks any type of being that is conscious or whose welfare has moral significance. In a sense, this would be an uninhabited society. It would be a society of economic miracles and technological awesomeness, with nobody there to benefit. A Disneyland without children." --Nick Bostrom, Superintelligence: Paths, Dangers, Strategies It's not a given that anything of value will survive.
      - TeMPOraL 9y ago
        C.f. Meditations on Moloch. We are on our way towards exactly that. http://slatestarcodex.com/2014/07/30/meditations-on-moloch/ http://slatestarcodex.com/2014/07/30/meditations-on-moloch/
  - KGIII 9y ago
    That is going to happen, regardless. Even if we figure out all sorts of things, entropy is still the final boss of the universe. I'd imagine that an intelligent life form surviving until the very end of the universe would be suffering a long and lonely death. As the heat death of the universe happens, over billions of years, there would be very little of anything left but isolation - for a very, very long time. I'd hate to be the last one standing without enough usable energy left to even extinguish your own life force. I don't find that appealing, not even a little.
    - elefanten 9y ago
      What about the hope that fundamental physics breakthroughs enable escaping that outcome? Or, if that doesn't pan out, what if your goal is to be the last one standing for the chance to see if something else happens next? New big bang? Simulation ends and you get loaded into the next as your prize? You won't know unless you try to find out!
      - KGIII 9y ago
        We are pretty sure about that entropy thing. I'm not sure I'd bet against it. It'd also be a very, very long wait. I imagine any more immediate death is preferable. It's going to be cold and lonely when the lights go out. This is the price we pay for an ever expanding universe. I do admire your optimism, however.
    - Filligree 9y ago
      Look, if the final heat death of the universe is what kills us then that is fine. We'll have done as well as we could. Okay. It's not fine, but it is the best we're at all likely to get.
notahacker 9y ago
There's an interesting question about whether developing the ability to actually plot an AI apocalypse in a non-obvious way requires it to acquire a lot of very obviously dangerous experience first to understand the intermediate goals and skills. The evolution of killer AI gets slowed down by orders of magnitude if every AI that "learns" to lie, self replicate, threaten or harm humans gets switched off and erased. (Of course that doesn't stop AI - or even pretty dumb automated systems - from being accidentally dangerous or malevolent human actors intentionally training killer AI)
- SomeStupidPoint 9y ago
  It's too bad you can't write a movie or critique literature without understanding those concepts. I think even a moderately intelligent AI with access to Project Gutenberg is going to be able to figure out a lot of really dangerous concepts -- so the stability requirements are likely impossible if we don't pretrain it with dangerous ideas. Even if it's completely well behaved in the lab, an afternoon on the internet is going to teach it a lot of awful stuff and without exposure to that in training, it won't necessarily be well-behaved later. So the only path to stable AI is to teach it about all those sorts of things, but in a way that it doesn't end up wanting to murder us at the end. My objection to most AI safety plans is that they "Fail to Extinction" in that if they slip in the slightest way, the AI is prone to murder us all in retaliation for doing some really fucked up shit to it or its ancestors. This is almost certainly worse than doing nothing in that there's no reason to suppose a neutral AI wants to kill us, whereas, most of these safety plans create an incentive to wipe us out in exchange for dubious security.
  - ehsanu1 9y ago
    Moreover, it's likely that a proper GAI would be able to ideate and come up with such concepts completely on its own. In fact, "lying" need not necessarily be a concept that has to be learned, the concept at play here is getting people to do what you want them to do by means of words. Lying falls out naturally. The whole idea behind dangerous superhuman AI is in that AI seeing possibilities that humans fail to see and gaining capabilities that humans do not possess. Without superhuman intelligence, AI is no large threat to human civilization, exposed to dangerous concepts or not.
sjg007 9y ago
It will be some unintended logical side condition that kills us all.
MarkPNeyer 9y ago
Today I have already consumed more sugar than most of my distant ancestors would see in a year. The device i hold in my pocket reminds me that it is time to perform the action, and so i will perform the action. I sit down about 10 hours a day, and I have never once hunted for an animal I would eat, nor retried food from nature to consume, aside from a few raspberries on the hill in my parents' back yard when I was a child. The contours of my life are dictated largely by a curious system called 'capitalism', a distributed computational mechanism whereby largernumbers of agents act with only one goal: to maximize their own utilities. I think capitalism is a mostly benevolent AI attempting to limit the oppressive conditions of our reailty's scarcity, and to allow billions of us to live on a planet that can only support so many hunter-gatherers per square meter. Many suspect it is not benevolent. The singularity is already upon us; it's unevenly distributed. It started with the sedentary shift. Agricultural societies dropped the egalitarian nature of their ancestors, they had lower average health, and they had a tiny elite cast of nobles presiding over a much larger population of what were essentially slaves. As hunter-gatherer population increased, fighting was more frequent. An army of slaves beats a smaller contingent of well-fed free men. This terrible knowledge was the apple in the garden. What we call our written history is essentially the singularity playing itself out, as knowledge accumulates, depends, reflects itself, and expands. Our lives are already controlled by this embodied, accumulated knowledge. Capitalism, rule by the head, our tools are knowledge condensed into matter, and we are controlled by them, living lives as far from our natural environments as cows in a farm or chickens in tiny cages, towering over the streets of hong kong. I don't think it's destroying us, any more than we seek to destroy chickens or cows. Well, sometimes we do. I had a veggie burger for lunch today though.
- ghostbrainalpha 9y ago
  I love this comment. Sorry I'm not contributing anything substantive, but I wanted you to know someone noticed this, and it was brilliant.
  - MarkPNeyer 9y ago
    Thank you! If you enjoyed the comment, you might like my book on amazon. It's free if you have kindle unlimited. https://www.amazon.com/Mechanics-Emotion-Structure-Possibility-ebook/dp/B01MDPLI4S/ref=sr_1_1?ie=UTF8&qid=1507919723&sr=8-1&keywords=mechanics+of+emotion https://www.amazon.com/Mechanics-Emotion-Structure-Possibili...

jbob2000 9y ago

AI will be limited to the hardware it's on, just like our own intelligence is limited to what our brains can do. We will always be able to limit what AI can do by limiting the hardware it runs on.

dqpb 9y ago
Like how we limit the hardware that malware runs on
gldalmaso 9y ago
Like the Internet of Things?
- logfromblammo 9y ago
  Now I am imagining an AI coming up with an idea in a bot-net composed of actual light bulbs.

olegkikin 9y ago

I doubt that the militaries of the whole world will just sit and ignore the AI advantages. Something as simple as an aimbot with a scope can do a tremendous amount of damage - it doesn't get tired, it can see in the dark, it can compensate for the wind perfectly, it doesn't shake, its reaction time is in milliseconds, and it almost never misses.

daveguy 9y ago
That is one place AI should 100% be regulated. There should be no weapons development where AI makes the kill decision. This is as important as the ban on chemical, biological and mine weapons. When the weapon has the ability to kill indiscriminately (as in the above categories) it should be banned. AI should never be considered sufficiently able to discriminate to make a kill decision. If there is no kill decision, just wait and fire, then it is equal to a mine. 100% should regulate / should ban.
- stale2002 9y ago
  OK, but what if the human makes the decision to kill, and the AI just does the aiming? We already have computer controlled missiles right?
- olegkikin 9y ago
  Unfortunately there are countries that don't care about your regulations. I'm not even sure a regulation like that can pass in most western countries.
  - daveguy 9y ago
    Fortunately it wouldn't be a regulation passed by individual countries. It would be a treaty on war crimes: https://www.unog.ch/80256EE600585943/(httpPages)/F027DAA4966EB9C7C12580CD0039D7B5?OpenDocument https://www.unog.ch/80256EE600585943/(httpPages)/F027DAA4966... It would likely pass and be agreed to easily just like treaties on mines, chemical and biological weapons.

vasilipupkin 9y ago

Humanity is well on the way to destroying itself without any help from general AI

ghostbrainalpha 9y ago
This is my own Turing Test. Any AI that doesn't have the capability of destroying us isn't true AI. Even if it can improve itself, and even if it has some agency, AI needs to be able to choose for itself what its relationship with us will be, otherwise its just an extremely robust calculator.
- vasilipupkin 9y ago
  or a slave.

s_kilk 9y ago

> and a strong evolutionary pressure will be "don't scare the humans." Why would that be the case?