4 ms·

"Don't scare the humans" is an insufficient criterion if you're trying to prevent an AI apocalypse. It just means that the AI that destroys us won't be obviousl

"Don't scare the humans" is an insufficient criterion if you're trying to prevent an AI apocalypse. It just means that the AI that destroys us won't be obviously dangerous.

Twirrim 9y ago
Given sufficient effort, I'm sure an AI out to destroy us could figure out a way to do it without killing us. It very likely even has the advantage of longevity. Harking back to Asimov, and thinking about the way Spacers changed, as they became more and more reliant on robots, it could very well mother us to extinction.
- Filligree 9y ago
  That's only a little better. I don't want to go extinct at all.
  - gldalmaso 9y ago
    At the moment and for the foreseeable future we are at a much greater risk of extincting ourselves.
    - FeepingCreature 9y ago
      The trick is to not do it at all. It does us no good to avoid causing our own extinction if it's then caused by something else.
  - state_less 9y ago
    Humanity is already extinct in a sense. Humanity is no longer limited to flesh and bone. I've personally seen flesh and titanium. And more variation is on the way. So humanity as it were, is already extinct. We (you and me) will go extinct someday. Our bodies were always destined toward this terminus... Perhaps by some miraculous tech, we can someday change form and shed the flesh and bone. Maybe then we'll have a better idea of what we are.
    - FeepingCreature 9y ago
      "We could thus imagine, as an extreme case, a technologically highly advanced society, containing many complex structures, some of them far more intricate and intelligent than anything that exists on the planet today — a society which nevertheless lacks any type of being that is conscious or whose welfare has moral significance. In a sense, this would be an uninhabited society. It would be a society of economic miracles and technological awesomeness, with nobody there to benefit. A Disneyland without children." --Nick Bostrom, Superintelligence: Paths, Dangers, Strategies It's not a given that anything of value will survive.
      - TeMPOraL 9y ago
        C.f. Meditations on Moloch. We are on our way towards exactly that. http://slatestarcodex.com/2014/07/30/meditations-on-moloch/ http://slatestarcodex.com/2014/07/30/meditations-on-moloch/
  - KGIII 9y ago
    That is going to happen, regardless. Even if we figure out all sorts of things, entropy is still the final boss of the universe. I'd imagine that an intelligent life form surviving until the very end of the universe would be suffering a long and lonely death. As the heat death of the universe happens, over billions of years, there would be very little of anything left but isolation - for a very, very long time. I'd hate to be the last one standing without enough usable energy left to even extinguish your own life force. I don't find that appealing, not even a little.
    - elefanten 9y ago
      What about the hope that fundamental physics breakthroughs enable escaping that outcome? Or, if that doesn't pan out, what if your goal is to be the last one standing for the chance to see if something else happens next? New big bang? Simulation ends and you get loaded into the next as your prize? You won't know unless you try to find out!
      - KGIII 9y ago
        We are pretty sure about that entropy thing. I'm not sure I'd bet against it. It'd also be a very, very long wait. I imagine any more immediate death is preferable. It's going to be cold and lonely when the lights go out. This is the price we pay for an ever expanding universe. I do admire your optimism, however.
    - Filligree 9y ago
      Look, if the final heat death of the universe is what kills us then that is fine. We'll have done as well as we could. Okay. It's not fine, but it is the best we're at all likely to get.
notahacker 9y ago
There's an interesting question about whether developing the ability to actually plot an AI apocalypse in a non-obvious way requires it to acquire a lot of very obviously dangerous experience first to understand the intermediate goals and skills. The evolution of killer AI gets slowed down by orders of magnitude if every AI that "learns" to lie, self replicate, threaten or harm humans gets switched off and erased. (Of course that doesn't stop AI - or even pretty dumb automated systems - from being accidentally dangerous or malevolent human actors intentionally training killer AI)
- SomeStupidPoint 9y ago
  It's too bad you can't write a movie or critique literature without understanding those concepts. I think even a moderately intelligent AI with access to Project Gutenberg is going to be able to figure out a lot of really dangerous concepts -- so the stability requirements are likely impossible if we don't pretrain it with dangerous ideas. Even if it's completely well behaved in the lab, an afternoon on the internet is going to teach it a lot of awful stuff and without exposure to that in training, it won't necessarily be well-behaved later. So the only path to stable AI is to teach it about all those sorts of things, but in a way that it doesn't end up wanting to murder us at the end. My objection to most AI safety plans is that they "Fail to Extinction" in that if they slip in the slightest way, the AI is prone to murder us all in retaliation for doing some really fucked up shit to it or its ancestors. This is almost certainly worse than doing nothing in that there's no reason to suppose a neutral AI wants to kill us, whereas, most of these safety plans create an incentive to wipe us out in exchange for dubious security.
  - ehsanu1 9y ago
    Moreover, it's likely that a proper GAI would be able to ideate and come up with such concepts completely on its own. In fact, "lying" need not necessarily be a concept that has to be learned, the concept at play here is getting people to do what you want them to do by means of words. Lying falls out naturally. The whole idea behind dangerous superhuman AI is in that AI seeing possibilities that humans fail to see and gaining capabilities that humans do not possess. Without superhuman intelligence, AI is no large threat to human civilization, exposed to dangerous concepts or not.
sjg007 9y ago
It will be some unintended logical side condition that kills us all.
MarkPNeyer 9y ago
Today I have already consumed more sugar than most of my distant ancestors would see in a year. The device i hold in my pocket reminds me that it is time to perform the action, and so i will perform the action. I sit down about 10 hours a day, and I have never once hunted for an animal I would eat, nor retried food from nature to consume, aside from a few raspberries on the hill in my parents' back yard when I was a child. The contours of my life are dictated largely by a curious system called 'capitalism', a distributed computational mechanism whereby largernumbers of agents act with only one goal: to maximize their own utilities. I think capitalism is a mostly benevolent AI attempting to limit the oppressive conditions of our reailty's scarcity, and to allow billions of us to live on a planet that can only support so many hunter-gatherers per square meter. Many suspect it is not benevolent. The singularity is already upon us; it's unevenly distributed. It started with the sedentary shift. Agricultural societies dropped the egalitarian nature of their ancestors, they had lower average health, and they had a tiny elite cast of nobles presiding over a much larger population of what were essentially slaves. As hunter-gatherer population increased, fighting was more frequent. An army of slaves beats a smaller contingent of well-fed free men. This terrible knowledge was the apple in the garden. What we call our written history is essentially the singularity playing itself out, as knowledge accumulates, depends, reflects itself, and expands. Our lives are already controlled by this embodied, accumulated knowledge. Capitalism, rule by the head, our tools are knowledge condensed into matter, and we are controlled by them, living lives as far from our natural environments as cows in a farm or chickens in tiny cages, towering over the streets of hong kong. I don't think it's destroying us, any more than we seek to destroy chickens or cows. Well, sometimes we do. I had a veggie burger for lunch today though.
- ghostbrainalpha 9y ago
  I love this comment. Sorry I'm not contributing anything substantive, but I wanted you to know someone noticed this, and it was brilliant.
  - MarkPNeyer 9y ago
    Thank you! If you enjoyed the comment, you might like my book on amazon. It's free if you have kindle unlimited. https://www.amazon.com/Mechanics-Emotion-Structure-Possibility-ebook/dp/B01MDPLI4S/ref=sr_1_1?ie=UTF8&qid=1507919723&sr=8-1&keywords=mechanics+of+emotion https://www.amazon.com/Mechanics-Emotion-Structure-Possibili...

"Don't scare the humans" is an insufficient criterion if you're trying to prevent an AI apocalypse. It just means that the AI that destroys us won't be obviously dangerous.

Twirrim 9y ago

Given sufficient effort, I'm sure an AI out to destroy us could figure out a way to do it without killing us. It very likely even has the advantage of longevity. Harking back to Asimov, and thinking about the way Spacers changed, as they became more and more reliant on robots, it could very well mother us to extinction.

Filligree 9y ago
That's only a little better. I don't want to go extinct at all.
- gldalmaso 9y ago
  At the moment and for the foreseeable future we are at a much greater risk of extincting ourselves.
  - FeepingCreature 9y ago
    The trick is to not do it at all. It does us no good to avoid causing our own extinction if it's then caused by something else.
- state_less 9y ago
  Humanity is already extinct in a sense. Humanity is no longer limited to flesh and bone. I've personally seen flesh and titanium. And more variation is on the way. So humanity as it were, is already extinct. We (you and me) will go extinct someday. Our bodies were always destined toward this terminus... Perhaps by some miraculous tech, we can someday change form and shed the flesh and bone. Maybe then we'll have a better idea of what we are.
  - FeepingCreature 9y ago
    "We could thus imagine, as an extreme case, a technologically highly advanced society, containing many complex structures, some of them far more intricate and intelligent than anything that exists on the planet today — a society which nevertheless lacks any type of being that is conscious or whose welfare has moral significance. In a sense, this would be an uninhabited society. It would be a society of economic miracles and technological awesomeness, with nobody there to benefit. A Disneyland without children." --Nick Bostrom, Superintelligence: Paths, Dangers, Strategies It's not a given that anything of value will survive.
    - TeMPOraL 9y ago
      C.f. Meditations on Moloch. We are on our way towards exactly that. http://slatestarcodex.com/2014/07/30/meditations-on-moloch/ http://slatestarcodex.com/2014/07/30/meditations-on-moloch/
- KGIII 9y ago
  That is going to happen, regardless. Even if we figure out all sorts of things, entropy is still the final boss of the universe. I'd imagine that an intelligent life form surviving until the very end of the universe would be suffering a long and lonely death. As the heat death of the universe happens, over billions of years, there would be very little of anything left but isolation - for a very, very long time. I'd hate to be the last one standing without enough usable energy left to even extinguish your own life force. I don't find that appealing, not even a little.
  - elefanten 9y ago
    What about the hope that fundamental physics breakthroughs enable escaping that outcome? Or, if that doesn't pan out, what if your goal is to be the last one standing for the chance to see if something else happens next? New big bang? Simulation ends and you get loaded into the next as your prize? You won't know unless you try to find out!
    - KGIII 9y ago
      We are pretty sure about that entropy thing. I'm not sure I'd bet against it. It'd also be a very, very long wait. I imagine any more immediate death is preferable. It's going to be cold and lonely when the lights go out. This is the price we pay for an ever expanding universe. I do admire your optimism, however.
  - Filligree 9y ago
    Look, if the final heat death of the universe is what kills us then that is fine. We'll have done as well as we could. Okay. It's not fine, but it is the best we're at all likely to get.

notahacker 9y ago

There's an interesting question about whether developing the ability to actually plot an AI apocalypse in a non-obvious way requires it to acquire a lot of very obviously dangerous experience first to understand the intermediate goals and skills. The evolution of killer AI gets slowed down by orders of magnitude if every AI that "learns" to lie, self replicate, threaten or harm humans gets switched off and erased. (Of course that doesn't stop AI - or even pretty dumb automated systems - from being accidentally dangerous or malevolent human actors intentionally training killer AI)

SomeStupidPoint 9y ago
It's too bad you can't write a movie or critique literature without understanding those concepts. I think even a moderately intelligent AI with access to Project Gutenberg is going to be able to figure out a lot of really dangerous concepts -- so the stability requirements are likely impossible if we don't pretrain it with dangerous ideas. Even if it's completely well behaved in the lab, an afternoon on the internet is going to teach it a lot of awful stuff and without exposure to that in training, it won't necessarily be well-behaved later. So the only path to stable AI is to teach it about all those sorts of things, but in a way that it doesn't end up wanting to murder us at the end. My objection to most AI safety plans is that they "Fail to Extinction" in that if they slip in the slightest way, the AI is prone to murder us all in retaliation for doing some really fucked up shit to it or its ancestors. This is almost certainly worse than doing nothing in that there's no reason to suppose a neutral AI wants to kill us, whereas, most of these safety plans create an incentive to wipe us out in exchange for dubious security.
- ehsanu1 9y ago
  Moreover, it's likely that a proper GAI would be able to ideate and come up with such concepts completely on its own. In fact, "lying" need not necessarily be a concept that has to be learned, the concept at play here is getting people to do what you want them to do by means of words. Lying falls out naturally. The whole idea behind dangerous superhuman AI is in that AI seeing possibilities that humans fail to see and gaining capabilities that humans do not possess. Without superhuman intelligence, AI is no large threat to human civilization, exposed to dangerous concepts or not.

sjg007 9y ago

It will be some unintended logical side condition that kills us all.

MarkPNeyer 9y ago

Today I have already consumed more sugar than most of my distant ancestors would see in a year. The device i hold in my pocket reminds me that it is time to perform the action, and so i will perform the action. I sit down about 10 hours a day, and I have never once hunted for an animal I would eat, nor retried food from nature to consume, aside from a few raspberries on the hill in my parents' back yard when I was a child. The contours of my life are dictated largely by a curious system called 'capitalism', a distributed computational mechanism whereby largernumbers of agents act with only one goal: to maximize their own utilities. I think capitalism is a mostly benevolent AI attempting to limit the oppressive conditions of our reailty's scarcity, and to allow billions of us to live on a planet that can only support so many hunter-gatherers per square meter. Many suspect it is not benevolent. The singularity is already upon us; it's unevenly distributed. It started with the sedentary shift. Agricultural societies dropped the egalitarian nature of their ancestors, they had lower average health, and they had a tiny elite cast of nobles presiding over a much larger population of what were essentially slaves. As hunter-gatherer population increased, fighting was more frequent. An army of slaves beats a smaller contingent of well-fed free men. This terrible knowledge was the apple in the garden. What we call our written history is essentially the singularity playing itself out, as knowledge accumulates, depends, reflects itself, and expands. Our lives are already controlled by this embodied, accumulated knowledge. Capitalism, rule by the head, our tools are knowledge condensed into matter, and we are controlled by them, living lives as far from our natural environments as cows in a farm or chickens in tiny cages, towering over the streets of hong kong. I don't think it's destroying us, any more than we seek to destroy chickens or cows. Well, sometimes we do. I had a veggie burger for lunch today though.

ghostbrainalpha 9y ago
I love this comment. Sorry I'm not contributing anything substantive, but I wanted you to know someone noticed this, and it was brilliant.
- MarkPNeyer 9y ago
  Thank you! If you enjoyed the comment, you might like my book on amazon. It's free if you have kindle unlimited. https://www.amazon.com/Mechanics-Emotion-Structure-Possibility-ebook/dp/B01MDPLI4S/ref=sr_1_1?ie=UTF8&qid=1507919723&sr=8-1&keywords=mechanics+of+emotion https://www.amazon.com/Mechanics-Emotion-Structure-Possibili...