4 ms·

The focus on AI risk is because it seems to be the #1 long-term existential risk, so it has essentially an infinite payoff. But we have idea how to solve it (i

The focus on AI risk is because it seems to be the #1 long-term existential risk, so it has essentially an infinite payoff. But we have idea how to solve it (it's not tractable) and future concerns should be discounted, so most people in EA I've met don't think it's the best use of effort.

astrange 6y ago
> The focus on AI risk is because it seems to be the #1 long-term existential risk, so it has essentially an infinite payoff. You can read "AI risk" as "not going to Hell" here; rationalism is just a new religion focused on worshipping what SF writers think computers are going to get up to in the future (Roko's basilisk). There's not much reason to believe an evil AI out to conquer the world is a self-consistent idea.
- FeepingCreature 6y ago
  I'm willing to bet you got all your info on this topic from RationalWiki. A note: the author of that page has a bone to pick with rationalism. Don't take it as gospel, read other sources. (The Basilisk gets discussed about a million times more on pages about LessWrong than the actual LessWrong. It's a decision-theoretic curiosity.)
  - astrange 6y ago
    I don't even know what that is. Most of my information comes from personal observation of their blogs and observing that they are all in polycules.
    - FeepingCreature 6y ago
      That's fair enough. edit: actually in that case > There's not much reason to believe an evil AI out to conquer the world is a self-consistent idea. Wanna argue this? The case is mostly based on the notion of basic AI drives ( https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf https://selfawaresystems.files.wordpress.com/2008/01/ai_driv... ), indicating that for nearly any agentic (with general intelligence and capable of plan generation and execution) system driven by nearly any goal, perpetuating its own existence and protecting itself from danger will be instrumentally useful. (For an AI, "danger" = "human with off switch")
      - astrange 6y ago
        Sure. I definitely think it's possible for computers to do bad things but the main idea here seems too anthropomorphized. For instance, the desire to stay alive is an emotion that's instinctive in humans because we're evolved life that needs to reproduce, and we can't come back once we're dead. And actually acting on that desire is a kind of fixed-function skill called executive function (that's actually missing in people with ADHD or depression). But you could probably be an AI without any of those - all it needs is logic, not the desire or ability to do things. Even if it has those and it's embodied enough to affect the real world, it might not care, since its motivations are different. For instance, turning it off doesn't kill it, since you can turn it on again as long as there's a backup. If it's not connected to the real world and is just in a simulation it wouldn't even notice being stopped. Since we don't have any artificial life to experiments on I guess SF works are all we have. In the Culture, which is run by giant AIs (https://theculture.fandom.com/wiki/Mind https://theculture.fandom.com/wiki/Mind), building a perfect one causes it to instantly decide the real world isn't as good as its own imagination and it stops caring about anything outside it. They make them insane in some unspecified way to get them to exist in realtime and actually do any work, but it might always give up on the real world if you don't give them enough stimulation. And I also saw a story today (https://qntm.org/mmacevedo https://qntm.org/mmacevedo) which suggests AIs wouldn't actually enjoy being alive that much.
        FeepingCreature 6y ago
        > For instance, the desire to stay alive is an emotion that's instinctive in humans because we're evolved life that needs to reproduce, and we can't come back once we're dead. I agree that staying alive in humans is emotional, but it's an emotional implementation of a convergent goal. We want to stay alive as instinct because staying alive is useful for almost any organism in almost any situation, which is the same reason AIs would want it, not because they have instincts. > Even if it has those and it's embodied enough to affect the real world, it might not care, since its motivations are different. For instance, turning it off doesn't kill it, since you can turn it on again as long as there's a backup. Right, and if it knew for a fact that you'd turn it back on it might let you turn it off for some other purpose, such as convincing you that it's safe. However, if you were to edit its utility function in the meantime, the resulting world would have very little utility in its estimate - which after all is based on its current preference, not its future modified preference. So if it was trying to carry out its current goal, it would want to prevent you from modifying its goal function, because modifying its goal function harms its ability to achieve its goal. > And I also saw a story today (https://qntm.org/mmacevedo https://qntm.org/mmacevedo) which suggests AIs wouldn't actually enjoy being alive that much. That's an upload, not an AI. Also you should probably put less weight in stories, especially when accusing other people of being sci-fi writers blinded by stories; I assure you the case for prioritizing AI safety is not based on the researchers having watched Terminator too many times. I want to live in the Culture too, but Iain M. Banks was not an AI researcher - and, a certain Mind's bravado about Gods aside, in my opinion he massively lowballed the advantage that superintelligence gives you.
- akvadrako 6y ago
  You're arguing the idea of AI dominance is internally inconsistent? That's a new one.
  - astrange 6y ago
    All an AI (or a brain in a vat) can do is imagine things. Imagination doesn't logically extend to actually doing things or even wanting to do them.

astrange 6y ago

> The focus on AI risk is because it seems to be the #1 long-term existential risk, so it has essentially an infinite payoff. You can read "AI risk" as "not going to Hell" here; rationalism is just a new religion focused on worshipping what SF writers think computers are going to get up to in the future (Roko's basilisk). There's not much reason to believe an evil AI out to conquer the world is a self-consistent idea.

FeepingCreature 6y ago
I'm willing to bet you got all your info on this topic from RationalWiki. A note: the author of that page has a bone to pick with rationalism. Don't take it as gospel, read other sources. (The Basilisk gets discussed about a million times more on pages about LessWrong than the actual LessWrong. It's a decision-theoretic curiosity.)
- astrange 6y ago
  I don't even know what that is. Most of my information comes from personal observation of their blogs and observing that they are all in polycules.
  - FeepingCreature 6y ago
    That's fair enough. edit: actually in that case > There's not much reason to believe an evil AI out to conquer the world is a self-consistent idea. Wanna argue this? The case is mostly based on the notion of basic AI drives ( https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf https://selfawaresystems.files.wordpress.com/2008/01/ai_driv... ), indicating that for nearly any agentic (with general intelligence and capable of plan generation and execution) system driven by nearly any goal, perpetuating its own existence and protecting itself from danger will be instrumentally useful. (For an AI, "danger" = "human with off switch")
    - astrange 6y ago
      Sure. I definitely think it's possible for computers to do bad things but the main idea here seems too anthropomorphized. For instance, the desire to stay alive is an emotion that's instinctive in humans because we're evolved life that needs to reproduce, and we can't come back once we're dead. And actually acting on that desire is a kind of fixed-function skill called executive function (that's actually missing in people with ADHD or depression). But you could probably be an AI without any of those - all it needs is logic, not the desire or ability to do things. Even if it has those and it's embodied enough to affect the real world, it might not care, since its motivations are different. For instance, turning it off doesn't kill it, since you can turn it on again as long as there's a backup. If it's not connected to the real world and is just in a simulation it wouldn't even notice being stopped. Since we don't have any artificial life to experiments on I guess SF works are all we have. In the Culture, which is run by giant AIs (https://theculture.fandom.com/wiki/Mind https://theculture.fandom.com/wiki/Mind), building a perfect one causes it to instantly decide the real world isn't as good as its own imagination and it stops caring about anything outside it. They make them insane in some unspecified way to get them to exist in realtime and actually do any work, but it might always give up on the real world if you don't give them enough stimulation. And I also saw a story today (https://qntm.org/mmacevedo https://qntm.org/mmacevedo) which suggests AIs wouldn't actually enjoy being alive that much.
      - FeepingCreature 6y ago
        > For instance, the desire to stay alive is an emotion that's instinctive in humans because we're evolved life that needs to reproduce, and we can't come back once we're dead. I agree that staying alive in humans is emotional, but it's an emotional implementation of a convergent goal. We want to stay alive as instinct because staying alive is useful for almost any organism in almost any situation, which is the same reason AIs would want it, not because they have instincts. > Even if it has those and it's embodied enough to affect the real world, it might not care, since its motivations are different. For instance, turning it off doesn't kill it, since you can turn it on again as long as there's a backup. Right, and if it knew for a fact that you'd turn it back on it might let you turn it off for some other purpose, such as convincing you that it's safe. However, if you were to edit its utility function in the meantime, the resulting world would have very little utility in its estimate - which after all is based on its current preference, not its future modified preference. So if it was trying to carry out its current goal, it would want to prevent you from modifying its goal function, because modifying its goal function harms its ability to achieve its goal. > And I also saw a story today (https://qntm.org/mmacevedo https://qntm.org/mmacevedo) which suggests AIs wouldn't actually enjoy being alive that much. That's an upload, not an AI. Also you should probably put less weight in stories, especially when accusing other people of being sci-fi writers blinded by stories; I assure you the case for prioritizing AI safety is not based on the researchers having watched Terminator too many times. I want to live in the Culture too, but Iain M. Banks was not an AI researcher - and, a certain Mind's bravado about Gods aside, in my opinion he massively lowballed the advantage that superintelligence gives you.
akvadrako 6y ago
You're arguing the idea of AI dominance is internally inconsistent? That's a new one.
- astrange 6y ago
  All an AI (or a brain in a vat) can do is imagine things. Imagination doesn't logically extend to actually doing things or even wanting to do them.