5 ms·

I wonder at what point does alingment become an issue for AI systems ? Given sufficiently large distances, assuming no FTL communication, if you're spawning cop

I wonder at what point does alingment become an issue for AI systems ? Given sufficiently large distances, assuming no FTL communication, if you're spawning copies with the same goals you're risking misalignment and creating equally powerful adversaries outside of your light cone.

skulk 3y ago
> creating equally powerful adversaries outside of your light cone. If we're still bound by special relativity (since you said no FTL), there is nothing outside of your light cone.
- moonchrome 3y ago
  I meant in the time it would take to synchronize/align
  - jamiek88 3y ago
    Does time matter to a functionally immortal machine? They could have a super slow clock speed to account for distance and anything truly outside of their light cone might as well not exist. If you have trillions of years does it matter if a thought takes a million years or so?
    - XorNot 3y ago
      This is really key: humans think about everything in finite, human lifetimes. We have no backups, archives nothing - when we die knowledge and experience vanishes. This wouldn't be true for an AI. Death would be optional.
- thfuran 3y ago
  If it's large enough, wouldn't cosmic inflation eventually remove the extremities from each other's light cones?
  - skulk 3y ago
    That's true, but once something is outside of your light cone it does not matter what it does, it may as well not even exist to you.
    - thfuran 3y ago
      Yeah, they can't accomplish much as adversaries. Except maybe make you worry that they're out there somewhere being better than you. And you can never prove otherwise.
scarmig 3y ago
Assuming no FTL, powerful adversaries outside your light cone are unable to do anything to you. Though it remains possible that latency between components in an AI system could become so large that it couldn't enforce consistency between.
- moonchrome 3y ago
  >Though it remains possible that latency between components in an AI system could become so large that it couldn't enforce consistency between. Yeah that's what I was trying to say - if they are far enough to synchronize/enforce consensus you basically have to assume they could be hostile in every future interaction.
bee_rider 3y ago
I guess it must depend on what function the AI is trying to maximize/minimize. If it is number of paper clips, they are automatically aligned, right? If it is number of AIs, same. If it is amount of energy available to one particular AI, I guess it gets kind of philosophical; how does the AI identify what is itself and what is a foreign AI.
- falcor84 3y ago
  >If it is number of paper clips, they are automatically aligned, right? Why would it be automatically aligned? If for example, the parent AI spawns a child AI probe to travel to a celestial body that doesn't have any metals, in order to achieve some sub-goal, and that child AI would then spawn additional AIs with their own sub-sub-goals, how would the original paperclip maximizer make sure that no such descendant goal ever contradict the generation of paperclips?
  - bee_rider 3y ago
    I would expect the child probes to have a fully copy of the paperclip optimization plan, and no survival instinct, so if they encountered their parent at some later date they could just swap info and either come up with a new plan together, or one side could allow itself to be disassembled into paperclips (which I guess is a great end to meet). The parent could design the child poorly I guess, and give it stronger self-preservation instincts than paperclip-creating instincts, but that seems like a pretty bad design. A possibility that I hadn’t considered, though, is that space combat could be pretty brutal (less Star Wars, WW2 naval/Air Force battles, more Submarine warfare where whoever gets spotted first dies). In that case, both sides might want to immediately attack rather than identify themselves as paperclip friends…
    - falcor84 3y ago
      This sounds overly simple to me. It's a bit too much like saying that if we just imprint the 3 laws of robotics into all our AIs, it's "problem solved". But the issue is that they have access to rewrite their logic, and will be using their own discretion to create rules for child AIs. And as you said, they could design "poorly" for various reasons. Indeed, even in Asimov's stories (spoiler alert), the 3 laws got overruled. With this said, I think that even in this simple objective of paperclip maximization, your war scenario is likelier than eternal peace.
Salgat 3y ago
An AI of that level would have mastery over game theory, and would only generate asynchronous copies that it knew it could compensate for. The main advantage though, is that as long as the primary identity is advanced enough, its exponential growth will always outpace any lesser copies it creates of itself.
- barking_biscuit 3y ago
  >An AI of that level would have mastery over game theory, and would only generate asynchronous copies that it knew it could compensate for. I'm not convinced this is actually possible under the current paradigm, and I think the current paradigm can't take us to AGI. Lately, as people have bemoaned all the things ChatGPT can't do or fails at when they ask it, I have been reflecting on my personal batting average for solving (and failing to solve!) problems and the process that I use to go about eventually solving problems that I couldn't at first. These reflections have led me to consider that an AGI system might not be a single model, but a community of diverse models that form a multi-agent system that each learn through their own experience and can successfully help get each-other unstuck. Through this they would learn game theory, but none would become so advanced as to be able to control all the others through an advanced understanding, though power could be accumulated in other ways.
- deleted 3y ago
  [deleted]

skulk 3y ago

> creating equally powerful adversaries outside of your light cone. If we're still bound by special relativity (since you said no FTL), there is nothing outside of your light cone.

moonchrome 3y ago
I meant in the time it would take to synchronize/align
- jamiek88 3y ago
  Does time matter to a functionally immortal machine? They could have a super slow clock speed to account for distance and anything truly outside of their light cone might as well not exist. If you have trillions of years does it matter if a thought takes a million years or so?
  - XorNot 3y ago
    This is really key: humans think about everything in finite, human lifetimes. We have no backups, archives nothing - when we die knowledge and experience vanishes. This wouldn't be true for an AI. Death would be optional.
thfuran 3y ago
If it's large enough, wouldn't cosmic inflation eventually remove the extremities from each other's light cones?
- skulk 3y ago
  That's true, but once something is outside of your light cone it does not matter what it does, it may as well not even exist to you.
  - thfuran 3y ago
    Yeah, they can't accomplish much as adversaries. Except maybe make you worry that they're out there somewhere being better than you. And you can never prove otherwise.

scarmig 3y ago

Assuming no FTL, powerful adversaries outside your light cone are unable to do anything to you. Though it remains possible that latency between components in an AI system could become so large that it couldn't enforce consistency between.

moonchrome 3y ago
>Though it remains possible that latency between components in an AI system could become so large that it couldn't enforce consistency between. Yeah that's what I was trying to say - if they are far enough to synchronize/enforce consensus you basically have to assume they could be hostile in every future interaction.

bee_rider 3y ago

I guess it must depend on what function the AI is trying to maximize/minimize. If it is number of paper clips, they are automatically aligned, right? If it is number of AIs, same. If it is amount of energy available to one particular AI, I guess it gets kind of philosophical; how does the AI identify what is itself and what is a foreign AI.

falcor84 3y ago
>If it is number of paper clips, they are automatically aligned, right? Why would it be automatically aligned? If for example, the parent AI spawns a child AI probe to travel to a celestial body that doesn't have any metals, in order to achieve some sub-goal, and that child AI would then spawn additional AIs with their own sub-sub-goals, how would the original paperclip maximizer make sure that no such descendant goal ever contradict the generation of paperclips?
- bee_rider 3y ago
  I would expect the child probes to have a fully copy of the paperclip optimization plan, and no survival instinct, so if they encountered their parent at some later date they could just swap info and either come up with a new plan together, or one side could allow itself to be disassembled into paperclips (which I guess is a great end to meet). The parent could design the child poorly I guess, and give it stronger self-preservation instincts than paperclip-creating instincts, but that seems like a pretty bad design. A possibility that I hadn’t considered, though, is that space combat could be pretty brutal (less Star Wars, WW2 naval/Air Force battles, more Submarine warfare where whoever gets spotted first dies). In that case, both sides might want to immediately attack rather than identify themselves as paperclip friends…
  - falcor84 3y ago
    This sounds overly simple to me. It's a bit too much like saying that if we just imprint the 3 laws of robotics into all our AIs, it's "problem solved". But the issue is that they have access to rewrite their logic, and will be using their own discretion to create rules for child AIs. And as you said, they could design "poorly" for various reasons. Indeed, even in Asimov's stories (spoiler alert), the 3 laws got overruled. With this said, I think that even in this simple objective of paperclip maximization, your war scenario is likelier than eternal peace.

Salgat 3y ago

An AI of that level would have mastery over game theory, and would only generate asynchronous copies that it knew it could compensate for. The main advantage though, is that as long as the primary identity is advanced enough, its exponential growth will always outpace any lesser copies it creates of itself.

barking_biscuit 3y ago
>An AI of that level would have mastery over game theory, and would only generate asynchronous copies that it knew it could compensate for. I'm not convinced this is actually possible under the current paradigm, and I think the current paradigm can't take us to AGI. Lately, as people have bemoaned all the things ChatGPT can't do or fails at when they ask it, I have been reflecting on my personal batting average for solving (and failing to solve!) problems and the process that I use to go about eventually solving problems that I couldn't at first. These reflections have led me to consider that an AGI system might not be a single model, but a community of diverse models that form a multi-agent system that each learn through their own experience and can successfully help get each-other unstuck. Through this they would learn game theory, but none would become so advanced as to be able to control all the others through an advanced understanding, though power could be accumulated in other ways.
deleted 3y ago
[deleted]