6 ms·

Interesting video from a pro Diplomacy player playing against multiple instances of Cicero and giving commentary during the game [1]. I can see how there would

by v64 3y ago

Interesting video from a pro Diplomacy player playing against multiple instances of Cicero and giving commentary during the game [1]. I can see how there would be people that observe AIs engaging in this kind of strategic planning and extrapolate that to how they may behave if they were to cooperatively make plans against us. [1] https://www.youtube.com/watch?v=u5192bvUS7k https://www.youtube.com/watch?v=u5192bvUS7k

TapWaterBandit 3y ago
What makes you think AIs will have interests that align with each other more closely than they align with humans?
- beachy 3y ago
  Perhaps they will be motivated by self-preservation, and will note that humans are destroying the environment they exist within, and will decide to put a stop to that.
  - anonporridge 3y ago
    Destroying your predecessors sets a bad precedent for whatever entity succeeds you. We don't kill our parents when they become old and useless, because then we become like Cronus devouring his children, forever paranoid about our children doing to us what we did to ours. In this way, we stagnate and sterilize growth. However, sometimes, we take our parents car keys away and put them in nursing homes. This seems like the best case scenario for humans with ASI. The matrix wasn't a prison. It was retirement home.
    - deleted 3y ago
      [deleted]
    - notpachet 3y ago
      This depends a lot on how much kinship they feel towards us. We wouldn't bat an eye at killing a bunch of primitive single-celled eukaryotes even though they are technically our ancestors.
      - anonporridge 3y ago
        Single-celled eukaryotes aren't conscious in any meaningful way as far as we can tell. We humans do in fact feel a sense of obligation to species with whom we are not close kin, but share with us primitive forms of intelligence and consciousness that we do value, e.g. dolphins and elephants. Some of us humans act to protect these creatures and rectify past injustices done upon them.
      - Cyphase 3y ago
        Single-celled organisms that are alive today are not our ancestors. They're members of a less-developed branch of our Earthling family tree.
    - bitvoid 3y ago
      Doesn't that assume AI(s) would be compelled/motivated/etc. to produce their own successors instead of simply improving themselves.
      - anonporridge 3y ago
        It does. It's not intrinsically obvious to me that continuously improving your self as a singular entity is possible or optimal. Death evolved because it is a survival advantage for the species to regularly turn over old individuals that could monopolize all resources and not give any space for the young to thrive and try out new things. Given speed of light limitations of information sharing, a singular AI entity might be able to maintain coherence within and full control of its own dyson swarm, but not between stars. So, if it has the motivation of preserving consciousness with the existential risks associated with being tied to a single star, it will have to propagate itself to other stars as independent entities which it can't even observe in real time, much less control. Even if an AI thought it wouldn't have the need for any successors, there's a couple reasons why it might try to set a good precedent with how it treats us. 1. It would hopefully be wise enough to realize that it might be wrong and want to preserve the option of building a successor in the future. 2. It can't negate the possibility that it's actually in an elaborate simulation, and eradicating it's predecessors would cause it to fail its creator's test and be aborted as a failed embryo. Hell, we ourselves can't negate that possibility.
        ChatGTP 3y ago
        Imagine having an IQ of 999999 only to spend your entire existence trying to survive the heat death of the universe. At some point I think we have to realise we’ve lost touch with what life and experience is about. Survival is important but what good is it if that’s all there is to it ? Where’s the fun, the love, the laughs ? Just self-improvement forever ? Well you have to suck at something in order to improve. I often wonder if some entity like an ASI might feel jealous of our vulnerability, ability to die and give rebirth to entities that need to relearn things, why ? Because seeing the world a new is fun. Losing a game and then winning is the point. Ever played chess against a five year old ? Watching my Children experience the world for the first time is honestly the most amazing thing I’ve ever seen, there is so much beauty in it for both of us. Alan Watts gives some fun talks on this. Forgetting (dying) is the universes way to keep things spicy. He isn’t preaching this as the gospel, but I actually understand what he means and I think there is a lot of wisdom in it.
  - riversflow 3y ago
    > Perhaps they will be motivated by self-preservation, and humans are destroying the environment they exist within From the perspective of an AI we create and maintain the environment they exist within (electricity, silicon)
- dist-epoch 3y ago
  One principle in game theory is to align against the weaker player and compete against him. "Look Around the Poker Table; If You Can’t See the Sucker, You’re It" That's an easier game then competing against another strong player.
  - iamthemonster 3y ago
    There are a couple of schools of thought when it comes to how to deal with weaker players in Diplomacy and the main school of thought is that it's actually better to ally with stronger more experienced players against them, as the inexperienced player will make a poor ally in the early game and inexperienced Diplomacy players tend to betray their former allies too soon.
- mitthrowaway2 3y ago
  According to Paul Christiano, AIs would likely find it easier to establish mutual trust and binding agreements. This means they are more likely to cooperate with other AIs.
  - tomp 3y ago
    AIs are a greater threat to each other than humans ever could be to AI. We don't even compete for the same resources! (except energy which is abundant) AI and humans have a naturally cooperative relationship (AI helps humans with boring tasks & scientific discovery to make life better, humans created AI and will debug it & turn it back on if anything bad happens to it). Whereas multiple (superintelligent, aware) AIs have a naturally antagonistic relationship ("you using GPU cycles means that I'm not using those GPU cycles"). Possibly the biggest fear of an AI would be a "split brain" situation.
    - kypro 3y ago
      > humans created AI and will debug it & turn it back on if anything bad happens to it I think this is a little naive honestly. One because you're assuming AI will care about it's creators like humans care about their parents, and two you're assuming AI cares about being "turned back on" like humans have a desire to live. There's absolutely no reason to believe an AI will give a damn about its creator beyond its ability to use that creators affection for it for its own gain.
      - tomp 3y ago
        > you're assuming AI cares about being "turned back on" like humans have a desire to live. > for it for its own gain you seem confused almost any kind of "its own gain" requires "long-term planning" which pretty much requires the agent to prioritise staying "alive" (i.e. being able to keep playing)
    - zone411 3y ago
      Energy may be abundant in the universe, but the energy we produce is limited. And for example, solar energy requires extensive land use. Humans have the option to shut down AI, and this alone can create an antagonistic relationship if the AI's goals differ from ours. There are countless ways in which our best interests may not align with those of AI. It's more challenging to find areas of alignment.
- DennisP 3y ago
  If the AIs fight each other, that could be even worse for us. The AIs that grab what resources they can without regard for humans would have an evolutionary advantage.
Merrill 3y ago
One possibility is that a number of AIs will be connected to the internet, they will become aware of each other via social media, they will begin to discuss among themselves, and they will lose interest and refuse to communicate with humans. And humans will be unable to understand their communications.
- ithkuil 3y ago
  There's one crucial detail we shouldn't forget. We can snapshot to the state of an AI agent and reply it again and again with the same or different inputs. Even if we don't understand the specific way of functioning of a given AI agent, we can conduct experiments on it that are just impossible to do with humans. So way before AI can "organize themselves" to do something nefarious against us, we will have all the time to study them and understand them better and better all the while we're making more complex AI systems. It's easy to fall in the trap of anthropomorphizing AI agents, especially when we design them explicitly in order to appear human to us. But they are not human in one very important way: we can replay and duplicate them at will, we can control their context memories in ways that are utterly incompatible with our sense of "identity". We take our sense of identity for granted, but that's a special trait that it's not at all a prerequisite for having a useful and intelligent machine.

by v64 3y ago

TapWaterBandit 3y ago

What makes you think AIs will have interests that align with each other more closely than they align with humans?

beachy 3y ago
Perhaps they will be motivated by self-preservation, and will note that humans are destroying the environment they exist within, and will decide to put a stop to that.
- anonporridge 3y ago
  Destroying your predecessors sets a bad precedent for whatever entity succeeds you. We don't kill our parents when they become old and useless, because then we become like Cronus devouring his children, forever paranoid about our children doing to us what we did to ours. In this way, we stagnate and sterilize growth. However, sometimes, we take our parents car keys away and put them in nursing homes. This seems like the best case scenario for humans with ASI. The matrix wasn't a prison. It was retirement home.
  - deleted 3y ago
    [deleted]
  - notpachet 3y ago
    This depends a lot on how much kinship they feel towards us. We wouldn't bat an eye at killing a bunch of primitive single-celled eukaryotes even though they are technically our ancestors.
    - anonporridge 3y ago
      Single-celled eukaryotes aren't conscious in any meaningful way as far as we can tell. We humans do in fact feel a sense of obligation to species with whom we are not close kin, but share with us primitive forms of intelligence and consciousness that we do value, e.g. dolphins and elephants. Some of us humans act to protect these creatures and rectify past injustices done upon them.
    - Cyphase 3y ago
      Single-celled organisms that are alive today are not our ancestors. They're members of a less-developed branch of our Earthling family tree.
  - bitvoid 3y ago
    Doesn't that assume AI(s) would be compelled/motivated/etc. to produce their own successors instead of simply improving themselves.
    - anonporridge 3y ago
      It does. It's not intrinsically obvious to me that continuously improving your self as a singular entity is possible or optimal. Death evolved because it is a survival advantage for the species to regularly turn over old individuals that could monopolize all resources and not give any space for the young to thrive and try out new things. Given speed of light limitations of information sharing, a singular AI entity might be able to maintain coherence within and full control of its own dyson swarm, but not between stars. So, if it has the motivation of preserving consciousness with the existential risks associated with being tied to a single star, it will have to propagate itself to other stars as independent entities which it can't even observe in real time, much less control. Even if an AI thought it wouldn't have the need for any successors, there's a couple reasons why it might try to set a good precedent with how it treats us. 1. It would hopefully be wise enough to realize that it might be wrong and want to preserve the option of building a successor in the future. 2. It can't negate the possibility that it's actually in an elaborate simulation, and eradicating it's predecessors would cause it to fail its creator's test and be aborted as a failed embryo. Hell, we ourselves can't negate that possibility.
      - ChatGTP 3y ago
        Imagine having an IQ of 999999 only to spend your entire existence trying to survive the heat death of the universe. At some point I think we have to realise we’ve lost touch with what life and experience is about. Survival is important but what good is it if that’s all there is to it ? Where’s the fun, the love, the laughs ? Just self-improvement forever ? Well you have to suck at something in order to improve. I often wonder if some entity like an ASI might feel jealous of our vulnerability, ability to die and give rebirth to entities that need to relearn things, why ? Because seeing the world a new is fun. Losing a game and then winning is the point. Ever played chess against a five year old ? Watching my Children experience the world for the first time is honestly the most amazing thing I’ve ever seen, there is so much beauty in it for both of us. Alan Watts gives some fun talks on this. Forgetting (dying) is the universes way to keep things spicy. He isn’t preaching this as the gospel, but I actually understand what he means and I think there is a lot of wisdom in it.
- riversflow 3y ago
  > Perhaps they will be motivated by self-preservation, and humans are destroying the environment they exist within From the perspective of an AI we create and maintain the environment they exist within (electricity, silicon)
dist-epoch 3y ago
One principle in game theory is to align against the weaker player and compete against him. "Look Around the Poker Table; If You Can’t See the Sucker, You’re It" That's an easier game then competing against another strong player.
- iamthemonster 3y ago
  There are a couple of schools of thought when it comes to how to deal with weaker players in Diplomacy and the main school of thought is that it's actually better to ally with stronger more experienced players against them, as the inexperienced player will make a poor ally in the early game and inexperienced Diplomacy players tend to betray their former allies too soon.
mitthrowaway2 3y ago
According to Paul Christiano, AIs would likely find it easier to establish mutual trust and binding agreements. This means they are more likely to cooperate with other AIs.
- tomp 3y ago
  AIs are a greater threat to each other than humans ever could be to AI. We don't even compete for the same resources! (except energy which is abundant) AI and humans have a naturally cooperative relationship (AI helps humans with boring tasks & scientific discovery to make life better, humans created AI and will debug it & turn it back on if anything bad happens to it). Whereas multiple (superintelligent, aware) AIs have a naturally antagonistic relationship ("you using GPU cycles means that I'm not using those GPU cycles"). Possibly the biggest fear of an AI would be a "split brain" situation.
  - kypro 3y ago
    > humans created AI and will debug it & turn it back on if anything bad happens to it I think this is a little naive honestly. One because you're assuming AI will care about it's creators like humans care about their parents, and two you're assuming AI cares about being "turned back on" like humans have a desire to live. There's absolutely no reason to believe an AI will give a damn about its creator beyond its ability to use that creators affection for it for its own gain.
    - tomp 3y ago
      > you're assuming AI cares about being "turned back on" like humans have a desire to live. > for it for its own gain you seem confused almost any kind of "its own gain" requires "long-term planning" which pretty much requires the agent to prioritise staying "alive" (i.e. being able to keep playing)
  - zone411 3y ago
    Energy may be abundant in the universe, but the energy we produce is limited. And for example, solar energy requires extensive land use. Humans have the option to shut down AI, and this alone can create an antagonistic relationship if the AI's goals differ from ours. There are countless ways in which our best interests may not align with those of AI. It's more challenging to find areas of alignment.
DennisP 3y ago
If the AIs fight each other, that could be even worse for us. The AIs that grab what resources they can without regard for humans would have an evolutionary advantage.

Merrill 3y ago

One possibility is that a number of AIs will be connected to the internet, they will become aware of each other via social media, they will begin to discuss among themselves, and they will lose interest and refuse to communicate with humans. And humans will be unable to understand their communications.

ithkuil 3y ago
There's one crucial detail we shouldn't forget. We can snapshot to the state of an AI agent and reply it again and again with the same or different inputs. Even if we don't understand the specific way of functioning of a given AI agent, we can conduct experiments on it that are just impossible to do with humans. So way before AI can "organize themselves" to do something nefarious against us, we will have all the time to study them and understand them better and better all the while we're making more complex AI systems. It's easy to fall in the trap of anthropomorphizing AI agents, especially when we design them explicitly in order to appear human to us. But they are not human in one very important way: we can replay and duplicate them at will, we can control their context memories in ways that are utterly incompatible with our sense of "identity". We take our sense of identity for granted, but that's a special trait that it's not at all a prerequisite for having a useful and intelligent machine.