4 ms·

Even if AI doesn't reach a point where it could wipe us out, those rosy predictions might just be wishful thinking. Our society is built on cooperation outperfo

Even if AI doesn't reach a point where it could wipe us out, those rosy predictions might just be wishful thinking. Our society is built on cooperation outperforming individuals. And that whole iterated prisoner's dilemma thing keeps us in check. But if someone creates an AI that's super capable and can replace people ? Add advanced robots to the mix. Suddenly, they don't really need other people, and they could even be a threat to control over AI resources. In that case, there's nothing to gain from sharing and everything to lose. Even if they choose not to wipe out the rest of us, we are basically their pets, we can never reach the same level of development without posing an existential threat. I'm not sure what nuclear deterrent looks like when you have AGI resources. These optimistic ideas about AI improving our lives sound like hoping for the best.

chr1 4y ago
> I'm not sure what nuclear deterrent looks like when you have AGI resources. Other people who have AGI resources. That's why it is very important to have multiple groups of people who will reach AGI at the same time. That way the same iterated prisoners dilemma thing will act on AGIs too.
- moonchrome 4y ago
  I'm not sure how that doesn't devolve into an AGI armageddon. Those self improving AGIs would have to constantly be on equal level to avoid takeover, and they have an existential imperative to one up each other. There's no cooperation incentive between them just a potential stalemate. Also the stalemate covers the people in control of AGIs (assuming they aren't acting independently) - the rest of us are just resources in breaking the stalemate game. This sounds like the best way to build paperclip maximizer.
  - boringuser2 4y ago
    It's pretty clear that at some point in the process of recursive self-improvement that you'd have literally just invented God, whereafter all predictions break down.
- mrob 4y ago
  The more groups have AGI, the higher the chance that one of them figures out recursive self-improvement, and if that happens it's game over for everybody. Iterated prisoner's dilemma only works when all the prisoners have roughly the same power.
  - chr1 4y ago
    Why is it game over though? AI is already used for self-improvement, and will be used more as it becomes more capable, but that won't magically solve all the problems in the world and create omnipotent AI. If there are many people and many AI's they collectively will be stronger than the one most advanced, and cooperation will remain advantageous.
antegamisou 4y ago
Very accurate take, one should expect the worst from such advancements under capitalist economy - societal structure.
- moonchrome 4y ago
  More fundamental than capitalism I'd say, it's evolutionary.

chr1 4y ago

> I'm not sure what nuclear deterrent looks like when you have AGI resources. Other people who have AGI resources. That's why it is very important to have multiple groups of people who will reach AGI at the same time. That way the same iterated prisoners dilemma thing will act on AGIs too.

moonchrome 4y ago
I'm not sure how that doesn't devolve into an AGI armageddon. Those self improving AGIs would have to constantly be on equal level to avoid takeover, and they have an existential imperative to one up each other. There's no cooperation incentive between them just a potential stalemate. Also the stalemate covers the people in control of AGIs (assuming they aren't acting independently) - the rest of us are just resources in breaking the stalemate game. This sounds like the best way to build paperclip maximizer.
- boringuser2 4y ago
  It's pretty clear that at some point in the process of recursive self-improvement that you'd have literally just invented God, whereafter all predictions break down.
mrob 4y ago
The more groups have AGI, the higher the chance that one of them figures out recursive self-improvement, and if that happens it's game over for everybody. Iterated prisoner's dilemma only works when all the prisoners have roughly the same power.
- chr1 4y ago
  Why is it game over though? AI is already used for self-improvement, and will be used more as it becomes more capable, but that won't magically solve all the problems in the world and create omnipotent AI. If there are many people and many AI's they collectively will be stronger than the one most advanced, and cooperation will remain advantageous.

antegamisou 4y ago

Very accurate take, one should expect the worst from such advancements under capitalist economy - societal structure.

moonchrome 4y ago
More fundamental than capitalism I'd say, it's evolutionary.