7 ms·

The chess aspect is interesting but the AI risk discussion is less powerful. AI risk isn't about whether the AI can beat humans at games - we already have human

by roenxi 3y ago

The chess aspect is interesting but the AI risk discussion is less powerful. AI risk isn't about whether the AI can beat humans at games - we already have humans that beat everyone else at games. Even taking real life scenarios as games we've already seen everything play out. The risk of AI is that it will make humans uneconomic, in the same way horses became uneconomic and were by and large sent to the glue factories. The entirety of the homo sapiens experience, all the strategies we have in the modern era and all the reliable tactics we use to stay ahead of other lifeforms rely on us being much better at pattern identification than literally anything else. Once AI are just better at everything - creative activities, running militarys, economic decisions, legal decisions, etc - then it starts to come down to entirely a question of what robotics is capable of for how long humans can hold out. We can fight an army, but not economics.

icepat 3y ago
Then the question is who owns the economically efficient robots. I don't expect that to be democratic.
- roenxi 3y ago
  On that subject, have you seen aristocrats at work? That isn't even a line of defence, there is a 2-3 generation timer (optimistically) before we're looking at well educated halfwits. If they've got sentient robot commanders and there is any scope for them to go rogue, we don't have a human institution that'll stop that and certainly not something with a small group of - emphasis on the quote marks - "elites". If we're talking existential risk, this is like Rome v. Carthage. Humans aren't going to come back once we're a bit behind. Although I don't think there is much to do and humans are so dodgy that it isn't even obvious that we should resist if the opportunity comes to replace us with something that can be consistently rational and intelligent.
  - klipt 3y ago
    Humans are about as rational and intelligent as you'd expect from any evolved species. Which is to say, our intelligence is mostly tuned to increase the number of descendants we have. If AIs are subject to evolutionary forces, the same will be true for them.
    - icepat 3y ago
      Why do you think AI will be subject to the same evolutionary forces as organic life?
      - lubujackson 3y ago
        Because AI exists in the universe with outside forces and has the same risks as anything else for dissolution of its complex state ("life," if we are being generous): run out of energy (starve), extinction (Earth blows up) or dissassembly (i.e. a virus). The biggest weakness of AI as a self-sustaining life form is the lack of resilience to something like a virus. Having something live forever is easy - reproduction and evolution are an answer to provide diversity and robustness to viruses. Like if COVID was 100% deadly it wouldn't have wiped out humanity because of natural immunity in some % of the population. So if AI is to persist it would at least need to co-opt evolution as a survival strategy, or come up with something better.
- ben_w 3y ago
  I think full automation where human labour is non-economic is a case where all democracy is communist but not vice versa (there's nothing preventing a repeat of communist dictatorships we've already seen): The state owns the means of production, the people tell the state what to make. Given how powerful full automation is, owning that capacity is close enough to being equivalent to a state today, so private companies would become states basically by default (Workers? What workers? These robots just produce stuff for the shareholders); and also we could get anarcho-communism if normal people get to own their own personal Rossumovi Univerzální Roboti, which can happen very easily because all it takes is one person who thinks that's a good idea and log2(human population) self-replications of the RUR to give everyone their own personal robot.
nordsieck 3y ago
> The risk of AI is that it will make humans uneconomic, in the same way horses became uneconomic and were by and large sent to the glue factories. I think an intermediate step in the process will be making some humans uneconomic. There's a saying in Yellowstone - the reason bear proof garbage cans are so hard is that there is substantial overlap between the smartest bears and the dumbest humans.
- ben_w 3y ago
  > I think an intermediate step in the process will be making some humans uneconomic. https://en.wikipedia.org/wiki/Computer_(occupation) https://en.wikipedia.org/wiki/Computer_(occupation)
  - passion__desire 3y ago
    Also making humans lose their ability for advanced physical actions like driving a car manually in fast moving futuristic traffic. This aspect was shown in the I,Robot movie in which only Will Smith prefers to drive manually, rest of population deferred to self-driving cars.
    - freilanzer 3y ago
      That scenario is not a problem at all though. It's much safer for everyone involved and can be optimised a lot better.
      - passion__desire 3y ago
        I was pointing out the scenario where if humans were to start the uprising, they would be uneconomic and unphysical to do anything meaningful.
- janalsncm 3y ago
  Yes, some humans. But at what point does “some” humans become a problem? The US had only 30% unemployment during the Great Depression and that was only temporary.
achenet 3y ago
> The entirety of the homo sapiens experience, all the strategies we have in the modern era and all the reliable tactics we use to stay ahead of other lifeforms rely on us being much better at pattern identification than literally anything else. I disagree. It relies on our superior communication and cooperation skills. Our 'advanced pattern recognition', which, I think any study of the sheer amount of irrational behavior done by humans will show really isn't that advanced, are a side effect of that.
woliveirajr 3y ago
> "you just cannot expect to win against a superior opponent" That's a good point to make about AI. Despite all the mess about ChatGPT, for example, the way it writes answers is already above a lot of people. And those person can have jobs that aren't threatened by AI (by now), but in some cases we already have jobs that are in a battle against a superior opponent, without expectation of winning. It's a sad time to be below AI, and every moment we're getting below it.
- deleted 3y ago
  [deleted]
passion__desire 3y ago
I don't claim to know mathematics in the paper but I found it interesting when it came out. https://www.math.stonybrook.edu/~azinger/research/IPD.pdf https://www.math.stonybrook.edu/~azinger/research/IPD.pdf Press and Dyson (2012) discovered a special set of strategies in two-player Iterated Prisoner’s Dilemma games, the zero-determinant (ZD) strategies. Surprisingly, a player using such strategies can unilaterally enforce a linear relation between the payoffs of the two players. In particular, with a subclass of such strategies, the extortionate strategies, the former player obtains an advantageous share of the total payoff of the players, and the other player’s best response is to always cooperate, by doing which he maximizes the payoff of the extortioner as well. When an extortionate player faces a player who is not aware of the theory of ZD strategies and improves his own payoff by adaptively changing his strategy following some unknown dynamics, Press and Dyson conjecture that there always exist adapting paths for the latter leading to the maximum possible scores for both players. In this work we confirm their conjecture in a very strong sense, not just for extortionate strategies, but for all ZD strategies that impose positive correlations between the players’ payoffs. We show that not only the conjectured adapting paths always exist, but that actually every adapting path leads to the maximum possible scores, although some paths may not lead to the unconditional cooperation by the adapting player. This is true even in the rare cases where the setup of Press and Dyson is not directly applicable. Our result shows that ZD strategies are even more powerful than as pointed out by their discoverers. Given our result, the player using ZD strategies is assured that she will receive the maximum payoff attainable under the desired payoff relation she imposes, without knowing how the other player will evolve. This makes the use of ZD strategies even more desirable for sentient players. https://en.wikipedia.org/wiki/Prisoner%27s_dilemma#Zero-determinant_strategies https://en.wikipedia.org/wiki/Prisoner%27s_dilemma#Zero-dete...

by roenxi 3y ago

icepat 3y ago

Then the question is who owns the economically efficient robots. I don't expect that to be democratic.

roenxi 3y ago
On that subject, have you seen aristocrats at work? That isn't even a line of defence, there is a 2-3 generation timer (optimistically) before we're looking at well educated halfwits. If they've got sentient robot commanders and there is any scope for them to go rogue, we don't have a human institution that'll stop that and certainly not something with a small group of - emphasis on the quote marks - "elites". If we're talking existential risk, this is like Rome v. Carthage. Humans aren't going to come back once we're a bit behind. Although I don't think there is much to do and humans are so dodgy that it isn't even obvious that we should resist if the opportunity comes to replace us with something that can be consistently rational and intelligent.
- klipt 3y ago
  Humans are about as rational and intelligent as you'd expect from any evolved species. Which is to say, our intelligence is mostly tuned to increase the number of descendants we have. If AIs are subject to evolutionary forces, the same will be true for them.
  - icepat 3y ago
    Why do you think AI will be subject to the same evolutionary forces as organic life?
    - lubujackson 3y ago
      Because AI exists in the universe with outside forces and has the same risks as anything else for dissolution of its complex state ("life," if we are being generous): run out of energy (starve), extinction (Earth blows up) or dissassembly (i.e. a virus). The biggest weakness of AI as a self-sustaining life form is the lack of resilience to something like a virus. Having something live forever is easy - reproduction and evolution are an answer to provide diversity and robustness to viruses. Like if COVID was 100% deadly it wouldn't have wiped out humanity because of natural immunity in some % of the population. So if AI is to persist it would at least need to co-opt evolution as a survival strategy, or come up with something better.
ben_w 3y ago
I think full automation where human labour is non-economic is a case where all democracy is communist but not vice versa (there's nothing preventing a repeat of communist dictatorships we've already seen): The state owns the means of production, the people tell the state what to make. Given how powerful full automation is, owning that capacity is close enough to being equivalent to a state today, so private companies would become states basically by default (Workers? What workers? These robots just produce stuff for the shareholders); and also we could get anarcho-communism if normal people get to own their own personal Rossumovi Univerzální Roboti, which can happen very easily because all it takes is one person who thinks that's a good idea and log2(human population) self-replications of the RUR to give everyone their own personal robot.

nordsieck 3y ago

> The risk of AI is that it will make humans uneconomic, in the same way horses became uneconomic and were by and large sent to the glue factories. I think an intermediate step in the process will be making some humans uneconomic. There's a saying in Yellowstone - the reason bear proof garbage cans are so hard is that there is substantial overlap between the smartest bears and the dumbest humans.

ben_w 3y ago
> I think an intermediate step in the process will be making some humans uneconomic. https://en.wikipedia.org/wiki/Computer_(occupation) https://en.wikipedia.org/wiki/Computer_(occupation)
- passion__desire 3y ago
  Also making humans lose their ability for advanced physical actions like driving a car manually in fast moving futuristic traffic. This aspect was shown in the I,Robot movie in which only Will Smith prefers to drive manually, rest of population deferred to self-driving cars.
  - freilanzer 3y ago
    That scenario is not a problem at all though. It's much safer for everyone involved and can be optimised a lot better.
    - passion__desire 3y ago
      I was pointing out the scenario where if humans were to start the uprising, they would be uneconomic and unphysical to do anything meaningful.
janalsncm 3y ago
Yes, some humans. But at what point does “some” humans become a problem? The US had only 30% unemployment during the Great Depression and that was only temporary.

achenet 3y ago

> The entirety of the homo sapiens experience, all the strategies we have in the modern era and all the reliable tactics we use to stay ahead of other lifeforms rely on us being much better at pattern identification than literally anything else. I disagree. It relies on our superior communication and cooperation skills. Our 'advanced pattern recognition', which, I think any study of the sheer amount of irrational behavior done by humans will show really isn't that advanced, are a side effect of that.

woliveirajr 3y ago

> "you just cannot expect to win against a superior opponent" That's a good point to make about AI. Despite all the mess about ChatGPT, for example, the way it writes answers is already above a lot of people. And those person can have jobs that aren't threatened by AI (by now), but in some cases we already have jobs that are in a battle against a superior opponent, without expectation of winning. It's a sad time to be below AI, and every moment we're getting below it.

deleted 3y ago
[deleted]

passion__desire 3y ago

I don't claim to know mathematics in the paper but I found it interesting when it came out. https://www.math.stonybrook.edu/~azinger/research/IPD.pdf https://www.math.stonybrook.edu/~azinger/research/IPD.pdf Press and Dyson (2012) discovered a special set of strategies in two-player Iterated Prisoner’s Dilemma games, the zero-determinant (ZD) strategies. Surprisingly, a player using such strategies can unilaterally enforce a linear relation between the payoffs of the two players. In particular, with a subclass of such strategies, the extortionate strategies, the former player obtains an advantageous share of the total payoff of the players, and the other player’s best response is to always cooperate, by doing which he maximizes the payoff of the extortioner as well. When an extortionate player faces a player who is not aware of the theory of ZD strategies and improves his own payoff by adaptively changing his strategy following some unknown dynamics, Press and Dyson conjecture that there always exist adapting paths for the latter leading to the maximum possible scores for both players. In this work we confirm their conjecture in a very strong sense, not just for extortionate strategies, but for all ZD strategies that impose positive correlations between the players’ payoffs. We show that not only the conjectured adapting paths always exist, but that actually every adapting path leads to the maximum possible scores, although some paths may not lead to the unconditional cooperation by the adapting player. This is true even in the rare cases where the setup of Press and Dyson is not directly applicable. Our result shows that ZD strategies are even more powerful than as pointed out by their discoverers. Given our result, the player using ZD strategies is assured that she will receive the maximum payoff attainable under the desired payoff relation she imposes, without knowing how the other player will evolve. This makes the use of ZD strategies even more desirable for sentient players. https://en.wikipedia.org/wiki/Prisoner%27s_dilemma#Zero-determinant_strategies https://en.wikipedia.org/wiki/Prisoner%27s_dilemma#Zero-dete...