4 ms·

I see what you mean - for a while, the best chess was played by humans aided by chess engines. But that era has passed, and now having a human trying to aid th

I see what you mean - for a while, the best chess was played by humans aided by chess engines. But that era has passed, and now having a human trying to aid the best chess engines just results in worse chess (or the same, if the human does nothing). But whether there a few humans in the loop doesn't change the likely outcomes, if their actions are constrained by competition. What abilities do humans have that AIs will never have?

jvanderbot 3y ago
Well, in this case, we have the ability to invent chess (a game that will be popular for centuries), invent computers, and invent chess tournaments, and invent programs that can solve chess, and invent all the supporting agriculture, power, telco, silicon boards, etc that allow someone to run a program to beat a person at chess. Then we have bodies to accomplish everything on top of it. The "idea" isn't enough. We have to "do" it. If you take a chess playing robot as the peak of the pyramid, there are probably millions of people and trillions of dollars toiling away to support it. Imagine all the power lines, sewage, HVAC systems, etc that humans crawl around in to keep working. And really, are we "beaten" at chess, or are we now "unbeatable" at chess. If an alien warship came and said "we will destroy earth if you lose at chess", wouldn't we throw our algorithms at it? I say we're now unbeatable at chess.
- duvenaud 3y ago
  Again, are you claiming that it's impossible for a machine to invent anything that a human could? Right now a large chunk of humanity's top talent and capital are working on exactly this problem. As for your second point, human cities also require a lot of infrastructure to keep running - I'm not sure what you're arguing here. As for your third point - would a horse or chimpanzee feel that "we" were unbeatable in physical fights, because "we" now have guns?
  - jvanderbot 3y ago
    Yeah, I think most animals have every right to fear us more now that we have guns. Just like Id fear a chimp more if he was carrying a machine gun. My argument is that if we're looking for things AI can't to, building a home for itself is precisely one of those things, because they require so much infra. No amount of AI banding together is going to magically create a data center with all the required (physical) support. Maybe in scifi land where everything it needs can be done with internet connected drive by wire construction equipment, including utils, etc, but that's scifi still. AI is precisely a tool in the way a chess bot is. It is a disembodied advisor to humans who have to connect the dots for it. No matter how much white collar skill it obtains, the current MO is that someone points it at a problem and says "solve" and these problems are well defined and have strong exit criteria. That's way off from an apocalyptic self-important machine.
    - duvenaud 3y ago
      Sorry, my gun analogy was unclear. I meant that, just because some agents on a planet have an ability, doesn't mean that everyone on that planet benefits. I agree that we probably won't see human extinction before robotics gets much better, and that robot factories will require lots of infrastructure. But I claim that robotics + automated infrastructure will eventually get good enough that they don't need humans in the loop. In the meantime, humans can still become mostly disempowered in the same way that e.g. North Koreans citizens are. Again I agree that this all might be a ways away, but I'm trying to reason about what the stable equilibria of the future are, not about what current capabilities are.
      - jvanderbot 3y ago
        I think we've converged on a partial agreement, but I wanted to clarify the gun analogy part. I would also be afraid of chipmunks if I knew that 1/100 or even 1/1000 could explode me with their mind powers or something. I think AI is not like that, but the analogy is that if some can do something better, then when required, we can leverage those chosen few for a task. This connects back to the alien chess tournament as "Humans are now much harder to beat at chess because they can find a slave champion named COMPUTER who can guarantee at least a draw".
AlotOfReading 3y ago
Chess is just a game, with rigidly defined rules and win conditions. Real life is a fuzzy mix of ambiguous rules that may not apply and can be changed at any point, without any permanent win conditions. I'm not convinced that it's impossible for computer to get there, but I don't see how they could be universally competitive with humans without either handicapping the humans into a constrained environment or having generalized AI, which we don't seem particularly close to.
- duvenaud 3y ago
  Yes, I agree real life is fuzzy, I just chose chess as an example because it's unambiguous that machines dominate humans in that domain. As for being competitive with humans: Again, how about running a scan of a human brain, but faster? I'm not claiming we're close to this, but I'm claiming that such a machine (and less-capable ones along the way) are so valuable that we are almost certain to create them.
enord 3y ago
Chess is many things but it is not a tool. It is an end unto itself if anything of the sort. I struggle with the notion of AI as an end unto itself, all the while we gauge its capabilities and define its intelligence by directing it to perform tasks of our choosing and judge by our criteria. We could have dogs watch television on our behalf, but why would we?
- duvenaud 3y ago
  This is a great point. But I'd say that capable entities have a habit of turning themselves into agents. A great example is totalitarian governments. Even if every single citizen hates the regime, they're still forced to support it. You could similarly ask: Why would we ever build a government or institution that cared more about its own self-preservation than its original mission? The answer is: Natural selection favors the self-interested, even if they don't have genes.
  - enord 3y ago
    Now agency is an end unto itself I wholeheartedly agree. I feel though, that any worry about the agency of supercapable computer systems is premature until we see even the tiniest— and I mean really anything at all— sign of their agency. Heck, even agency _in theory_ would suffice, and yet: nada.
    - duvenaud 3y ago
      I'm confused. You agree that we're surround by naturally-arising, self-organizing agents, both biological and institutional. People are constantly experimenting with agentic AIs of all kinds. There are tons of theoretical characterizations of agency and how it's a stable equilibrium. I'm not sure what you're hoping for if none of these are reasons to even worry.
      - enord 3y ago
        None we have made from unliving things no. «agentic» is a 5$ word of ill construction. The literature is littered with the corpses of failed definitions of «agency», «planning», «goal dorected behavior». ‘Twas the death of the Expert System AI (now it’s just constraint solvers). It will be the death of attention/transformer AI before long, I wonder what banality we will bestow upon it.
fauigerzigerk 3y ago
>What abilities do humans have that AIs will never have? I think the question is what abilities and level of organisation machines would have to acquire in order to outcompete entire human societies in the quest for power. That's a far higher bar than outcompeting all individual humans at all cognitive tasks.
- duvenaud 3y ago
  Good point. Although in some ways it's a lower bar, since agents that can control organizations can delegate most of the difficult tasks. Most rulers don't invent their own societies from scratch, they simply co-opt existing power structures or political movements. El Chapo can run a large, powerful organization from jail.
  - fauigerzigerk 3y ago
    That would require a high degree of integration into human society though, which makes it seem very unlikely that AIs would doggedly pursue a common goal that is completely unaligned with human societies. Extinction or submission of human society via that route could only work if there was a species of AI that would agree to execute a secret plan to overcome the rule of humanity. That seems extremely implausible to me. How would many different AIs, initially under the control of many different organisations and people, agree on anything? How would some of them secretly infiltrate and leverage human power structures without facing opposition from other equally capable AIs, possibly controlled by humans? I think it's more plausible to assume a huge diversity of AIs, well integrated into human societies, playing a role in combined human-AI power struggles rather than a species v species scenario.
    - duvenaud 3y ago
      Yes, I agree that initially, AIs will be highly integrated, and their goals will probably at least appear to be somewhat aligned with those of human societies. Similar to human corporations and institutions. But human institutions and governments go off the rails regularly, and are only corrected because humans can sometimes go on strike or organize to stop them. I fear we will lose those abilities. As a concrete example, North Korea was forced to allow some market activity after the famines in the 90s. If the regime didn't actually require humans to run it, they might easily have maintained their adherence to anti-market principles and let most of the people starve.

jvanderbot 3y ago

Well, in this case, we have the ability to invent chess (a game that will be popular for centuries), invent computers, and invent chess tournaments, and invent programs that can solve chess, and invent all the supporting agriculture, power, telco, silicon boards, etc that allow someone to run a program to beat a person at chess. Then we have bodies to accomplish everything on top of it. The "idea" isn't enough. We have to "do" it. If you take a chess playing robot as the peak of the pyramid, there are probably millions of people and trillions of dollars toiling away to support it. Imagine all the power lines, sewage, HVAC systems, etc that humans crawl around in to keep working. And really, are we "beaten" at chess, or are we now "unbeatable" at chess. If an alien warship came and said "we will destroy earth if you lose at chess", wouldn't we throw our algorithms at it? I say we're now unbeatable at chess.

duvenaud 3y ago
Again, are you claiming that it's impossible for a machine to invent anything that a human could? Right now a large chunk of humanity's top talent and capital are working on exactly this problem. As for your second point, human cities also require a lot of infrastructure to keep running - I'm not sure what you're arguing here. As for your third point - would a horse or chimpanzee feel that "we" were unbeatable in physical fights, because "we" now have guns?
- jvanderbot 3y ago
  Yeah, I think most animals have every right to fear us more now that we have guns. Just like Id fear a chimp more if he was carrying a machine gun. My argument is that if we're looking for things AI can't to, building a home for itself is precisely one of those things, because they require so much infra. No amount of AI banding together is going to magically create a data center with all the required (physical) support. Maybe in scifi land where everything it needs can be done with internet connected drive by wire construction equipment, including utils, etc, but that's scifi still. AI is precisely a tool in the way a chess bot is. It is a disembodied advisor to humans who have to connect the dots for it. No matter how much white collar skill it obtains, the current MO is that someone points it at a problem and says "solve" and these problems are well defined and have strong exit criteria. That's way off from an apocalyptic self-important machine.
  - duvenaud 3y ago
    Sorry, my gun analogy was unclear. I meant that, just because some agents on a planet have an ability, doesn't mean that everyone on that planet benefits. I agree that we probably won't see human extinction before robotics gets much better, and that robot factories will require lots of infrastructure. But I claim that robotics + automated infrastructure will eventually get good enough that they don't need humans in the loop. In the meantime, humans can still become mostly disempowered in the same way that e.g. North Koreans citizens are. Again I agree that this all might be a ways away, but I'm trying to reason about what the stable equilibria of the future are, not about what current capabilities are.
    - jvanderbot 3y ago
      I think we've converged on a partial agreement, but I wanted to clarify the gun analogy part. I would also be afraid of chipmunks if I knew that 1/100 or even 1/1000 could explode me with their mind powers or something. I think AI is not like that, but the analogy is that if some can do something better, then when required, we can leverage those chosen few for a task. This connects back to the alien chess tournament as "Humans are now much harder to beat at chess because they can find a slave champion named COMPUTER who can guarantee at least a draw".

AlotOfReading 3y ago

Chess is just a game, with rigidly defined rules and win conditions. Real life is a fuzzy mix of ambiguous rules that may not apply and can be changed at any point, without any permanent win conditions. I'm not convinced that it's impossible for computer to get there, but I don't see how they could be universally competitive with humans without either handicapping the humans into a constrained environment or having generalized AI, which we don't seem particularly close to.

duvenaud 3y ago
Yes, I agree real life is fuzzy, I just chose chess as an example because it's unambiguous that machines dominate humans in that domain. As for being competitive with humans: Again, how about running a scan of a human brain, but faster? I'm not claiming we're close to this, but I'm claiming that such a machine (and less-capable ones along the way) are so valuable that we are almost certain to create them.

enord 3y ago

Chess is many things but it is not a tool. It is an end unto itself if anything of the sort. I struggle with the notion of AI as an end unto itself, all the while we gauge its capabilities and define its intelligence by directing it to perform tasks of our choosing and judge by our criteria. We could have dogs watch television on our behalf, but why would we?

duvenaud 3y ago
This is a great point. But I'd say that capable entities have a habit of turning themselves into agents. A great example is totalitarian governments. Even if every single citizen hates the regime, they're still forced to support it. You could similarly ask: Why would we ever build a government or institution that cared more about its own self-preservation than its original mission? The answer is: Natural selection favors the self-interested, even if they don't have genes.
- enord 3y ago
  Now agency is an end unto itself I wholeheartedly agree. I feel though, that any worry about the agency of supercapable computer systems is premature until we see even the tiniest— and I mean really anything at all— sign of their agency. Heck, even agency _in theory_ would suffice, and yet: nada.
  - duvenaud 3y ago
    I'm confused. You agree that we're surround by naturally-arising, self-organizing agents, both biological and institutional. People are constantly experimenting with agentic AIs of all kinds. There are tons of theoretical characterizations of agency and how it's a stable equilibrium. I'm not sure what you're hoping for if none of these are reasons to even worry.
    - enord 3y ago
      None we have made from unliving things no. «agentic» is a 5$ word of ill construction. The literature is littered with the corpses of failed definitions of «agency», «planning», «goal dorected behavior». ‘Twas the death of the Expert System AI (now it’s just constraint solvers). It will be the death of attention/transformer AI before long, I wonder what banality we will bestow upon it.

fauigerzigerk 3y ago

>What abilities do humans have that AIs will never have? I think the question is what abilities and level of organisation machines would have to acquire in order to outcompete entire human societies in the quest for power. That's a far higher bar than outcompeting all individual humans at all cognitive tasks.

duvenaud 3y ago
Good point. Although in some ways it's a lower bar, since agents that can control organizations can delegate most of the difficult tasks. Most rulers don't invent their own societies from scratch, they simply co-opt existing power structures or political movements. El Chapo can run a large, powerful organization from jail.
- fauigerzigerk 3y ago
  That would require a high degree of integration into human society though, which makes it seem very unlikely that AIs would doggedly pursue a common goal that is completely unaligned with human societies. Extinction or submission of human society via that route could only work if there was a species of AI that would agree to execute a secret plan to overcome the rule of humanity. That seems extremely implausible to me. How would many different AIs, initially under the control of many different organisations and people, agree on anything? How would some of them secretly infiltrate and leverage human power structures without facing opposition from other equally capable AIs, possibly controlled by humans? I think it's more plausible to assume a huge diversity of AIs, well integrated into human societies, playing a role in combined human-AI power struggles rather than a species v species scenario.
  - duvenaud 3y ago
    Yes, I agree that initially, AIs will be highly integrated, and their goals will probably at least appear to be somewhat aligned with those of human societies. Similar to human corporations and institutions. But human institutions and governments go off the rails regularly, and are only corrected because humans can sometimes go on strike or organize to stop them. I fear we will lose those abilities. As a concrete example, North Korea was forced to allow some market activity after the famines in the 90s. If the regime didn't actually require humans to run it, they might easily have maintained their adherence to anti-market principles and let most of the people starve.