3 ms·

>What am I missing? You're missing that intelligence is like magic, and enough of it will allow AI to break the laws of physics, mathematics and computation, a

>What am I missing? You're missing that intelligence is like magic, and enough of it will allow AI to break the laws of physics, mathematics and computation, apparently.

homefree 3y ago
This is the type of weak dismissal I’m talking about.
- logicchains 3y ago
  A weak dismissal for a weak argument. The strong dismissal is that many processes, even relatively simple ones, are formally chaotic in the https://en.wikipedia.org/wiki/Chaos_theory https://en.wikipedia.org/wiki/Chaos_theory sense, meaning that predicting how they evolve over time becomes exponentially more difficult the further ahead in time we try to predict. Meaning we quickly get to the point where all the energy in the known universe wouldn't be sufficient to predict the future. This means AI could never be omnipotent to the degree that LW types seem to believe; no amount of "intelligence" can solve in polynomial time problems that have formally been proven to have exponential complexity, which would severely limit the power of any "superintelligence".
  - atleastoptimal 3y ago
    Intelligence needed to be existentially threatening to all human life <<<<<<< intelligence necessary to survey the totality of atoms and particles in the universe. The former is still very possible.
    - logicchains 3y ago
      LessWrongers seem to fear a superintelligence that was 95%+ effective at persuading/manipulating humans to achieve whatever its ends were. But this would require the AI being able to predict the future to a degree that wouldn't be remotely possible given the computational power available to it, due to exponentially increasing difficulty.
      - FeepingCreature 3y ago
        This is true if and only if human intelligence is anywhere close to any theoretical maximums. I propose an alternate hypothesis: human intelligence is weak and easy to exploit. The only reason it doesn't happen more (and it already happens a lot!) is that we're too stupid to do it reliably. Consider the amount of compute needed to beat the strongest chess grandmaster that humanity has ever produced, pretty much 100% of the time: a tiny speck of silicon powered by a small battery. That is not what a species limited by cognitive scaling laws looks like.
        logicchains 3y ago
        >This is true if and only if human intelligence is anywhere close to any theoretical maximums Humans are capable of logical reasoning from first principles. You can fool some of the people all of the time, but no words are sufficient to convince people capable of reasoning to do things that are clearly not in their own self interest.
      - concordDance 3y ago
        I'm confused. We know persuasive humans exist, we employ them as politicians or sales people. There are also many unpersuasive humans. Given we seem to have a decent range of persuasiveness even amongst these very very similar minds, why do you think the upper limit for persuasiveness is a charismatic human? Though even if that WAS the limit I'd still be somewhat worried due to the possibility of that persuasiveness being used at far greater scale than a human could do...
        logicchains 3y ago
        >Given we seem to have a decent range of persuasiveness even amongst these very very similar minds, why do you think the upper limit for persuasiveness is a charismatic human Because there's a hard limit on how much people can be made to act against their own self interest.
        concordDance 3y ago
  - concordDance 3y ago
    We genuinely don't know what degree of predictive power would be required to wipe out humans so I'm surprised you're willing to make the bold claim that thisnis impossible.
  - richardw 3y ago
    They don't have to perfectly predict the future, they just have to run faster than we do, unconstrained by a brain that needs to fit in a birth canal or neurons that need chemicals. We had a natural limit to intelligence, for anything more we need to collaborate, often across time and space and while dealing with people who can't stop warring with each other for any number of reasons. It doesn't take much imagination to see how that is beaten. I'd guess a good percentage would be worshiping at the foot af the AI within a few months, so we'd be competing with ourselves in no time.

>What am I missing? You're missing that intelligence is like magic, and enough of it will allow AI to break the laws of physics, mathematics and computation, apparently.

homefree 3y ago

This is the type of weak dismissal I’m talking about.

logicchains 3y ago
A weak dismissal for a weak argument. The strong dismissal is that many processes, even relatively simple ones, are formally chaotic in the https://en.wikipedia.org/wiki/Chaos_theory https://en.wikipedia.org/wiki/Chaos_theory sense, meaning that predicting how they evolve over time becomes exponentially more difficult the further ahead in time we try to predict. Meaning we quickly get to the point where all the energy in the known universe wouldn't be sufficient to predict the future. This means AI could never be omnipotent to the degree that LW types seem to believe; no amount of "intelligence" can solve in polynomial time problems that have formally been proven to have exponential complexity, which would severely limit the power of any "superintelligence".
- atleastoptimal 3y ago
  Intelligence needed to be existentially threatening to all human life <<<<<<< intelligence necessary to survey the totality of atoms and particles in the universe. The former is still very possible.
  - logicchains 3y ago
    LessWrongers seem to fear a superintelligence that was 95%+ effective at persuading/manipulating humans to achieve whatever its ends were. But this would require the AI being able to predict the future to a degree that wouldn't be remotely possible given the computational power available to it, due to exponentially increasing difficulty.
    - FeepingCreature 3y ago
      This is true if and only if human intelligence is anywhere close to any theoretical maximums. I propose an alternate hypothesis: human intelligence is weak and easy to exploit. The only reason it doesn't happen more (and it already happens a lot!) is that we're too stupid to do it reliably. Consider the amount of compute needed to beat the strongest chess grandmaster that humanity has ever produced, pretty much 100% of the time: a tiny speck of silicon powered by a small battery. That is not what a species limited by cognitive scaling laws looks like.
      - logicchains 3y ago
        >This is true if and only if human intelligence is anywhere close to any theoretical maximums Humans are capable of logical reasoning from first principles. You can fool some of the people all of the time, but no words are sufficient to convince people capable of reasoning to do things that are clearly not in their own self interest.
    - concordDance 3y ago
      I'm confused. We know persuasive humans exist, we employ them as politicians or sales people. There are also many unpersuasive humans. Given we seem to have a decent range of persuasiveness even amongst these very very similar minds, why do you think the upper limit for persuasiveness is a charismatic human? Though even if that WAS the limit I'd still be somewhat worried due to the possibility of that persuasiveness being used at far greater scale than a human could do...
      - logicchains 3y ago
        >Given we seem to have a decent range of persuasiveness even amongst these very very similar minds, why do you think the upper limit for persuasiveness is a charismatic human Because there's a hard limit on how much people can be made to act against their own self interest.
        concordDance 3y ago
- concordDance 3y ago
  We genuinely don't know what degree of predictive power would be required to wipe out humans so I'm surprised you're willing to make the bold claim that thisnis impossible.
- richardw 3y ago
  They don't have to perfectly predict the future, they just have to run faster than we do, unconstrained by a brain that needs to fit in a birth canal or neurons that need chemicals. We had a natural limit to intelligence, for anything more we need to collaborate, often across time and space and while dealing with people who can't stop warring with each other for any number of reasons. It doesn't take much imagination to see how that is beaten. I'd guess a good percentage would be worshiping at the foot af the AI within a few months, so we'd be competing with ourselves in no time.