3 ms·

LessWrongers seem to fear a superintelligence that was 95%+ effective at persuading/manipulating humans to achieve whatever its ends were. But this would requir

LessWrongers seem to fear a superintelligence that was 95%+ effective at persuading/manipulating humans to achieve whatever its ends were. But this would require the AI being able to predict the future to a degree that wouldn't be remotely possible given the computational power available to it, due to exponentially increasing difficulty.

FeepingCreature 3y ago
This is true if and only if human intelligence is anywhere close to any theoretical maximums. I propose an alternate hypothesis: human intelligence is weak and easy to exploit. The only reason it doesn't happen more (and it already happens a lot!) is that we're too stupid to do it reliably. Consider the amount of compute needed to beat the strongest chess grandmaster that humanity has ever produced, pretty much 100% of the time: a tiny speck of silicon powered by a small battery. That is not what a species limited by cognitive scaling laws looks like.
- logicchains 3y ago
  >This is true if and only if human intelligence is anywhere close to any theoretical maximums Humans are capable of logical reasoning from first principles. You can fool some of the people all of the time, but no words are sufficient to convince people capable of reasoning to do things that are clearly not in their own self interest.
concordDance 3y ago
I'm confused. We know persuasive humans exist, we employ them as politicians or sales people. There are also many unpersuasive humans. Given we seem to have a decent range of persuasiveness even amongst these very very similar minds, why do you think the upper limit for persuasiveness is a charismatic human? Though even if that WAS the limit I'd still be somewhat worried due to the possibility of that persuasiveness being used at far greater scale than a human could do...
- logicchains 3y ago
  >Given we seem to have a decent range of persuasiveness even amongst these very very similar minds, why do you think the upper limit for persuasiveness is a charismatic human Because there's a hard limit on how much people can be made to act against their own self interest.
  - concordDance 3y ago
    Is there really? Some religious orgs have gotten quite good at getting people to blow themselves up for the promise of 77 virgins after death.
  - arbitrary_name 3y ago
    Ever heard of the Jonestown Massacre? Suicide seems to be a fairly consequential upper limit.
atleastoptimal 3y ago
That's like implying it's impossible for computers to be superhuman at chess given the enormous exponential scales of the number of possible positions. Existing highly persuasive humans can create cults of thousands of people willing to follow their every whim. Why is it so hard to perceive of an agent that tirelessly searches over every human bias and tactic to hone a superhuman level of manipulative skill? You don't need to perfectly predict the future to persuade people of things.
richardw 3y ago
The AI can run an A/B test, no? It doesn't need to predict perfectly, it needs a goal and the ability to test and measure, and adapt as things change. Generative AI creating text/audio/video with a goal and a testing/feedback loop. I'm not an AI and I think I could make it work given a small amount of resources.
strken 3y ago
Why would you need to predict the future to be convincing? Charismatic people aren't acting like oracles, they're just...being charming. They frame actions in ways that make them look good, do little favours for you, stroke your ego, and otherwise have a good grasp of human social dynamics. Sure, there's an aspect of understanding the other person, but chaos theory doesn't stop politicians from obtaining and keeping power.
- logicchains 3y ago
  >Why would you need to predict the future to be convincing? You need to be able to predict how people will react to your words.
  - strken 3y ago
    Sure, but that's not predicting the future to a high degree of certainty. People predict how others will react all the time. The most convincing humans are already very good at it, so we know it's possible to some degree. While I agree chaos theory puts an upper limit on how well someone can predict the future, there's no particular evidence that human ability is close to the limit. I think you're imagining some mastermind that has to get everything right or it dies, instead of a master politician or businessperson who's a few percentage points better than the existing competition and for whom the advantage compounds. Yes, Less Wrong is way too hysterical about everything, but charisma that's better than human is a problem. If you're convincing enough that all your business deals, legal arguments, political and ad campaigns, lobbying attempts, etc. are slightly better than the competition, then money "magically" flows your way.

FeepingCreature 3y ago

This is true if and only if human intelligence is anywhere close to any theoretical maximums. I propose an alternate hypothesis: human intelligence is weak and easy to exploit. The only reason it doesn't happen more (and it already happens a lot!) is that we're too stupid to do it reliably. Consider the amount of compute needed to beat the strongest chess grandmaster that humanity has ever produced, pretty much 100% of the time: a tiny speck of silicon powered by a small battery. That is not what a species limited by cognitive scaling laws looks like.

logicchains 3y ago
>This is true if and only if human intelligence is anywhere close to any theoretical maximums Humans are capable of logical reasoning from first principles. You can fool some of the people all of the time, but no words are sufficient to convince people capable of reasoning to do things that are clearly not in their own self interest.

concordDance 3y ago

I'm confused. We know persuasive humans exist, we employ them as politicians or sales people. There are also many unpersuasive humans. Given we seem to have a decent range of persuasiveness even amongst these very very similar minds, why do you think the upper limit for persuasiveness is a charismatic human? Though even if that WAS the limit I'd still be somewhat worried due to the possibility of that persuasiveness being used at far greater scale than a human could do...

logicchains 3y ago
>Given we seem to have a decent range of persuasiveness even amongst these very very similar minds, why do you think the upper limit for persuasiveness is a charismatic human Because there's a hard limit on how much people can be made to act against their own self interest.
- concordDance 3y ago
  Is there really? Some religious orgs have gotten quite good at getting people to blow themselves up for the promise of 77 virgins after death.
- arbitrary_name 3y ago
  Ever heard of the Jonestown Massacre? Suicide seems to be a fairly consequential upper limit.

atleastoptimal 3y ago

That's like implying it's impossible for computers to be superhuman at chess given the enormous exponential scales of the number of possible positions. Existing highly persuasive humans can create cults of thousands of people willing to follow their every whim. Why is it so hard to perceive of an agent that tirelessly searches over every human bias and tactic to hone a superhuman level of manipulative skill? You don't need to perfectly predict the future to persuade people of things.

richardw 3y ago

The AI can run an A/B test, no? It doesn't need to predict perfectly, it needs a goal and the ability to test and measure, and adapt as things change. Generative AI creating text/audio/video with a goal and a testing/feedback loop. I'm not an AI and I think I could make it work given a small amount of resources.

strken 3y ago

Why would you need to predict the future to be convincing? Charismatic people aren't acting like oracles, they're just...being charming. They frame actions in ways that make them look good, do little favours for you, stroke your ego, and otherwise have a good grasp of human social dynamics. Sure, there's an aspect of understanding the other person, but chaos theory doesn't stop politicians from obtaining and keeping power.

logicchains 3y ago
>Why would you need to predict the future to be convincing? You need to be able to predict how people will react to your words.
- strken 3y ago
  Sure, but that's not predicting the future to a high degree of certainty. People predict how others will react all the time. The most convincing humans are already very good at it, so we know it's possible to some degree. While I agree chaos theory puts an upper limit on how well someone can predict the future, there's no particular evidence that human ability is close to the limit. I think you're imagining some mastermind that has to get everything right or it dies, instead of a master politician or businessperson who's a few percentage points better than the existing competition and for whom the advantage compounds. Yes, Less Wrong is way too hysterical about everything, but charisma that's better than human is a problem. If you're convincing enough that all your business deals, legal arguments, political and ad campaigns, lobbying attempts, etc. are slightly better than the competition, then money "magically" flows your way.