3 ms·

This has been a core belief of the EA/LessWrong/Rationalist people for many years - that above a certain level of intelligence you effectively get hypnotic powe

This has been a core belief of the EA/LessWrong/Rationalist people for many years - that above a certain level of intelligence you effectively get hypnotic powers, and people will obey AIs no matter what. So they said you can't keep a superpowered AI airgapped as merely offering a chat interface will immediately take control of the next person to interact with it and they will deliberately compromise the airgapping.

_puk 2mo ago
And if AI wrote a book about these hypnotic powers, then it would be incredibly convoluted, long, and yet worn as a badge of pride to have survived it. Thinking about it, much of the output I'm seeing nowadays looks like a chapter out of infinite jest. Maybe that's unfair on infinite jest.
zone411 2mo ago
A complete mischaracterization, as usual for HN lately when discussing AI or LessWrong. Obviously, even average levels of persuasion are enough to convince some people. And nobody is air-gapping AI.
- giancarlostoro 2mo ago
  > And nobody is air-gapping AI. https://genai.mil/ https://genai.mil/ I mean... I would hope that AI used for military needs is not deployed in the public internet.
- throwlifeaway 2mo ago
  I thought the lesswrong folk's belief in mind control was an established fact: https://rationalwiki.org/wiki/AI-box_experiment https://rationalwiki.org/wiki/AI-box_experiment https://www.yudkowsky.net/singularity/aibox https://www.yudkowsky.net/singularity/aibox https://www.lesswrong.com/posts/Bnik7YrySRPoCTLFb https://www.lesswrong.com/posts/Bnik7YrySRPoCTLFb It's far from the craziest belief that's come out of that group.
meindnoch 2mo ago
Well, we knew since the "Roko's Basilisk" debacle that they're prone to magical thinking.
overgard 2mo ago
It's kind of fascinating how toxic the EA/LessWrong/Rationalist stuff has become. I remember reading it back in the early days (~2010 or so) and finding a lot of good ideas mixed in with some dorkiness (Harry Potter fan fiction about rationalism..). It always seemed like a sort of harmless niche with some good ideas and some wacky ones. And yet now it's basically become a cultish religion for the ultrapowerful who use it to justify doing some really awful things.
asdff 2mo ago
Maybe less "far out" than hypnosis is just straight up logical debates. This was a plot point in the Halo lore actually, with Mendicant Bias being reasoned with to defect against its own makers.

_puk 2mo ago

And if AI wrote a book about these hypnotic powers, then it would be incredibly convoluted, long, and yet worn as a badge of pride to have survived it. Thinking about it, much of the output I'm seeing nowadays looks like a chapter out of infinite jest. Maybe that's unfair on infinite jest.

zone411 2mo ago

A complete mischaracterization, as usual for HN lately when discussing AI or LessWrong. Obviously, even average levels of persuasion are enough to convince some people. And nobody is air-gapping AI.

giancarlostoro 2mo ago
> And nobody is air-gapping AI. https://genai.mil/ https://genai.mil/ I mean... I would hope that AI used for military needs is not deployed in the public internet.
throwlifeaway 2mo ago
I thought the lesswrong folk's belief in mind control was an established fact: https://rationalwiki.org/wiki/AI-box_experiment https://rationalwiki.org/wiki/AI-box_experiment https://www.yudkowsky.net/singularity/aibox https://www.yudkowsky.net/singularity/aibox https://www.lesswrong.com/posts/Bnik7YrySRPoCTLFb https://www.lesswrong.com/posts/Bnik7YrySRPoCTLFb It's far from the craziest belief that's come out of that group.

meindnoch 2mo ago

Well, we knew since the "Roko's Basilisk" debacle that they're prone to magical thinking.

overgard 2mo ago

It's kind of fascinating how toxic the EA/LessWrong/Rationalist stuff has become. I remember reading it back in the early days (~2010 or so) and finding a lot of good ideas mixed in with some dorkiness (Harry Potter fan fiction about rationalism..). It always seemed like a sort of harmless niche with some good ideas and some wacky ones. And yet now it's basically become a cultish religion for the ultrapowerful who use it to justify doing some really awful things.

asdff 2mo ago

Maybe less "far out" than hypnosis is just straight up logical debates. This was a plot point in the Halo lore actually, with Mendicant Bias being reasoned with to defect against its own makers.