2 ms·

I guess there's 3 players in the games that often get invoked in discussions about AI cogsec: - the corporation running the AI - the user - the AI, sometimes

by pizza 2y ago

I guess there's 3 players in the games that often get invoked in discussions about AI cogsec: - the corporation running the AI - the user - the AI, sometimes - depending on the particular conversation's ascription of agency to the AI It seems the downstream harms are of 2 kinds: - social engineering to give up otherwise secured systems - 'people engineering' - the kind of thing people complain about when it comes to recommender systems. "Mind control", basically, y'know. [0] Things like r/LocalLlama and the general "tinkerer" environment of open source AI makes me wonder if it wouldn't be rather trivial in some sense for users to build the same capabilities but for personal protection from malign influences. Like a kind of user-controlled "debrief AI". But then, of course, you might get a superintelligence that can pretend to be your friend but is actually more like Iago from Othello. But is that really a likelihood in a situation where the user can make their own RLHF dataset and fit it to the desired behavior? Generally I'd expect the user to get the behavior they were looking for. Plus, like immune system memory, people could continually train new examples of sleights into it. I guess maybe there could be a hint of the "Waluigi problem", perhaps. [0] I think it does the people who are distressed about it a disservice to saturate all their news channels with reports about how they are utterly incapable of outwitting an algorithmic super intelligence. But that's a different discussion altogether

klabb3 2y ago
> it wouldn't be rather trivial in some sense for users to build the same capabilities but for personal protection from malign influences. Yes the implication of AI in economics and society is like that of scraping, spam, scalping or fraud, ie leading to an arms race. Insurance companies denying claims, citizens fighting back filing appeals with AI. Captchas and fingerprinting to protect against bots, that use OCR and now AI to bypass the defenses. Well, this idea of increased net productivity relies on the fact that we don’t waste the excess on fighting each other in the same games we were playing all along. It’s like a feud between tribes going from sticks to swords to guns. It’s only when you replace the zero-sum activity with a positive value that the world actually improves. The techno-optimists see only potential in an all-else-equal world, which isn’t the world we live in. Potential is irrelevant in the face of incentives.

by pizza 2y ago

klabb3 2y ago

> it wouldn't be rather trivial in some sense for users to build the same capabilities but for personal protection from malign influences. Yes the implication of AI in economics and society is like that of scraping, spam, scalping or fraud, ie leading to an arms race. Insurance companies denying claims, citizens fighting back filing appeals with AI. Captchas and fingerprinting to protect against bots, that use OCR and now AI to bypass the defenses. Well, this idea of increased net productivity relies on the fact that we don’t waste the excess on fighting each other in the same games we were playing all along. It’s like a feud between tribes going from sticks to swords to guns. It’s only when you replace the zero-sum activity with a positive value that the world actually improves. The techno-optimists see only potential in an all-else-equal world, which isn’t the world we live in. Potential is irrelevant in the face of incentives.