3 ms·

I mean... what you're communicating right now is "I have built a strawman in my head which I thoroughly hate, can anyone please come defend it before me"? I gu

I mean... what you're communicating right now is "I have built a strawman in my head which I thoroughly hate, can anyone please come defend it before me"? I guess I'll try anyway. What's your opinion on the orthogonality thesis, ie intelligence not being fundamentally correlated with altruism? The concept of mesa-optimization, that a process that tries to teach a system to optimize for X might instead instead make it optimize for thing-that-leads-to-X (eg evolution made us want sex because it leads to reproduction, but now we have a lot of sex with birth control). The concept of instrumental convergence, that any optimization process powerful enough will eventually optimize for "develop agency, survive longer, accumulate more power"? There's a lot of literature written about AI safety, a lot of it from very skeptical people, but it's not like there's any single insight that can be reliably transmitted to a guy at a table with a "you're all idiots, change my mind" sign.

ThrowawayTestr 3y ago
>intelligence not being fundamentally correlated with altruism I don't think were anywhere close to intelligence >a process that tries to teach a system to optimize for X might instead instead make it optimize for thing-that-leads-to-X Just another software bug to fix. >any optimization process powerful enough will eventually optimize for "develop agency, survive longer, accumulate more power" Pascal's wager for nerds. >There's a lot of literature written about AI safety There's also a lot of science fiction about killer robots exterminating humanity, same genre of writing.
- PoignardAzur 3y ago
  > Pascal's wager for nerds. You're mixing up your pre-written dismissals. Instrumental convergence has nothing to do with Pascal's Wager. Pascal's Wager for nerds would be a somewhat valid answer to "AI has the potential to destroy the world therefore we should worry about it even if it's very unlikely", but that's not the argument I made.
unethical_ban 3y ago
Judging by their posts, the person you responded to doesn't ever follow up on their commentary.

ThrowawayTestr 3y ago

>intelligence not being fundamentally correlated with altruism I don't think were anywhere close to intelligence >a process that tries to teach a system to optimize for X might instead instead make it optimize for thing-that-leads-to-X Just another software bug to fix. >any optimization process powerful enough will eventually optimize for "develop agency, survive longer, accumulate more power" Pascal's wager for nerds. >There's a lot of literature written about AI safety There's also a lot of science fiction about killer robots exterminating humanity, same genre of writing.

PoignardAzur 3y ago
> Pascal's wager for nerds. You're mixing up your pre-written dismissals. Instrumental convergence has nothing to do with Pascal's Wager. Pascal's Wager for nerds would be a somewhat valid answer to "AI has the potential to destroy the world therefore we should worry about it even if it's very unlikely", but that's not the argument I made.

unethical_ban 3y ago

Judging by their posts, the person you responded to doesn't ever follow up on their commentary.