3 ms·
The keyword you're looking for is "instrumental convergence" (https://www.lesswrong.com/tag/instrumental-convergence https://www.lesswrong.com/tag/instrumental-
by jimrandomh 4y ago
The keyword you're looking for is "instrumental convergence" (https://www.lesswrong.com/tag/instrumental-convergence https://www.lesswrong.com/tag/instrumental-convergence). It's not an intuition or obviousness based argument; there's significant prior work, if someone calls it obvious and skips over the argument they're being lazy by not providing links but the links do exist.
The expert-system/GOFAI people are mostly not the same people.
- astrange 4y agoThe argument that an imaginary AGI is going to take over the world because of an imaginary principle is not actually very convincing though. It would be better if you said they'd do something humans already did; at least we know that'd be possible. Most of this "instrumental convergence" assumes that it has an infinite attention span, infinite energy, all its plans work properly… the only reason it all goes that well is you're imagining it did. I think last week someone on twitter told me an AI could be bad because it'd "kill us all by releasing viruses into the atmosphere", but if you had a baby it could also grow up to do that, and (correctly) nobody worries about this. Not clear why these are supposed to be different. More importantly, it mostly relies on thinking AGIs behave like computer programs in that 1. they don't get bored 2. they can be copied or copy themselves. Is that necessarily true? If someone copied you could you take over the world easier? Maybe you'd just fight over who gets the bank account.