3 ms·
Technical people, or tech-adjacent people at least, seem to be even worse. They seem to assume that since they know how computers work that AGI “obviously” work
by astrange 4y ago
Technical people, or tech-adjacent people at least, seem to be even worse. They seem to assume that since they know how computers work that AGI “obviously” works the same way and will obviously instantly self-reproduce and take over the world for… some reason.
Of course, they were equally confident logic and expert systems could build an AI and despite none of that working out at all, it doesn’t seem to have stopped them.
- jimrandomh 4y agoThe keyword you're looking for is "instrumental convergence" (https://www.lesswrong.com/tag/instrumental-convergence https://www.lesswrong.com/tag/instrumental-convergence). It's not an intuition or obviousness based argument; there's significant prior work, if someone calls it obvious and skips over the argument they're being lazy by not providing links but the links do exist. The expert-system/GOFAI people are mostly not the same people.
- astrange 4y agoThe argument that an imaginary AGI is going to take over the world because of an imaginary principle is not actually very convincing though. It would be better if you said they'd do something humans already did; at least we know that'd be possible. Most of this "instrumental convergence" assumes that it has an infinite attention span, infinite energy, all its plans work properly… the only reason it all goes that well is you're imagining it did. I think last week someone on twitter told me an AI could be bad because it'd "kill us all by releasing viruses into the atmosphere", but if you had a baby it could also grow up to do that, and (correctly) nobody worries about this. Not clear why these are supposed to be different. More importantly, it mostly relies on thinking AGIs behave like computer programs in that 1. they don't get bored 2. they can be copied or copy themselves. Is that necessarily true? If someone copied you could you take over the world easier? Maybe you'd just fight over who gets the bank account.