3 ms·
Skepticism is an absolutely appropriate response. However, failing in the way you describe isn't a massive problem, it's just simple failure. It's not the wor
by JoshTriplett 10y ago
Skepticism is an absolutely appropriate response. However, failing in the way you describe isn't a massive problem, it's just simple failure. It's not the worst-case scenario. And it's appropriate to consider and plan for the worst-case scenario, especially when that scenario reaches the existential-threat level.
I absolutely agree that it's very hard to get to recursive self-improvement. And until you get there, certain classes of failure modes can't really happen.
(Others can, like the grey goo scenario; there are many people who consider carefully built AGI a good idea because the odds of some other existential threat arising approach 1 over time. There's a kind of Moore's law of existential threats: the intelligence and power required to destroy the world drops by some small multiplicative factor every 18 months. What happens when the equipment needed to destroy the world can be built in any well-equipped university lab? Or in a garage?)
The failure modes I'm talking about are that once you do hit recursive self-improvement, it's a lot easier to build something deadly than something safe, let alone something useful.