3 ms·

Thanks for a concise summary. I guess these assumptions don't appeal to me because they are mostly justified by Pascal-wager-like argument: "if they are true an

Thanks for a concise summary. I guess these assumptions don't appeal to me because they are mostly justified by Pascal-wager-like argument: "if they are true and we disregard them, we are fucked; so we'd better assume they are true". Claims of recursively reaching godlike intelligence seem particularly dubious to me. Why would AGI be superhuman at all? What if the hypothetical paperclip maximizer produces one paperclip and then dies out of exhaustion? Why would it recursively self-improve to a godlike status? Humans can recursively self-improve to an extent too and it doesn't happen to them. Why would it be particularly interested in recursive self improvement (this one seems to be imported straight from a particular human value system)? I guess this skeptical way of thinking is part of reflexes of an engineer. When presented with the whitepaper of a database with miraculous consistency properties you don't immediately start to fantasize about the possible applications. Instead you ask "Can I play with it? Gimme the demo. Does it pass jepsen?"

idlewords 10y ago
Humans can't recursively self-improve in the sense AI adherents mean it, because we are stuck in our meat brains and can't redesign them or move to faster hardware (yet!, they'll say). What's being smuggled under the philosophical carpet here is exactly what you point out. There's no guarantee that an AI would be a humorless utilitarian rationalist obsessed with self-improvement. It's also possible that general intelligence is a tradeoff against something else (like existential despair, or a tendency to veer off into writing extropian fanfic) and entities hundreds, or thousands, or tens of times smarter than human beings can't exist. Scio me nihil scire.
JoshTriplett 10y ago
Skepticism is an absolutely appropriate response. However, failing in the way you describe isn't a massive problem, it's just simple failure. It's not the worst-case scenario. And it's appropriate to consider and plan for the worst-case scenario, especially when that scenario reaches the existential-threat level. I absolutely agree that it's very hard to get to recursive self-improvement. And until you get there, certain classes of failure modes can't really happen. (Others can, like the grey goo scenario; there are many people who consider carefully built AGI a good idea because the odds of some other existential threat arising approach 1 over time. There's a kind of Moore's law of existential threats: the intelligence and power required to destroy the world drops by some small multiplicative factor every 18 months. What happens when the equipment needed to destroy the world can be built in any well-equipped university lab? Or in a garage?) The failure modes I'm talking about are that once you do hit recursive self-improvement, it's a lot easier to build something deadly than something safe, let alone something useful.

idlewords 10y ago

Humans can't recursively self-improve in the sense AI adherents mean it, because we are stuck in our meat brains and can't redesign them or move to faster hardware (yet!, they'll say). What's being smuggled under the philosophical carpet here is exactly what you point out. There's no guarantee that an AI would be a humorless utilitarian rationalist obsessed with self-improvement. It's also possible that general intelligence is a tradeoff against something else (like existential despair, or a tendency to veer off into writing extropian fanfic) and entities hundreds, or thousands, or tens of times smarter than human beings can't exist. Scio me nihil scire.

JoshTriplett 10y ago

Skepticism is an absolutely appropriate response. However, failing in the way you describe isn't a massive problem, it's just simple failure. It's not the worst-case scenario. And it's appropriate to consider and plan for the worst-case scenario, especially when that scenario reaches the existential-threat level. I absolutely agree that it's very hard to get to recursive self-improvement. And until you get there, certain classes of failure modes can't really happen. (Others can, like the grey goo scenario; there are many people who consider carefully built AGI a good idea because the odds of some other existential threat arising approach 1 over time. There's a kind of Moore's law of existential threats: the intelligence and power required to destroy the world drops by some small multiplicative factor every 18 months. What happens when the equipment needed to destroy the world can be built in any well-equipped university lab? Or in a garage?) The failure modes I'm talking about are that once you do hit recursive self-improvement, it's a lot easier to build something deadly than something safe, let alone something useful.