3 ms·

There's a whole quasi-religious belief system in the parent comment that you seem to be unfamiliar with, that a lot of people here espouse. That may be why you'

There's a whole quasi-religious belief system in the parent comment that you seem to be unfamiliar with, that a lot of people here espouse. That may be why you're being downvoted. Very basically, it goes like this: - the space of possible minds is very vast. We are animals that evolved in one set of circumstances, and so our minds (and thoughts we can think) occupy only a tiny corner of the space. - an artificial AI would operate on computer-like timescales, not human like ones. Therefore we would have only hours to react before it got control of the physical world (hacking the internet, making nanothings, ordering and synthesizing DNA). - an AI could (and would) recursively self-improve until it hit some kind of physical limit very far from our current capabilities. From a human standpoint, it would have God-like intelligence - such an AI could model people and their motivations successfully enough to persuade them to do whatever it wanted - because the space of minds is so vast, a generic AI would have goals that would be inscrutable and alien to us. To survive as a species, we would have to design them in from the beginning, in a way that survives all the levels of recursive self-redesign. There are too many assumptions about the nature of intelligence to unpack here, but I believe it fairly describes what a lot of people believe about AI, thanks to people like Yudkowsky who have a flair for writing about it engagingly.

dunkelheit 10y ago
Thanks for a concise summary. I guess these assumptions don't appeal to me because they are mostly justified by Pascal-wager-like argument: "if they are true and we disregard them, we are fucked; so we'd better assume they are true". Claims of recursively reaching godlike intelligence seem particularly dubious to me. Why would AGI be superhuman at all? What if the hypothetical paperclip maximizer produces one paperclip and then dies out of exhaustion? Why would it recursively self-improve to a godlike status? Humans can recursively self-improve to an extent too and it doesn't happen to them. Why would it be particularly interested in recursive self improvement (this one seems to be imported straight from a particular human value system)? I guess this skeptical way of thinking is part of reflexes of an engineer. When presented with the whitepaper of a database with miraculous consistency properties you don't immediately start to fantasize about the possible applications. Instead you ask "Can I play with it? Gimme the demo. Does it pass jepsen?"
- idlewords 10y ago
  Humans can't recursively self-improve in the sense AI adherents mean it, because we are stuck in our meat brains and can't redesign them or move to faster hardware (yet!, they'll say). What's being smuggled under the philosophical carpet here is exactly what you point out. There's no guarantee that an AI would be a humorless utilitarian rationalist obsessed with self-improvement. It's also possible that general intelligence is a tradeoff against something else (like existential despair, or a tendency to veer off into writing extropian fanfic) and entities hundreds, or thousands, or tens of times smarter than human beings can't exist. Scio me nihil scire.
- JoshTriplett 10y ago
  Skepticism is an absolutely appropriate response. However, failing in the way you describe isn't a massive problem, it's just simple failure. It's not the worst-case scenario. And it's appropriate to consider and plan for the worst-case scenario, especially when that scenario reaches the existential-threat level. I absolutely agree that it's very hard to get to recursive self-improvement. And until you get there, certain classes of failure modes can't really happen. (Others can, like the grey goo scenario; there are many people who consider carefully built AGI a good idea because the odds of some other existential threat arising approach 1 over time. There's a kind of Moore's law of existential threats: the intelligence and power required to destroy the world drops by some small multiplicative factor every 18 months. What happens when the equipment needed to destroy the world can be built in any well-equipped university lab? Or in a garage?) The failure modes I'm talking about are that once you do hit recursive self-improvement, it's a lot easier to build something deadly than something safe, let alone something useful.

dunkelheit 10y ago

Thanks for a concise summary. I guess these assumptions don't appeal to me because they are mostly justified by Pascal-wager-like argument: "if they are true and we disregard them, we are fucked; so we'd better assume they are true". Claims of recursively reaching godlike intelligence seem particularly dubious to me. Why would AGI be superhuman at all? What if the hypothetical paperclip maximizer produces one paperclip and then dies out of exhaustion? Why would it recursively self-improve to a godlike status? Humans can recursively self-improve to an extent too and it doesn't happen to them. Why would it be particularly interested in recursive self improvement (this one seems to be imported straight from a particular human value system)? I guess this skeptical way of thinking is part of reflexes of an engineer. When presented with the whitepaper of a database with miraculous consistency properties you don't immediately start to fantasize about the possible applications. Instead you ask "Can I play with it? Gimme the demo. Does it pass jepsen?"

idlewords 10y ago
Humans can't recursively self-improve in the sense AI adherents mean it, because we are stuck in our meat brains and can't redesign them or move to faster hardware (yet!, they'll say). What's being smuggled under the philosophical carpet here is exactly what you point out. There's no guarantee that an AI would be a humorless utilitarian rationalist obsessed with self-improvement. It's also possible that general intelligence is a tradeoff against something else (like existential despair, or a tendency to veer off into writing extropian fanfic) and entities hundreds, or thousands, or tens of times smarter than human beings can't exist. Scio me nihil scire.
JoshTriplett 10y ago
Skepticism is an absolutely appropriate response. However, failing in the way you describe isn't a massive problem, it's just simple failure. It's not the worst-case scenario. And it's appropriate to consider and plan for the worst-case scenario, especially when that scenario reaches the existential-threat level. I absolutely agree that it's very hard to get to recursive self-improvement. And until you get there, certain classes of failure modes can't really happen. (Others can, like the grey goo scenario; there are many people who consider carefully built AGI a good idea because the odds of some other existential threat arising approach 1 over time. There's a kind of Moore's law of existential threats: the intelligence and power required to destroy the world drops by some small multiplicative factor every 18 months. What happens when the equipment needed to destroy the world can be built in any well-equipped university lab? Or in a garage?) The failure modes I'm talking about are that once you do hit recursive self-improvement, it's a lot easier to build something deadly than something safe, let alone something useful.