4 ms·
"you've already lost and have zero hope before you know what happened". That seems like a huge leap. You don't think anyone would notice anything at all? Or is
by TheSpiceIsLife 10y ago
"you've already lost and have zero hope before you know what happened".
That seems like a huge leap. You don't think anyone would notice anything at all? Or is this supposed to be a slippery slop argument?
The far broader target is simply that the AGI has a goal function, you're not part of the goal function, and you're made of matter that isn't optimally organized to meet the goal function.
The Borg? Again, no one would care we're being optimised away? No one would notice? Maybe it would just ignore us. Maybe it'd want to be our friend?
is that the success path for AGI solves literally every single problem humans currently face
This must be the bit that made the whole comment feel off. If AGI solves literally every single problem humans currently face, how can AGI be a problem? How are you resolving the contradiction here?
- idlewords 10y agoThere's a whole quasi-religious belief system in the parent comment that you seem to be unfamiliar with, that a lot of people here espouse. That may be why you're being downvoted. Very basically, it goes like this: - the space of possible minds is very vast. We are animals that evolved in one set of circumstances, and so our minds (and thoughts we can think) occupy only a tiny corner of the space. - an artificial AI would operate on computer-like timescales, not human like ones. Therefore we would have only hours to react before it got control of the physical world (hacking the internet, making nanothings, ordering and synthesizing DNA). - an AI could (and would) recursively self-improve until it hit some kind of physical limit very far from our current capabilities. From a human standpoint, it would have God-like intelligence - such an AI could model people and their motivations successfully enough to persuade them to do whatever it wanted - because the space of minds is so vast, a generic AI would have goals that would be inscrutable and alien to us. To survive as a species, we would have to design them in from the beginning, in a way that survives all the levels of recursive self-redesign. There are too many assumptions about the nature of intelligence to unpack here, but I believe it fairly describes what a lot of people believe about AI, thanks to people like Yudkowsky who have a flair for writing about it engagingly.
- dunkelheit 10y agoThanks for a concise summary. I guess these assumptions don't appeal to me because they are mostly justified by Pascal-wager-like argument: "if they are true and we disregard them, we are fucked; so we'd better assume they are true". Claims of recursively reaching godlike intelligence seem particularly dubious to me. Why would AGI be superhuman at all? What if the hypothetical paperclip maximizer produces one paperclip and then dies out of exhaustion? Why would it recursively self-improve to a godlike status? Humans can recursively self-improve to an extent too and it doesn't happen to them. Why would it be particularly interested in recursive self improvement (this one seems to be imported straight from a particular human value system)? I guess this skeptical way of thinking is part of reflexes of an engineer. When presented with the whitepaper of a database with miraculous consistency properties you don't immediately start to fantasize about the possible applications. Instead you ask "Can I play with it? Gimme the demo. Does it pass jepsen?"
- idlewords 10y agoHumans can't recursively self-improve in the sense AI adherents mean it, because we are stuck in our meat brains and can't redesign them or move to faster hardware (yet!, they'll say). What's being smuggled under the philosophical carpet here is exactly what you point out. There's no guarantee that an AI would be a humorless utilitarian rationalist obsessed with self-improvement. It's also possible that general intelligence is a tradeoff against something else (like existential despair, or a tendency to veer off into writing extropian fanfic) and entities hundreds, or thousands, or tens of times smarter than human beings can't exist. Scio me nihil scire.
- JoshTriplett 10y agoSkepticism is an absolutely appropriate response. However, failing in the way you describe isn't a massive problem, it's just simple failure. It's not the worst-case scenario. And it's appropriate to consider and plan for the worst-case scenario, especially when that scenario reaches the existential-threat level. I absolutely agree that it's very hard to get to recursive self-improvement. And until you get there, certain classes of failure modes can't really happen. (Others can, like the grey goo scenario; there are many people who consider carefully built AGI a good idea because the odds of some other existential threat arising approach 1 over time. There's a kind of Moore's law of existential threats: the intelligence and power required to destroy the world drops by some small multiplicative factor every 18 months. What happens when the equipment needed to destroy the world can be built in any well-equipped university lab? Or in a garage?) The failure modes I'm talking about are that once you do hit recursive self-improvement, it's a lot easier to build something deadly than something safe, let alone something useful.
- JoshTriplett 10y ago> The Borg? More like smarter grey-goo. The classic idea of a nanotechnological disaster ("grey goo") is building a tiny self-replicating robot that consumes matter to replicate. It's "smart" enough to disassemble matter to construct more of itself. Imagine something with similar "smarts", enough to work towards a much more complex goal ("build and expand into more compute nodes to compute faster"), but with absolutely no term in its value function related to humans. You're not its enemy, any more than a rock is its enemy; it doesn't have enemies, it just has matter not yet in the shape of more compute nodes. Your existence or non-existence isn't something it notices. > Again, no one would care we're being optimised away? No one would notice? You can notice something without having any ability to stop it. You can notice an explosion happening right next to you, right before you're consumed by it. You can notice everything dissolving into component atoms, but you can't do anything about it. The science fiction scenario is "build an army of killer robots"; that's an understandable threat, and one you can imagine fighting against. The more likely scenario is "dissolve all adjacent matter into component atoms and rearrange it". > Maybe it'd want to be our friend? (Aside: my first reaction to this was to think of the whale from Hitchhiker's Guide to the Galaxy: "Ground! Ha! I wonder if it'll be friends with me? Hello, Ground!") A recognizably human-like mind (let alone a concept of "friend") is a ludicrously tiny possibility, compared to being utterly inscrutable. As much as people like to anthropomorphize computers, an AGI isn't necessarily a "person" in any recognizable sense. The most likely kind of failed AGI wouldn't have a useful concept of "human", or "friend", or "life". (There's a separate tier of failed AGI possibilities that does have a concept of "human" but an utterly broken value system; those are the ones that lead to the "worse than death" scenarios, rather than just the "destroyed by accident" scenarios.) > If AGI solves literally every single problem humans currently face, how can AGI be a problem? An AGI with the correct value system would. And an AGI with the correct value system is the most reliable protection against an AGI with the wrong one; by the nature of such a value system, it's unlikely that there'd ever be a second AGI, rather than just extensions of the first.
- dunkelheit 10y agoAre humans considered AGI? Reading https://intelligence.org/2013/08/11/what-is-agi/ https://intelligence.org/2013/08/11/what-is-agi/ I'd say yes. Then why doesn't all this scary stuff like godlike status and instant recursive self-improvement happen to us? Personally, I think that AGI smarter (maybe much smarter) than humans is possible and even probable, but assuming that it will reach physical limits of intelligence and will do it quickly seems to require a leap of faith.