3 ms·
> The Borg? More like smarter grey-goo. The classic idea of a nanotechnological disaster ("grey goo") is building a tiny self-replicating robot that consumes
by JoshTriplett 10y ago
> The Borg?
More like smarter grey-goo. The classic idea of a nanotechnological disaster ("grey goo") is building a tiny self-replicating robot that consumes matter to replicate. It's "smart" enough to disassemble matter to construct more of itself. Imagine something with similar "smarts", enough to work towards a much more complex goal ("build and expand into more compute nodes to compute faster"), but with absolutely no term in its value function related to humans. You're not its enemy, any more than a rock is its enemy; it doesn't have enemies, it just has matter not yet in the shape of more compute nodes. Your existence or non-existence isn't something it notices.
> Again, no one would care we're being optimised away? No one would notice?
You can notice something without having any ability to stop it. You can notice an explosion happening right next to you, right before you're consumed by it. You can notice everything dissolving into component atoms, but you can't do anything about it.
The science fiction scenario is "build an army of killer robots"; that's an understandable threat, and one you can imagine fighting against. The more likely scenario is "dissolve all adjacent matter into component atoms and rearrange it".
> Maybe it'd want to be our friend?
(Aside: my first reaction to this was to think of the whale from Hitchhiker's Guide to the Galaxy: "Ground! Ha! I wonder if it'll be friends with me? Hello, Ground!")
A recognizably human-like mind (let alone a concept of "friend") is a ludicrously tiny possibility, compared to being utterly inscrutable. As much as people like to anthropomorphize computers, an AGI isn't necessarily a "person" in any recognizable sense. The most likely kind of failed AGI wouldn't have a useful concept of "human", or "friend", or "life".
(There's a separate tier of failed AGI possibilities that does have a concept of "human" but an utterly broken value system; those are the ones that lead to the "worse than death" scenarios, rather than just the "destroyed by accident" scenarios.)
> If AGI solves literally every single problem humans currently face, how can AGI be a problem?
An AGI with the correct value system would. And an AGI with the correct value system is the most reliable protection against an AGI with the wrong one; by the nature of such a value system, it's unlikely that there'd ever be a second AGI, rather than just extensions of the first.
- dunkelheit 10y agoAre humans considered AGI? Reading https://intelligence.org/2013/08/11/what-is-agi/ https://intelligence.org/2013/08/11/what-is-agi/ I'd say yes. Then why doesn't all this scary stuff like godlike status and instant recursive self-improvement happen to us? Personally, I think that AGI smarter (maybe much smarter) than humans is possible and even probable, but assuming that it will reach physical limits of intelligence and will do it quickly seems to require a leap of faith.
- JoshTriplett 10y agoWe're not artificial, but we're general intelligence. However, we don't have the ability to rapidly improve ourselves in exponential ways, or to debug and self-improve our thought processes in fundamental ways, or even to multiplicatively improve ourselves by running on faster hardware. We can't perfectly and instantaneously transfer knowledge between people, and we lose large quantities of knowledge and experience by the hundred-fifty-thousand per day. An AGI doesn't have to reach physical limits of intelligence (if there are any) to rapidly become far more intelligent than humans.
- idlewords 10y agoThere are a lot of leaps of faith baked into these scenarios. It appeals to fundamentally religious instincts. Notice that people start with a seemingly sensible set of propositions, and what falls out is the Apocalypse, immortality, heaven (or AI hell) and a sense of urgency, up to and including passing the collection plate. Since you asked about the argument for recursive self-improvement goes like this. Say Alice is a human engineer, who designs an AI (Bob) who is her intellectual equal. Then she can run Bob on faster hardware, and the now-smarter-than-her Bob can repeat the trick to design Clara, and so on. It's AIs all the way up. If you're a certain kind of beardo, you imagine this process could be compressed into a few hours.
- idlewords 10y agoNo natural intelligence has this kind of simplistic 'value function', yet it's taken as a given by AI-doom fantasists that an AI could, and would. My suspicion is that GAI will turn out to be a big hot mess of interdependent systems and drives, just like every other brain. The paperclip maximizer will write songs about paperclips and argue with copies of itself about whether staplers are immoral.
- TheSpiceIsLife 10y agoWhat JoshTriplett is describing is cockroaches. Or maybe bacteria. The only differences between grey-goo and cockroaches is that cockroaches have come up against the limitations of the physical world. That is to say the physical world consists of minerals that have a strong tendency to not dissolve in to their constituant atoms, energy that is difficult to collect and use with any efficiency, and literally millions of other creatures eating your lunch. It seems more plausible that this self replicating grey-goo would simply become cockroach food before it became a problem. Maybe ants would farm it like they do aphids and fungi.