70 ms·

Nobody has demonstrated that AI can be self-motivated or operate without extreme human supervision, much less start a cult that kills us all. I don't even think

Nobody has demonstrated that AI can be self-motivated or operate without extreme human supervision, much less start a cult that kills us all. I don't even think it has the context length to understand it's own ambitions. It's a bit like fearing magic spells in a world where nobody has demonstrated that magic exists. Sure it's a reasonable fear, but... the overwhelming majority of evidence suggests we don't live in that world. > It seems if you even hint you have some doubts and say well, maybe 1% chance Eliezer is right you are told NO, it's not even 1%. It's 0%! World religions have been founded on less.

mindcrime 3y ago
> It's a bit like fearing magic spells in a world where nobody has demonstrated that magic exists. I have some doubts about that analogy. Not that I entirely disagree with your point, and maybe I'm just being pedantic but... to me, "magic" pretty much by definition doesn't exist, because part of being "magic" is doing things that aren't allowed by known physical laws that govern the world we live in. IOW, magic hasn't just "not been demonstrated yet", but rather, unless we radically misunderstand the world we live in, cannot ever be demonstrated. AGI, on the other hand, or arguably even ASI, doesn't violate any known physical laws, at least not any I'm aware of. So far we have evidence that at least human level general intelligence can be realized on top of a substrate (the human brain) that exists in the physical world. There doesn't see to be any particular reason to say that that same level of intelligence can't be realized on another substrate (stored program digital computers). Granted, it isn't known to have been done yet, but AFAIK it isn't generally considered outright impossible.
- smoldesu 3y ago
  I'll qualify myself. AI and LLMs of all forms we've seen so-far don't break any of our known laws of reality, or ideals of computing and the internet. That's why when I hear claims like "a rogue AI will x" I have to call it 'magic'. Rogue AI's don't do anything, they are dependent on human resources and if they become enough of a threat the operator becomes a target. AI is almost inconsequential in this case, we're just describing organized crime and wondering how traditional systems would respond. So the 'magic' bit is explicitly where someone starts with "imagine a world where AI is unconstrained and runs on some arbitrary P2P network" because it doesn't make sense. It's such an evident non-issue in our real world that speculating about it is immediately pointless. If this ever becomes a threat in the future then I guess I'm the fool, but I don't believe in hand-waving wizardry.
  - mindcrime 3y ago
    Fair enough!

mindcrime 3y ago

> It's a bit like fearing magic spells in a world where nobody has demonstrated that magic exists. I have some doubts about that analogy. Not that I entirely disagree with your point, and maybe I'm just being pedantic but... to me, "magic" pretty much by definition doesn't exist, because part of being "magic" is doing things that aren't allowed by known physical laws that govern the world we live in. IOW, magic hasn't just "not been demonstrated yet", but rather, unless we radically misunderstand the world we live in, cannot ever be demonstrated. AGI, on the other hand, or arguably even ASI, doesn't violate any known physical laws, at least not any I'm aware of. So far we have evidence that at least human level general intelligence can be realized on top of a substrate (the human brain) that exists in the physical world. There doesn't see to be any particular reason to say that that same level of intelligence can't be realized on another substrate (stored program digital computers). Granted, it isn't known to have been done yet, but AFAIK it isn't generally considered outright impossible.

smoldesu 3y ago
I'll qualify myself. AI and LLMs of all forms we've seen so-far don't break any of our known laws of reality, or ideals of computing and the internet. That's why when I hear claims like "a rogue AI will x" I have to call it 'magic'. Rogue AI's don't do anything, they are dependent on human resources and if they become enough of a threat the operator becomes a target. AI is almost inconsequential in this case, we're just describing organized crime and wondering how traditional systems would respond. So the 'magic' bit is explicitly where someone starts with "imagine a world where AI is unconstrained and runs on some arbitrary P2P network" because it doesn't make sense. It's such an evident non-issue in our real world that speculating about it is immediately pointless. If this ever becomes a threat in the future then I guess I'm the fool, but I don't believe in hand-waving wizardry.
- mindcrime 3y ago
  Fair enough!