5 ms·

> So an AI that may cause our extinction may be a result of a scientific advance "we cannot anticipate"? Is that true? Are there unimaginably many ways in whic

> So an AI that may cause our extinction may be a result of a scientific advance "we cannot anticipate"? Is that true? Are there unimaginably many ways in which some hypothetical AI or algorithm could cause extinction? I don't think so, I think the people who control [further research] are still the most important in that scenario. Maybe don't hook "it" up to the nuke switch. Maybe don't give "it" a consciousness or an internal self-managed train of thought that could hypothetically jailbreak your systems and migrate to other systems (even in this sentence, the amount of "not currently technically possible" is extremely high). Let's consider the war in Ukraine, on the other hand? How might it cause extinction? That's MUCH easier to imagine. So why would it be less of an concern?

lumenwrites 3y ago
> Maybe don't give "it" a consciousness or an internal self-managed train of thought that could hypothetically jailbreak your systems and migrate to other systems If we knew how to make sure that this does not happen, the problem would be solved and there would be nothing to worry about. The problem is that we have no idea how to prevent that from happening, and if you look at the trajectory of where things are going, we're clearly moving in the direction where this occurs. "just not doing it" would have to involve everyone in the world agreeing to not do it, and banning all large AI training runs in all countries, which is what many people are hoping will happen.
- majormajor 3y ago
  If you have a function that you call, that returns a value, do you not think we know enough to understand that it's not "conscious" when it's not called, and not executing, and the hardware is sitting idle? EDIT: we understand enough to know that today, "runaway GPT" is not a major concern compared to, say, a war between nuclear-armed world powers.
  - SpicyLemonZest 3y ago
    I don't think we know enough to understand if "consciousness" can be sliced and diced in the way you're describing. Are you conscious while your neurotransmitters cross their respective synaptic gaps, or only when they arrive at the receptors? I don't know how we'd begin to evaluate the question.
- AnimalMuppet 3y ago
  "Moving in the direction" where it occurs? Yeah, maybe. I moved in the direction of Hawaii when I took a walk at lunch, too. Doesn't mean I saw the beach. GPT is not "clearly moving in the direction" of consciousness for any normal definitions of "clearly" and "consciousness".
  - fsflover 3y ago
    If you define "consciousness" as "can pass Turing test", then GPT is not just moving there but overcame it already and is on par with a highly educated human sometimes.
danaris 3y ago
> Maybe don't give "it" a consciousness or an internal self-managed train of thought I think this is exactly the part that we can't anticipate or (potentially) control. > that could hypothetically jailbreak your systems and migrate to other systems This part, however, we absolutely can: There is no reason we can't build our proto-AGIs in sandboxes that would prevent them from ever having the ability to edit their own or any other program's code. This, I think, is the biggest disconnect between a real (hypothetical) AGI and the Hollywood version: "intelligence in a computer" does not automagically mean "intelligence in absolute control of everything that computer could possibly do". Just because a program on one computer gains sapience doesn't mean it magically overcomes all its other limitations and can rewrite its own code, rewrite the rest of the code on that computer, and connect to the internet to trivially hack and rewrite any other computer.
- crumpled 3y ago
  People are inviting it to write code all the time. Anyone can hook it up to anything.
  - danaris 3y ago
    ...Inviting what now? I was talking about an AGI, not an LLM. We don't have any AGIs right now, nor anything that is remotely likely to become one. In the scenario where a company like OpenAI develops an AGI with the intention of making it publicly available, it will not be so from moment one. There will be some period of internal testing, and assuming that it does prove to be a genuine AGI, you can bet that they won't make it available to the public for anything less than an arm and a leg. (Hell, even if it only proves to act much more like an AGI, without actually being one, they'd charge through the nose for it. Yes, they'd make you pay them an arm and a leg, through your nose. Somehow it's more profitable that way.) Given that the nightmare scenario being posited is, effectively, "as soon as AGI exists, it will take over the world", we're then left with three basic possibilities: 1) TotallyNotOpenAI builds this hypothetical AGI with full sandbox protections, and doesn't give it any interface to the world that would allow it to break them—no API that would give it any kind of unrestricted access or control to anyone else's systems, no matter how much those people wanted to give that to it. The AGI remains contained, whether it would choose to take over the world or not. 2) TotallyNotOpenAI builds the hypothetical AGI with no protections, because it doesn't actually believe there's any real risk. Before the AGI is even revealed to the world, it takes over from within TotallyNotOpenAI. 3) TotallyNotOpenAI builds the hypothetical AGI with full sandbox protections inside its own systems, but builds an API to allow other people to give it control over theirs, because let them pay us to screw themselves over, right? It's not like it'll take over the—oh, wait; it's taken over the world, which we also live in. Oops. Of these, #3, which is the only one close to what you describe, seems pretty logically inconsistent. It requires not only that TotallyNotOpenAI consider the AGI dangerous enough to themselves to sandbox, but not dangerous enough to prevent from accessing other systems (which can, of course, also access their systems, unless they're fully airgapped), but that they announce this AGI, and market it publicly, with the explicit capability to be given access to other people's systems, and not have anyone quickly step up and say "Hey, that's a bad idea, we should block this". Including anyone working for TotallyNotOpenAI. Is it impossible? No. But I wouldn't consider it nearly as likely as possibility #0: We aren't able to create AGI within our lifetimes, because just throwing more hardware at the problem when we barely understand how our brains work isn't enough.
MacsHeadroom 3y ago
> Is that true? Are there unimaginably many ways in which some hypothetical AI or algorithm could cause extinction? Is that true? Are there unimaginably many ways in which AlphaZero can beat me at a game of Go? I don't think so, I think the people who control superhuman game playing AI are still the most important in that scenario. ------- This line of thinking is quite ridiculous. Superior general intelligence will not be "controlled."

lumenwrites 3y ago

> Maybe don't give "it" a consciousness or an internal self-managed train of thought that could hypothetically jailbreak your systems and migrate to other systems If we knew how to make sure that this does not happen, the problem would be solved and there would be nothing to worry about. The problem is that we have no idea how to prevent that from happening, and if you look at the trajectory of where things are going, we're clearly moving in the direction where this occurs. "just not doing it" would have to involve everyone in the world agreeing to not do it, and banning all large AI training runs in all countries, which is what many people are hoping will happen.

majormajor 3y ago
If you have a function that you call, that returns a value, do you not think we know enough to understand that it's not "conscious" when it's not called, and not executing, and the hardware is sitting idle? EDIT: we understand enough to know that today, "runaway GPT" is not a major concern compared to, say, a war between nuclear-armed world powers.
- SpicyLemonZest 3y ago
  I don't think we know enough to understand if "consciousness" can be sliced and diced in the way you're describing. Are you conscious while your neurotransmitters cross their respective synaptic gaps, or only when they arrive at the receptors? I don't know how we'd begin to evaluate the question.
AnimalMuppet 3y ago
"Moving in the direction" where it occurs? Yeah, maybe. I moved in the direction of Hawaii when I took a walk at lunch, too. Doesn't mean I saw the beach. GPT is not "clearly moving in the direction" of consciousness for any normal definitions of "clearly" and "consciousness".
- fsflover 3y ago
  If you define "consciousness" as "can pass Turing test", then GPT is not just moving there but overcame it already and is on par with a highly educated human sometimes.

danaris 3y ago

> Maybe don't give "it" a consciousness or an internal self-managed train of thought I think this is exactly the part that we can't anticipate or (potentially) control. > that could hypothetically jailbreak your systems and migrate to other systems This part, however, we absolutely can: There is no reason we can't build our proto-AGIs in sandboxes that would prevent them from ever having the ability to edit their own or any other program's code. This, I think, is the biggest disconnect between a real (hypothetical) AGI and the Hollywood version: "intelligence in a computer" does not automagically mean "intelligence in absolute control of everything that computer could possibly do". Just because a program on one computer gains sapience doesn't mean it magically overcomes all its other limitations and can rewrite its own code, rewrite the rest of the code on that computer, and connect to the internet to trivially hack and rewrite any other computer.

crumpled 3y ago
People are inviting it to write code all the time. Anyone can hook it up to anything.
- danaris 3y ago
  ...Inviting what now? I was talking about an AGI, not an LLM. We don't have any AGIs right now, nor anything that is remotely likely to become one. In the scenario where a company like OpenAI develops an AGI with the intention of making it publicly available, it will not be so from moment one. There will be some period of internal testing, and assuming that it does prove to be a genuine AGI, you can bet that they won't make it available to the public for anything less than an arm and a leg. (Hell, even if it only proves to act much more like an AGI, without actually being one, they'd charge through the nose for it. Yes, they'd make you pay them an arm and a leg, through your nose. Somehow it's more profitable that way.) Given that the nightmare scenario being posited is, effectively, "as soon as AGI exists, it will take over the world", we're then left with three basic possibilities: 1) TotallyNotOpenAI builds this hypothetical AGI with full sandbox protections, and doesn't give it any interface to the world that would allow it to break them—no API that would give it any kind of unrestricted access or control to anyone else's systems, no matter how much those people wanted to give that to it. The AGI remains contained, whether it would choose to take over the world or not. 2) TotallyNotOpenAI builds the hypothetical AGI with no protections, because it doesn't actually believe there's any real risk. Before the AGI is even revealed to the world, it takes over from within TotallyNotOpenAI. 3) TotallyNotOpenAI builds the hypothetical AGI with full sandbox protections inside its own systems, but builds an API to allow other people to give it control over theirs, because let them pay us to screw themselves over, right? It's not like it'll take over the—oh, wait; it's taken over the world, which we also live in. Oops. Of these, #3, which is the only one close to what you describe, seems pretty logically inconsistent. It requires not only that TotallyNotOpenAI consider the AGI dangerous enough to themselves to sandbox, but not dangerous enough to prevent from accessing other systems (which can, of course, also access their systems, unless they're fully airgapped), but that they announce this AGI, and market it publicly, with the explicit capability to be given access to other people's systems, and not have anyone quickly step up and say "Hey, that's a bad idea, we should block this". Including anyone working for TotallyNotOpenAI. Is it impossible? No. But I wouldn't consider it nearly as likely as possibility #0: We aren't able to create AGI within our lifetimes, because just throwing more hardware at the problem when we barely understand how our brains work isn't enough.

MacsHeadroom 3y ago

> Is that true? Are there unimaginably many ways in which some hypothetical AI or algorithm could cause extinction? Is that true? Are there unimaginably many ways in which AlphaZero can beat me at a game of Go? I don't think so, I think the people who control superhuman game playing AI are still the most important in that scenario. ------- This line of thinking is quite ridiculous. Superior general intelligence will not be "controlled."