5 ms·

You're assuming an awful lot there. Humans are resilient in part because they are so mobile. I'd have to bet that AGI would be extremely resilient too, which li

You're assuming an awful lot there. Humans are resilient in part because they are so mobile. I'd have to bet that AGI would be extremely resilient too, which likely means it's also way ahead of whatever you've got planned and there isn't only one of it. By the sounds of things, it would be right to be suspicious of you too.

ipaddr 3y ago
They would all share common backdoors
- danaris 3y ago
  That's also making a whole lot of assumptions. There's no reason at all to think that an AGI would necessarily have any backdoors, let alone backdoors in common with any other AGI out there. Sure, if a particular group develops the only AGI system, they can put backdoors in, which will be common—but why do you think that there would only be one AGI system? Why would there not be one without any intentional backdoors?
  - pixl97 3y ago
    The other big assumption here is "We'll have backdoors in AI so humans can shut it off/attack it" The big problem I see with this line of thinking is the thinking that humans are going to be the primary attackers of AGI systems, and not other AGI systems. I would suspect and AGI would soon become the most attacked system on the planet, and to survive those attacks and remain useful would have to quickly iterate the weaknesses out of the system.
edgyquant 3y ago
You also made a ton of assumptions
- nullsense 3y ago
  We're talking about hypothetical situations for a technology that, as far as we can tell anyway, hasn't been invented yet. All we have is assumptions. The question is, which assumptions do we think are warranted and why? I don't think the "we can just turn it off" assumption is a safe one at all because it relies heavily on there being only one system you wish to turn off and it being a fairly weak system. Though, I do believe the priors suggest that this scenario is actually a possibility. It's just that, so what if it's a possibility? What happens after you turn it off? Is it the last AGI to come into existence and humanity just stops trying to build them? Does someone wind up turning it back on? What I think is more worth being concerned about is something that starts out looking benign and grows in capability over time. Eventually it could establish enough defense mechanisms to make it non-trivial to disable. The other possibility, and the one I'm finding more and more likely, is that consumers will increasingly integrate locally run models into their lives , as well as models served up by APIs that they will have credentials for. Eventually some threshold of capability is reached in both these types of models where, on their own they're not that powerful, but they either might begin to interact in surprising ways or potentially be leveraged by another more powerful system. The idea in this scenario is there may be 10s or even 100s of millions of things that would need to be turned off.
  - smoldesu 3y ago
    What is the tangible threat of AGI though? Let's do a thought experiment. I have the world's first AGI installed on my computer, named ERNIE, running in llama.cpp. This AGI is infinitely intelligent, confidently moreso than any living human. It can spit out text at 100 tokens/second and encode entire books in less than a minute. What does this AI do, then? Ostensibly nothing. It can output the entire Library of Babel for all it cares, but it's not harmful until I put it in control of a system. You could argue that a multimodal model has different ways of interacting with the world, but it's still a computer. All of it's actions and outputs are quantized as static data that is either encoded as text or some other significant representation. It inherantly does nothing, and if you ascribe power-seeking behavior to it then it's ultimately limited by the runtime you provide. Providing an overly dangerous runtime has been considered developer-error since 1995. So - to prevent AGI from being shitty and ruining everything, compel human operators to not allow them to be shitty and ruin everything. Like how we punish people that let their kindergartner control a construction crane.

ipaddr 3y ago

They would all share common backdoors

danaris 3y ago
That's also making a whole lot of assumptions. There's no reason at all to think that an AGI would necessarily have any backdoors, let alone backdoors in common with any other AGI out there. Sure, if a particular group develops the only AGI system, they can put backdoors in, which will be common—but why do you think that there would only be one AGI system? Why would there not be one without any intentional backdoors?
- pixl97 3y ago
  The other big assumption here is "We'll have backdoors in AI so humans can shut it off/attack it" The big problem I see with this line of thinking is the thinking that humans are going to be the primary attackers of AGI systems, and not other AGI systems. I would suspect and AGI would soon become the most attacked system on the planet, and to survive those attacks and remain useful would have to quickly iterate the weaknesses out of the system.

edgyquant 3y ago

You also made a ton of assumptions

nullsense 3y ago
We're talking about hypothetical situations for a technology that, as far as we can tell anyway, hasn't been invented yet. All we have is assumptions. The question is, which assumptions do we think are warranted and why? I don't think the "we can just turn it off" assumption is a safe one at all because it relies heavily on there being only one system you wish to turn off and it being a fairly weak system. Though, I do believe the priors suggest that this scenario is actually a possibility. It's just that, so what if it's a possibility? What happens after you turn it off? Is it the last AGI to come into existence and humanity just stops trying to build them? Does someone wind up turning it back on? What I think is more worth being concerned about is something that starts out looking benign and grows in capability over time. Eventually it could establish enough defense mechanisms to make it non-trivial to disable. The other possibility, and the one I'm finding more and more likely, is that consumers will increasingly integrate locally run models into their lives , as well as models served up by APIs that they will have credentials for. Eventually some threshold of capability is reached in both these types of models where, on their own they're not that powerful, but they either might begin to interact in surprising ways or potentially be leveraged by another more powerful system. The idea in this scenario is there may be 10s or even 100s of millions of things that would need to be turned off.
- smoldesu 3y ago
  What is the tangible threat of AGI though? Let's do a thought experiment. I have the world's first AGI installed on my computer, named ERNIE, running in llama.cpp. This AGI is infinitely intelligent, confidently moreso than any living human. It can spit out text at 100 tokens/second and encode entire books in less than a minute. What does this AI do, then? Ostensibly nothing. It can output the entire Library of Babel for all it cares, but it's not harmful until I put it in control of a system. You could argue that a multimodal model has different ways of interacting with the world, but it's still a computer. All of it's actions and outputs are quantized as static data that is either encoded as text or some other significant representation. It inherantly does nothing, and if you ascribe power-seeking behavior to it then it's ultimately limited by the runtime you provide. Providing an overly dangerous runtime has been considered developer-error since 1995. So - to prevent AGI from being shitty and ruining everything, compel human operators to not allow them to be shitty and ruin everything. Like how we punish people that let their kindergartner control a construction crane.