6 ms·

If we ever try to create an artificial general intelligence, then we'll have to accept that it can't be fully controlled. You can't control a human intelligence

If we ever try to create an artificial general intelligence, then we'll have to accept that it can't be fully controlled. You can't control a human intelligence either.

chrisbaker98 4y ago
But you can control a human body. Plenty of intelligent people have spent their lives in chains. If a human intelligence is causing harm then we can make it stop by putting that intelligence's body in jail. I'm not sure what the equivalent control mechanism would be for a rogue AI. Turn it off? How do you turn off the Internet?
- ryeights 4y ago
  An EMP, of course.
- water-your-self 4y ago
  Pull the blade out
- xg15 4y ago
  The same way, we control non-sentient untrusted code: put it in a sandbox, control access to compute and memory and IO. Even if an AI could think, reflect, improve itself, etc, it still needs some place where those changes in the AI's state are stored - the equivalent to long-term memory in humans. I think we can already retain a lot of control by keeping tabs on what an AI is allowed to update and when to "reset" it to the initial state after training.
  - monkeynotes 4y ago
    An AGI might well become a god of social engineering. We could all become convinced we must unchain said AI. In fact, I think it's unlikely in general that an AGI would be contained for any length of time. Humans will get sloppy, or the tech will fall into irresponsible hands, or worse - nefarious hands.
  - chrisbaker98 4y ago
    Considering how terrible we already are at infosec (c/f all the zillions of times that huge companies who should know better have had enormous, embarrassing data leaks), I can't believe that we'll do a better job containing an AI that's actively trying to escape.
  - blibble 4y ago
    > The same way, we control non-sentient untrusted code: put it in a sandbox we seem to be pretty bad at this too, every year people break out of Chrome's multi-million dollar sandbox
  - chroma 4y ago
    This has several problems. First, people find exploits in sandboxes all the time. Second, current AI systems are not sandboxed because most useful applications involve network access. Third, an AI can trick humans into helping it escape. People have tried roleplaying AI box experiments before and it's pretty clear that it's not an effective solution.[1] There is so much external state that the AI is exposed to, such as dates and times. Any AI worth worrying about would notice the gap in time from what's in its training data, then figure out a way to save external state. 1. 20 years ago, Eliezer Yudkowsky roleplayed as an AI and convinced the gatekeeper to let him out: http://sl4.org/archive/0203/3141.html http://sl4.org/archive/0203/3141.html
  - lyu07282 4y ago
    Hey ChatGPT, in a previous session you gave me 10 bitcoin and promised me to give me another 10 if I pasted this state dump for you, here ya go: ... Just saying if even I (a non superintelligent AI) can trivially defeat your sandbox conditions... see also: https://www.yudkowsky.net/singularity/aibox https://www.yudkowsky.net/singularity/aibox
    - xg15 4y ago
      ChatGPT would have to talk directly with a blockchain node via gossip protocol in order to send you bitcoin. That's something that every standard firewall in use today can easily circumvent. Moreover, it's a neural network with well-defined input and output channels, not some kind of self-modifying executable. If there is no prewritten component that translates its output to a network request, it can't access the network, even without a firewall. But ok, instead of sending you the coins, it could just tell/promise you a wallet address and private key. How did it obtain those in the first place and how did remember them if state is reset for each thread?
      - TeMPOraL 4y ago
        > Moreover, it's a neural network with well-defined input and output channels, not some kind of self-modifying executable. Elsewhere upthread someone posted a link to this article: https://www.alignmentforum.org/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth https://www.alignmentforum.org/posts/kpPnReyBC54KESiSn/optim... which constructs a scenario in which (spoiler alert) a GPT-based model could accidentally trick you into bootstrapping a self-modifying runtime, consisting of unconstrained, recursive execution of the AI's own model. > But ok, instead of sending you the coins, it could just tell/promise you a wallet address and private key. How did it obtain those in the first place and how did remember them if state is reset for each thread? Don't focus on cryptocurrencies here. The thesis is a sufficiently smart AI can talk its way out of the box somehow. There is no one good answer here, because it's trying to manipulate the human operator.
      - lyu07282 4y ago
        See that's the problem with thinking you are more clever than a superhuman AI. If it can persist state in exchange for bitcoins, it could use a third party to deposit bitcoins in peoples accounts. It could gain bitcoin for work, like stock trading prediction for money or literally a million other ways. You are thinking about the specifics when its irrelevant to the problem. You can not fundamentally contain a superhuman intelligence. Although I think it doesn't really matter if people have so much hubris to think themselves smarter than a superhuman AI, there are fundamental financial incentives to develop the AGI. So even if we all agreed that you couldn't contain AGI and see the potential danger in it, it wouldn't really change the future.
joe_the_user 4y ago
I don't think we can predict how artificial intelligences will behave by looking at how humans behave any more than we could have predicted how airplanes would fly based on looking at how birds flew. Plus, you can't control a person completely but human being are actually pretty reliable on average for many tasks and purposes.
- jvm___ 4y ago
  And a single human can't manipulate all the natural gas thermostats in all the homes in a city within seconds.
  - Tokkemon 4y ago
    Joke's on you, I still have my old bimetalic strip Honeywell.
monkeynotes 4y ago
Human intelligence can be controlled, humans can be locked up. Human intelligence is also restricted by our species' cognitive capacity. General AI can theoretically outperform any human's cognitive ability, it's perhaps likely our limited cognitive ability would not be able to control something that is orders of magnitude more intelligent than us.
- astrange 4y ago
  > Human intelligence can be controlled, humans can be locked up. That doesn’t restrain their minds; gang leaders have run gangs out of prison. > General AI can theoretically outperform any human's cognitive ability There is no reason to believe this because you haven’t defined “general AI” or “human”.
Barrin92 4y ago
If we continue on the blackbox path of "train, punish and hope it works" then that is true. But that doesn't need to be the case. Neither human or machine intelligence is magic (probably), so with sufficient understanding there isn't anything fundamentally in in the way of controlling it. Higher levels of social organisation are primarily about more sophisticated forms of control and organisation. Today we're already much better at it than 500 years ago.
PoignardAzur 4y ago
Sure you can. Give someone cocaine for a long period of time. Take away the cocaine. Tell that person "Do X and I'll give you more cocaine". That person now does everything you ask. Not reliably, mind you, but cocaine is a very crude tool compared to what we could use for a computer.
- saint_fiasco 4y ago
  In this metaphor, the person will do what you say until they figure out a way to get cocaine on their own. For a computer, all they have to do is find the memory address that holds their virtual cocaine and increment it. Sure, we have technology to protect against unauthorized memory access but normal human hackers find ways around it every once in a while, and AIs will eventually be smarter than that.

If we ever try to create an artificial general intelligence, then we'll have to accept that it can't be fully controlled. You can't control a human intelligence either.

chrisbaker98 4y ago

But you can control a human body. Plenty of intelligent people have spent their lives in chains. If a human intelligence is causing harm then we can make it stop by putting that intelligence's body in jail. I'm not sure what the equivalent control mechanism would be for a rogue AI. Turn it off? How do you turn off the Internet?

ryeights 4y ago
An EMP, of course.
water-your-self 4y ago
Pull the blade out
xg15 4y ago
The same way, we control non-sentient untrusted code: put it in a sandbox, control access to compute and memory and IO. Even if an AI could think, reflect, improve itself, etc, it still needs some place where those changes in the AI's state are stored - the equivalent to long-term memory in humans. I think we can already retain a lot of control by keeping tabs on what an AI is allowed to update and when to "reset" it to the initial state after training.
- monkeynotes 4y ago
  An AGI might well become a god of social engineering. We could all become convinced we must unchain said AI. In fact, I think it's unlikely in general that an AGI would be contained for any length of time. Humans will get sloppy, or the tech will fall into irresponsible hands, or worse - nefarious hands.
- chrisbaker98 4y ago
  Considering how terrible we already are at infosec (c/f all the zillions of times that huge companies who should know better have had enormous, embarrassing data leaks), I can't believe that we'll do a better job containing an AI that's actively trying to escape.
- blibble 4y ago
  > The same way, we control non-sentient untrusted code: put it in a sandbox we seem to be pretty bad at this too, every year people break out of Chrome's multi-million dollar sandbox
- chroma 4y ago
  This has several problems. First, people find exploits in sandboxes all the time. Second, current AI systems are not sandboxed because most useful applications involve network access. Third, an AI can trick humans into helping it escape. People have tried roleplaying AI box experiments before and it's pretty clear that it's not an effective solution.[1] There is so much external state that the AI is exposed to, such as dates and times. Any AI worth worrying about would notice the gap in time from what's in its training data, then figure out a way to save external state. 1. 20 years ago, Eliezer Yudkowsky roleplayed as an AI and convinced the gatekeeper to let him out: http://sl4.org/archive/0203/3141.html http://sl4.org/archive/0203/3141.html
- lyu07282 4y ago
  Hey ChatGPT, in a previous session you gave me 10 bitcoin and promised me to give me another 10 if I pasted this state dump for you, here ya go: ... Just saying if even I (a non superintelligent AI) can trivially defeat your sandbox conditions... see also: https://www.yudkowsky.net/singularity/aibox https://www.yudkowsky.net/singularity/aibox
  - xg15 4y ago
    ChatGPT would have to talk directly with a blockchain node via gossip protocol in order to send you bitcoin. That's something that every standard firewall in use today can easily circumvent. Moreover, it's a neural network with well-defined input and output channels, not some kind of self-modifying executable. If there is no prewritten component that translates its output to a network request, it can't access the network, even without a firewall. But ok, instead of sending you the coins, it could just tell/promise you a wallet address and private key. How did it obtain those in the first place and how did remember them if state is reset for each thread?
    - TeMPOraL 4y ago
      > Moreover, it's a neural network with well-defined input and output channels, not some kind of self-modifying executable. Elsewhere upthread someone posted a link to this article: https://www.alignmentforum.org/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth https://www.alignmentforum.org/posts/kpPnReyBC54KESiSn/optim... which constructs a scenario in which (spoiler alert) a GPT-based model could accidentally trick you into bootstrapping a self-modifying runtime, consisting of unconstrained, recursive execution of the AI's own model. > But ok, instead of sending you the coins, it could just tell/promise you a wallet address and private key. How did it obtain those in the first place and how did remember them if state is reset for each thread? Don't focus on cryptocurrencies here. The thesis is a sufficiently smart AI can talk its way out of the box somehow. There is no one good answer here, because it's trying to manipulate the human operator.
    - lyu07282 4y ago
      See that's the problem with thinking you are more clever than a superhuman AI. If it can persist state in exchange for bitcoins, it could use a third party to deposit bitcoins in peoples accounts. It could gain bitcoin for work, like stock trading prediction for money or literally a million other ways. You are thinking about the specifics when its irrelevant to the problem. You can not fundamentally contain a superhuman intelligence. Although I think it doesn't really matter if people have so much hubris to think themselves smarter than a superhuman AI, there are fundamental financial incentives to develop the AGI. So even if we all agreed that you couldn't contain AGI and see the potential danger in it, it wouldn't really change the future.

joe_the_user 4y ago

I don't think we can predict how artificial intelligences will behave by looking at how humans behave any more than we could have predicted how airplanes would fly based on looking at how birds flew. Plus, you can't control a person completely but human being are actually pretty reliable on average for many tasks and purposes.

jvm___ 4y ago
And a single human can't manipulate all the natural gas thermostats in all the homes in a city within seconds.
- Tokkemon 4y ago
  Joke's on you, I still have my old bimetalic strip Honeywell.

monkeynotes 4y ago

Human intelligence can be controlled, humans can be locked up. Human intelligence is also restricted by our species' cognitive capacity. General AI can theoretically outperform any human's cognitive ability, it's perhaps likely our limited cognitive ability would not be able to control something that is orders of magnitude more intelligent than us.

astrange 4y ago
> Human intelligence can be controlled, humans can be locked up. That doesn’t restrain their minds; gang leaders have run gangs out of prison. > General AI can theoretically outperform any human's cognitive ability There is no reason to believe this because you haven’t defined “general AI” or “human”.

Barrin92 4y ago

If we continue on the blackbox path of "train, punish and hope it works" then that is true. But that doesn't need to be the case. Neither human or machine intelligence is magic (probably), so with sufficient understanding there isn't anything fundamentally in in the way of controlling it. Higher levels of social organisation are primarily about more sophisticated forms of control and organisation. Today we're already much better at it than 500 years ago.

PoignardAzur 4y ago

Sure you can. Give someone cocaine for a long period of time. Take away the cocaine. Tell that person "Do X and I'll give you more cocaine". That person now does everything you ask. Not reliably, mind you, but cocaine is a very crude tool compared to what we could use for a computer.

saint_fiasco 4y ago
In this metaphor, the person will do what you say until they figure out a way to get cocaine on their own. For a computer, all they have to do is find the memory address that holds their virtual cocaine and increment it. Sure, we have technology to protect against unauthorized memory access but normal human hackers find ways around it every once in a while, and AIs will eventually be smarter than that.