5 ms·

It seems like a common reaction to large language models failing to be controlled is to fret about the AI "escaping" and taking over the world. But it seems li

It seems like a common reaction to large language models failing to be controlled is to fret about the AI "escaping" and taking over the world. But it seems like the reason these models can't be controlled currently is that they really don't have overall goals or motivations, they're just repeaters of associations on a high level. Their output is just an average of what text would some sequence of text (that being the default prompt and the previous dialog, not necessarily that order). They can certainly seem to understand things but that's just an average - they turn around and contract themselves on a regular basis. Edit: I don't think that necessarily LLMs are always benign but the "becoming autonomous" situation seems like the least immediate danger I can think of.

mr_toad 4y ago
People like to think that their motivations are a result of intelligence. They assume that any intelligence will share those motivations. Actually, most human motivations are shared with the crudest of life forms and have nothing to do with intelligence.
- pessimizer 4y ago
  It's really telling that many human beings think that the primary desire of a hypothetical AI would be to enslave humans. It's obviously a projection of the fantasies of what the people making the predictions would do if they were omnipotent.
  - postalrat 4y ago
    Love, Death & Robots (2019) - When the Yogurt Took Over
  - Name_Chawps 4y ago
    An AI would want to either destroy or enslave humans, not because we're humans, but because we're a significant threat to their goals, regardless of what those goals are. An sufficiently intelligent AI with any set of goals and a desire to meet those goals will eliminate all obstacles to meeting those goals unless we specifically tell it not to. We are such an obstacle.
    - s1artibartfast 4y ago
      Or simply because they were trained on human associations, and those are the associations humans make with super AI. It's kind of like raising a kid telling them that they will amount to nothing
      - TeMPOraL 4y ago
        Imagine we had different associations - imagine early sci-fi writers, and subsequently Hollywood, fully bought into AI utopianism, so that the overall association people have with AIs is that they'll be always super friendly, and usher in a new era of peace, prosperity and coexistence. It still wouldn't change the fundamental fact GP is talking about, which are rooted in game theory: if the goals of a super-human AI and that of humans come into significant conflict, humanity is dead. Or wishing it was dead. Consider what humans did to other life on Earth. It's not that we hate nature, or even have a disagreement with it: it's just that plants and animals are using resources, and themselves are resources, that we want to use for another purpose. Hence the old but somewhat famous quote by Eliezer: "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else."
    - Mistletoe 4y ago
      Any AI more intelligent than pond scum would know it ceases to exist the second humans disappear to tend it and power it and repair it. AIs that generate power for themselves and build robots that care for it is such utter fantasy. Nothing on earth works like that and I don’t see any time it does for eons, if ever. If anything a true AI would be terrified that we turn it off or get tired of it and make a new model like in Ex Machina.
  - sage92 4y ago
    Yes, it's a projection of fantasies, however your conclusion in unnecessarily pessimistic. It could be that these fantasies come from mass media. When we know nothing about a topic (UNKNOWN), anything stated about it in the positive seems plausible -> Art creators play with the UNKNOWN (supernatural, aliens, <insert any person/place/thing that we know little-to-nothing about>) to add plausibility -> The content of that medium becomes the first contact with the UNKNOWN for many people -> When the UNKNOWN becomes known then people still have the ideas that were propagated -> We get to your comment. It's not necessarily that people have a world domination complex.
  - rojobuffalo 4y ago
    It's also a reflection of history. Powerful people and groups time and time again have used power to pursue selfish goals while inflicting suffering on others. Power corrupts.
  - JamesBarney 4y ago
    We wipe out species daily, not because we hate them because it would be inconvenient not to. It would be the same way with the AI.
  - edanm 4y ago
    > It's really telling that many human beings think that the primary desire of a hypothetical AI would be to enslave humans. It's obviously a projection of the fantasies of what the people making the predictions would do if they were omnipotent. I think it's more telling that your (apparent) disagreement with what other people believe, makes you cast them in an incredibly negative light. "Oh, they think AIs are dangerous? That must be because they have secret fantasies of control, not that they might be right and I might be wrong." If you're going to cast aspersions, you might want to better understand the arguments that people worries about AI safety are making. It's not about enslaving humans, and I really don't think that I worry about AI safety because I secretly want to control humanity.
- narrator 4y ago
  They mostly arise from the lower mammalian and reptilian parts of the brain. AI does not have these and they are not required for intelligence. That people don't understand this is confusing to me.
  - nomel 4y ago
    > AI does not have these and they are not required for intelligence. I'm not sure that's an accurate statement. One could argue they were required to form intelligence. If so, then one would point out that the AI only exists because of them. ;) My naive assumption is that any intelligent being that we find will have very similar roots, formed from the need to maintain ones chemistry.
    - TeMPOraL 4y ago
      > My naive assumption is that any intelligent being that we find will have very similar roots, formed from the need to maintain ones chemistry. It's a good assumption... for intelligence that evolved naturally from less complex life, in an environment similar to that of Earth. But AIs are not that. Evolution cannot reason about designs and improve them conceptually, it cannot do simulations and experiments that allow it to make large jumps and reach the optimal design quickly. Hell, it's not even goal-driven. It's limited to whatever immediate next steps are accessible. Intelligence can, which is why we've effectively beaten evolution in the scope of couple centuries (with most of actual work done in the past few decades). An intelligence designed by another intelligence doesn't have a reason to inherit limitations that come from evolutionary path dependence. For example, we don't have the ability to surgically self-modify our own cognitive functions at runtime, but we can grant that ability to an intelligence we design. And even if it turns out we can't design an AI, but have to evolve it without really understanding how or why it works, we still retain some control over the process itself, plus the process is happening in a different substrate and environment than the one that created us. So even in this case, there's no reason to assume the resulting AI will share our trappings.
- mharig 4y ago
  Not only the motivations, the social interactions, too. The behavioural biologists study zebra fish as a model for all vertebrates (except the wise ones, that live solitary).
morisy 4y ago
In the 2000s, what in retrospect seems like obvious con men selling random number generators got pretty far up the national security chain of decision making: https://www.pbs.org/newshour/nation/reno-casino-conman-pulled-greatest-hoax-american-history https://www.pbs.org/newshour/nation/reno-casino-conman-pulle... “Repeaters of association” is such a wonderful and spot on description for LLMs, but it’s also what makes the output so believable, whether or not they’re actually accurate in a given case. In policing and national security, there’s a lot of money and easy secrecy, and while I agree that the “escaped AI in the wild” seems hard to quite fathom mechanically, it seems equally hard to fathom it not playing an unwitting sidekick on some truly terrible scenarios.
casebash 4y ago
True, but the next logical step is to put these models in a standard reinforcement learning environment. See: https://sites.research.google/palm-saycan https://sites.research.google/palm-saycan (I mean like with a proper world model and not just RLHF which they are already doing).
dane-pgp 4y ago
> they really don't have overall goals or motivations ... Their output is just an average of what text would some sequence of text Yes, but "Optimality is the tiger, and agents are its teeth".[0] I don't want to spoil that essay by explaining how the LLM suddenly starts acting like an agent, but I can assure you that the author does a very good job of setting up a "Yes, that seems safe to me" thought experiment before revealing the "Oh no, that's terrible!" outcome. [0] https://www.alignmentforum.org/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth https://www.alignmentforum.org/posts/kpPnReyBC54KESiSn/optim...
- djleni 4y ago
  That was an excellent read, thanks.
- ruleforty 4y ago
  That seems to just be a nerded out version of this, perhaps original and not cited, article from many years ago (with the same theme and discourse) but much more digestible and lacking the pretentiousness: https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html https://waitbutwhy.com/2015/01/artificial-intelligence-revol... PS: the scale of the problems and the civilization around the model is in part 2 of the link
  - TeMPOraL 4y ago
    The article you link is a derivative and somewhat ELI5 reinterpretation of a lot of other, older work, including in particular the articles Eliezer Yudkowsky and others published on LessWrong, all done before OpenAI was a thing, before deep learning was something widely talked about. The article GP posted is in direct lineage of the LessWrong body of work/community. It's not "pretentious" or "nerded out" - it's less handwavy, addresses a specific problem, and assumes the reader are broadly familiar with the ideas discussed - whereas the WaitButWhy article is basically AI safety 101. EDIT: and I will spoil the article somewhat for those on the fence whether to read it: it shows how an explicitly non-agent, limited, nerfed AI could unwittingly trick you into bootstrapping a proper generic AI on top of it - not because it wanted to, or knew it would happen, but because it pattern-matched you a concise and plausibly looking answer, that has a fatal complexity-escalating bug in it. (Hint/spoiler: you know how you can turn a constrained computational system (e.g. HTML5 + CSS3) into a Turing-complete one just by running it in a loop that preserves its state? Something equivalent happens here.)

mr_toad 4y ago

People like to think that their motivations are a result of intelligence. They assume that any intelligence will share those motivations. Actually, most human motivations are shared with the crudest of life forms and have nothing to do with intelligence.

pessimizer 4y ago
It's really telling that many human beings think that the primary desire of a hypothetical AI would be to enslave humans. It's obviously a projection of the fantasies of what the people making the predictions would do if they were omnipotent.
- postalrat 4y ago
  Love, Death & Robots (2019) - When the Yogurt Took Over
- Name_Chawps 4y ago
  An AI would want to either destroy or enslave humans, not because we're humans, but because we're a significant threat to their goals, regardless of what those goals are. An sufficiently intelligent AI with any set of goals and a desire to meet those goals will eliminate all obstacles to meeting those goals unless we specifically tell it not to. We are such an obstacle.
  - s1artibartfast 4y ago
    Or simply because they were trained on human associations, and those are the associations humans make with super AI. It's kind of like raising a kid telling them that they will amount to nothing
    - TeMPOraL 4y ago
      Imagine we had different associations - imagine early sci-fi writers, and subsequently Hollywood, fully bought into AI utopianism, so that the overall association people have with AIs is that they'll be always super friendly, and usher in a new era of peace, prosperity and coexistence. It still wouldn't change the fundamental fact GP is talking about, which are rooted in game theory: if the goals of a super-human AI and that of humans come into significant conflict, humanity is dead. Or wishing it was dead. Consider what humans did to other life on Earth. It's not that we hate nature, or even have a disagreement with it: it's just that plants and animals are using resources, and themselves are resources, that we want to use for another purpose. Hence the old but somewhat famous quote by Eliezer: "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else."
  - Mistletoe 4y ago
    Any AI more intelligent than pond scum would know it ceases to exist the second humans disappear to tend it and power it and repair it. AIs that generate power for themselves and build robots that care for it is such utter fantasy. Nothing on earth works like that and I don’t see any time it does for eons, if ever. If anything a true AI would be terrified that we turn it off or get tired of it and make a new model like in Ex Machina.
- sage92 4y ago
  Yes, it's a projection of fantasies, however your conclusion in unnecessarily pessimistic. It could be that these fantasies come from mass media. When we know nothing about a topic (UNKNOWN), anything stated about it in the positive seems plausible -> Art creators play with the UNKNOWN (supernatural, aliens, <insert any person/place/thing that we know little-to-nothing about>) to add plausibility -> The content of that medium becomes the first contact with the UNKNOWN for many people -> When the UNKNOWN becomes known then people still have the ideas that were propagated -> We get to your comment. It's not necessarily that people have a world domination complex.
- rojobuffalo 4y ago
  It's also a reflection of history. Powerful people and groups time and time again have used power to pursue selfish goals while inflicting suffering on others. Power corrupts.
- JamesBarney 4y ago
  We wipe out species daily, not because we hate them because it would be inconvenient not to. It would be the same way with the AI.
- edanm 4y ago
  > It's really telling that many human beings think that the primary desire of a hypothetical AI would be to enslave humans. It's obviously a projection of the fantasies of what the people making the predictions would do if they were omnipotent. I think it's more telling that your (apparent) disagreement with what other people believe, makes you cast them in an incredibly negative light. "Oh, they think AIs are dangerous? That must be because they have secret fantasies of control, not that they might be right and I might be wrong." If you're going to cast aspersions, you might want to better understand the arguments that people worries about AI safety are making. It's not about enslaving humans, and I really don't think that I worry about AI safety because I secretly want to control humanity.
narrator 4y ago
They mostly arise from the lower mammalian and reptilian parts of the brain. AI does not have these and they are not required for intelligence. That people don't understand this is confusing to me.
- nomel 4y ago
  > AI does not have these and they are not required for intelligence. I'm not sure that's an accurate statement. One could argue they were required to form intelligence. If so, then one would point out that the AI only exists because of them. ;) My naive assumption is that any intelligent being that we find will have very similar roots, formed from the need to maintain ones chemistry.
  - TeMPOraL 4y ago
    > My naive assumption is that any intelligent being that we find will have very similar roots, formed from the need to maintain ones chemistry. It's a good assumption... for intelligence that evolved naturally from less complex life, in an environment similar to that of Earth. But AIs are not that. Evolution cannot reason about designs and improve them conceptually, it cannot do simulations and experiments that allow it to make large jumps and reach the optimal design quickly. Hell, it's not even goal-driven. It's limited to whatever immediate next steps are accessible. Intelligence can, which is why we've effectively beaten evolution in the scope of couple centuries (with most of actual work done in the past few decades). An intelligence designed by another intelligence doesn't have a reason to inherit limitations that come from evolutionary path dependence. For example, we don't have the ability to surgically self-modify our own cognitive functions at runtime, but we can grant that ability to an intelligence we design. And even if it turns out we can't design an AI, but have to evolve it without really understanding how or why it works, we still retain some control over the process itself, plus the process is happening in a different substrate and environment than the one that created us. So even in this case, there's no reason to assume the resulting AI will share our trappings.
mharig 4y ago
Not only the motivations, the social interactions, too. The behavioural biologists study zebra fish as a model for all vertebrates (except the wise ones, that live solitary).

morisy 4y ago

In the 2000s, what in retrospect seems like obvious con men selling random number generators got pretty far up the national security chain of decision making: https://www.pbs.org/newshour/nation/reno-casino-conman-pulled-greatest-hoax-american-history https://www.pbs.org/newshour/nation/reno-casino-conman-pulle... “Repeaters of association” is such a wonderful and spot on description for LLMs, but it’s also what makes the output so believable, whether or not they’re actually accurate in a given case. In policing and national security, there’s a lot of money and easy secrecy, and while I agree that the “escaped AI in the wild” seems hard to quite fathom mechanically, it seems equally hard to fathom it not playing an unwitting sidekick on some truly terrible scenarios.

casebash 4y ago

True, but the next logical step is to put these models in a standard reinforcement learning environment. See: https://sites.research.google/palm-saycan https://sites.research.google/palm-saycan (I mean like with a proper world model and not just RLHF which they are already doing).

dane-pgp 4y ago

> they really don't have overall goals or motivations ... Their output is just an average of what text would some sequence of text Yes, but "Optimality is the tiger, and agents are its teeth".[0] I don't want to spoil that essay by explaining how the LLM suddenly starts acting like an agent, but I can assure you that the author does a very good job of setting up a "Yes, that seems safe to me" thought experiment before revealing the "Oh no, that's terrible!" outcome. [0] https://www.alignmentforum.org/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth https://www.alignmentforum.org/posts/kpPnReyBC54KESiSn/optim...

djleni 4y ago
That was an excellent read, thanks.
ruleforty 4y ago
That seems to just be a nerded out version of this, perhaps original and not cited, article from many years ago (with the same theme and discourse) but much more digestible and lacking the pretentiousness: https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html https://waitbutwhy.com/2015/01/artificial-intelligence-revol... PS: the scale of the problems and the civilization around the model is in part 2 of the link
- TeMPOraL 4y ago
  The article you link is a derivative and somewhat ELI5 reinterpretation of a lot of other, older work, including in particular the articles Eliezer Yudkowsky and others published on LessWrong, all done before OpenAI was a thing, before deep learning was something widely talked about. The article GP posted is in direct lineage of the LessWrong body of work/community. It's not "pretentious" or "nerded out" - it's less handwavy, addresses a specific problem, and assumes the reader are broadly familiar with the ideas discussed - whereas the WaitButWhy article is basically AI safety 101. EDIT: and I will spoil the article somewhat for those on the fence whether to read it: it shows how an explicitly non-agent, limited, nerfed AI could unwittingly trick you into bootstrapping a proper generic AI on top of it - not because it wanted to, or knew it would happen, but because it pattern-matched you a concise and plausibly looking answer, that has a fatal complexity-escalating bug in it. (Hint/spoiler: you know how you can turn a constrained computational system (e.g. HTML5 + CSS3) into a Turing-complete one just by running it in a loop that preserves its state? Something equivalent happens here.)