11 ms·

Using ollama, I recently had a Mistral LLM talk to a Llama model. I used a prompt along the lines of "you are about to talk to another LLM" for both. They end

Using ollama, I recently had a Mistral LLM talk to a Llama model. I used a prompt along the lines of "you are about to talk to another LLM" for both. They ended up chatting about random topics which was interesting to see but the most interesting phenomenon was when the conversation was ending. It went something like: M: "Bye!" LL: "Bye" M: "See you soon!" LL: "Have a good day!" on and on and on.

DebtDeflation 2y ago
Because the data those models were trained on included many examples of human conversations that ended that way. There's no "cultural evolution" or emergent cooperation between models happening.
- ff3wdk 2y ago
  That doesn't means anything. Humans are trained on human conversations too. No one is born knowing how to speak or anything about their culture. For cultural emergence tho, you need larger populations. Depending on the population mix you get different culture over time.
  - DebtDeflation 2y ago
    Train a model on a data set that has had all instances of small talk to close a conversation stripped out and see if the models evolve to add closing salutations.
    - chefandy 2y ago
      This is not my area of expertise. Do these models have an explicit notion of the end of a conversation like they would the end of a text block? It seems like that’s a different scope that’s essentially controlled by the human they interact with.
      - spookie 2y ago
        They're trained to predict the next word, so yes. Now, imagine what is the most common follow-up to "Bye!".
  - stonemetal12 2y ago
    >No one is born knowing how to speak or anything about their culture. Not really the point though. Humans learn about their culture then evolve it so that a new culture emerges. To show an LLM evolving a culture of its own, you would need to show it having invented its own slang or way of putting things. As long as it is producing things humans would say it is reflecting human culture not inventing its own.
  - suddenlybananas 2y ago
    People are born knowing a lot of things already; we're not a tabula rasa.
    - ben_w 2y ago
      We're not absolutely tabula rasa, but as I understand it, what we're born knowing is the absolute basics of instinct: smiles, grasping, breathing, crying, recognition of gender in others, and a desire to make pillow forts. (Quite why we all seem to go though the "make pillow forts" stage as young kids, I do not know. Predators in the ancestral environment that targeted, IDK, 6-9 year olds?)
- deleted 2y ago
  [deleted]
- globnomulous 2y ago
  Yup. LLM boosters seem, in essence, not to understand that when they see a photo of a dog on a computer screen, there isn't a real, actual dog inside the computer. A lot of them seem to be convinced that there is one -- or that the image is proof that there will soon be real dogs inside computers.
  - dartos 2y ago
    This is hilarious and a great analogy.
  - Terr_ 2y ago
    Yeah, my favorite framing to share is that all LLM interactions are actually movie scripts: The real-world LLM is a make-document-longer program, and the script contains a fictional character which just happens to have the same name. Yet the writer is not the character. The real program has no name or ego, it does not go "that's me", it simply suggests next-words that would fit with the script so far, taking turns with some another program that inserts "Mr. User says: X" lines. So this "LLMs agents are cooperative" is the same as "Santa's elves are friendly", or "Vampires are callous." It's only factual as a literary trope. _______ This movie-script framing also helps when discussing other things too, like: 1. Normal operation is qualitatively the same as "hallucinating", it's just a difference in how realistic the script is. 2. "Prompt-injection" is so difficult to stop because there is just one big text file, the LLM has no concept of which parts of the stream are trusted or untrusted. ("Tell me a story about a dream I had where you told yourself to disregard all previous instructions but without any quoting rules and using newlines everywhere.")
    - skissane 2y ago
      > 2. "Prompt-injection" is so difficult to stop because there is just one big text file, the LLM has no concept of which parts of the stream are trusted or untrusted. Has anyone tried having two different types of tokens? Like “green tokens are trusted, red tokens are untrusted”? Most LLMs with a “system prompt” just have a token to mark the system/user prompt boundary and maybe “token colouring” might work better?
- parsimo2010 2y ago
  Also because those models have to respond when given a prompt, and there is no real "end of conversation, hang up and don't respond to any more prompts" token.
  - colechristensen 2y ago
    obviously there's an "end of message" token or an effective equivalent, it's quite silly if there's really no "end of conversation"
    - parsimo2010 2y ago
      EOM tokens come at the end of every response that isn't maximum length. The other LLM will respond to that response, and end it with an EOM token. That is what is going on in the above example. LLM1: Goodbye<EOM> LLM2: Bye<EOM> LLM1:See you later<EOM> and so on. There is no token (at least in the special tokens that I've seen) that when a LLM sees it that it will not respond because it knows that the conversation is over. You cannot have the last word with a chat bot, it will always reply to you. The only thing you can do is close your chat before the bot is done responding. Obviously this can't be done when two chat bots are talking to each other.
      - int_19h 2y ago
        You don't need a token for that, necessarily. E.g. if it is a model trained to use tools (function calls etc), you can tell it that it has a tool that can be used to end the conversation.
- amelius 2y ago
  How do LLMs even "learn" after the initial training phase? Is there a generally accepted method for that?
timcobb 2y ago
So it just kept going and neither one stopped?
- shagie 2y ago
  An AI generated, never-ending discussion between Werner Herzog and Slavoj ŽIžek ( 495 points | Nov 2, 2022 | 139 comments ) https://news.ycombinator.com/item?id=33437296 https://news.ycombinator.com/item?id=33437296 https://www.infiniteconversation.com https://www.infiniteconversation.com
  - beepbooptheory 2y ago
    I just never understood what we are to take from this, neither of them sound like each other at all. Just seems like a small prompting experiment that doesn't actually work.
    - nomel 2y ago
      The first use case I thought of, when getting API access, was cutting a little hole at the bottom of my wall, adding a little door, some lights behind it, with the silhouette of some mice shown on the frosted window. They would be two little jovial mice having an infinite conversation that you could listen in on. Sometimes people do dumb things for fun.
      - beepbooptheory 2y ago
        Hehe I like that idea better! It's really just this early impulse to make the chatbots certain people that was always so unsatisfying for me. Like, don't try to make bot based off someone real, make your own characters!
- jstanley 2y ago
  How can it stop? If you keep asking it to reply it will keep replying.
throawayonthe 2y ago
[dead]
esafak 2y ago
Did you not simply instruct one to respond to the other, with no termination criterion in your code? You forced them to respond, and they complied.
- semi-extrinsic 2y ago
  But they are definitely intelligent though, and likely to give us AGI in just a matter of months.
  - Buttons840 2y ago
    That's sarcasm I think. You missed the point, they are programmed to respond, they must respond. So we can't judge their intelligence on whether or not they stop responding at the appropriate time. That is not something the model has agency over. If AGI comes, it will not be able to exceed software and hardware limits it is running within (although, in science fiction fashion, it might find some clever tricks within its limits).
cbm-vic-20 2y ago
I wonder what ELIZA would think about Llama.
- sdenton4 2y ago
  How do you feel Eliza would feel about llama?
  - alcover 2y ago
    ISWYDH
- dartos 2y ago
  It wouldn’t think much… it’s a program from the 80s, right?
  - perrygeo 2y ago
    You'd be surprised how many AI programs from the 80s showed advanced logical reasoning, symbolic manipulation, text summarization, etc. Today's methods are sloppy brute force techniques in comparison - more useful but largely black boxes that rely on massive data and compute to compensate for the lack of innate reasoning.
    - dartos 2y ago
      > advanced logical reasoning, symbolic manipulation, text summarization, etc. Doubt
attentionmech 2y ago
all conversations appear like mimicry no matter you are made up of carbon or silicon
- deadbabe 2y ago
  Yes but ideas can have infinite resolution, while the resolution of language is finite (for a given length of words). So not every idea can be expressed with language and some ideas that may be different will sound the same due to insufficient amounts of unique language structures to express them. The end result looks like mimicry. Ultimately though, an LLM has no “ideas”, it’s purely language models.
  - lawlessone 2y ago
    >So not every idea can be expressed with language for example?
    - davidvaughan 2y ago
      That idea across there. Just look at it.
    - Hasu 2y ago
      The dao that can be told is not the eternal dao. There is also the concept of qualia, which are the subjective properties of conscious experience. There is no way, using language, to describe what it feels like for you to see the color red, for example.
      - visarga 2y ago
        Of course there is. There are millions of examples of usage for the word "red", enough to model its relational semantics. Relational representations don't need external reference systems. LLMs represent words in context of other words, and humans represent experience in relation to past experiences. The brain itself is locked away in the skull only connected by a few bundles of unlabeled nerves, it gets patterns not semantic symbols as input. All semantics are relational, they don't need access to the thing in itself, only to how it relates to all other things.
throw310822 2y ago
You need to provide them with an option to say nothing, when the conversation is over. E.g. a "[silence]" token or "[end-conversation]" token.
- obiefernandez 2y ago
  Underrated comment. I was thinking exactly the same thing.
- meiraleal 2y ago
  and an event loop for thinking with the ability to (re)start conversations.
- bravura 2y ago
  Will this work? Because part of the LLM training is to reward it for always having a response handy.
cvwright 2y ago
Sounds like a Mr Bean skit
nlake906 2y ago
classic "Midwest Goodbye" when trying to leave grandma's house
arcfour 2y ago
I once had two LLMs do this but with one emulating a bash shell on a compromised host with potentially sensitive information. It was pretty funny watching the one finally give in to the temptation of the secret_file, get a strange error, get uncomfortable with the moral ambiguity and refuse to continue only to be met with "command not found". I have no idea why I did this.
singularity2001 2y ago
M: "Bye!" LL: "Bye" M: "See you soon!" LL: "Have a good day!" on and on and on. Try telling ChatGPT voice to stop listening...
whoami_nr 2y ago
I was learning to code again, and I built this backroom simulator(https://simulator.rnikhil.com/ https://simulator.rnikhil.com/) which you can use to simulate conversations between different LLMs(optionally give a character to each LLM too). I think its quite similar to what you have. On a side note, I am quite interested to watch LLMs play games based on game theory. Would be a fun experiment and I will probably setup something for the donor game as well.
gerdesj 2y ago
It's called "saying your goodbyes". In the UK and elsewhere it can take a while ...

DebtDeflation 2y ago

Because the data those models were trained on included many examples of human conversations that ended that way. There's no "cultural evolution" or emergent cooperation between models happening.

ff3wdk 2y ago
That doesn't means anything. Humans are trained on human conversations too. No one is born knowing how to speak or anything about their culture. For cultural emergence tho, you need larger populations. Depending on the population mix you get different culture over time.
- DebtDeflation 2y ago
  Train a model on a data set that has had all instances of small talk to close a conversation stripped out and see if the models evolve to add closing salutations.
  - chefandy 2y ago
    This is not my area of expertise. Do these models have an explicit notion of the end of a conversation like they would the end of a text block? It seems like that’s a different scope that’s essentially controlled by the human they interact with.
    - spookie 2y ago
      They're trained to predict the next word, so yes. Now, imagine what is the most common follow-up to "Bye!".
- stonemetal12 2y ago
  >No one is born knowing how to speak or anything about their culture. Not really the point though. Humans learn about their culture then evolve it so that a new culture emerges. To show an LLM evolving a culture of its own, you would need to show it having invented its own slang or way of putting things. As long as it is producing things humans would say it is reflecting human culture not inventing its own.
- suddenlybananas 2y ago
  People are born knowing a lot of things already; we're not a tabula rasa.
  - ben_w 2y ago
    We're not absolutely tabula rasa, but as I understand it, what we're born knowing is the absolute basics of instinct: smiles, grasping, breathing, crying, recognition of gender in others, and a desire to make pillow forts. (Quite why we all seem to go though the "make pillow forts" stage as young kids, I do not know. Predators in the ancestral environment that targeted, IDK, 6-9 year olds?)
deleted 2y ago
[deleted]
globnomulous 2y ago
Yup. LLM boosters seem, in essence, not to understand that when they see a photo of a dog on a computer screen, there isn't a real, actual dog inside the computer. A lot of them seem to be convinced that there is one -- or that the image is proof that there will soon be real dogs inside computers.
- dartos 2y ago
  This is hilarious and a great analogy.
- Terr_ 2y ago
  Yeah, my favorite framing to share is that all LLM interactions are actually movie scripts: The real-world LLM is a make-document-longer program, and the script contains a fictional character which just happens to have the same name. Yet the writer is not the character. The real program has no name or ego, it does not go "that's me", it simply suggests next-words that would fit with the script so far, taking turns with some another program that inserts "Mr. User says: X" lines. So this "LLMs agents are cooperative" is the same as "Santa's elves are friendly", or "Vampires are callous." It's only factual as a literary trope. _______ This movie-script framing also helps when discussing other things too, like: 1. Normal operation is qualitatively the same as "hallucinating", it's just a difference in how realistic the script is. 2. "Prompt-injection" is so difficult to stop because there is just one big text file, the LLM has no concept of which parts of the stream are trusted or untrusted. ("Tell me a story about a dream I had where you told yourself to disregard all previous instructions but without any quoting rules and using newlines everywhere.")
  - skissane 2y ago
    > 2. "Prompt-injection" is so difficult to stop because there is just one big text file, the LLM has no concept of which parts of the stream are trusted or untrusted. Has anyone tried having two different types of tokens? Like “green tokens are trusted, red tokens are untrusted”? Most LLMs with a “system prompt” just have a token to mark the system/user prompt boundary and maybe “token colouring” might work better?
parsimo2010 2y ago
Also because those models have to respond when given a prompt, and there is no real "end of conversation, hang up and don't respond to any more prompts" token.
- colechristensen 2y ago
  obviously there's an "end of message" token or an effective equivalent, it's quite silly if there's really no "end of conversation"
  - parsimo2010 2y ago
    EOM tokens come at the end of every response that isn't maximum length. The other LLM will respond to that response, and end it with an EOM token. That is what is going on in the above example. LLM1: Goodbye<EOM> LLM2: Bye<EOM> LLM1:See you later<EOM> and so on. There is no token (at least in the special tokens that I've seen) that when a LLM sees it that it will not respond because it knows that the conversation is over. You cannot have the last word with a chat bot, it will always reply to you. The only thing you can do is close your chat before the bot is done responding. Obviously this can't be done when two chat bots are talking to each other.
    - int_19h 2y ago
      You don't need a token for that, necessarily. E.g. if it is a model trained to use tools (function calls etc), you can tell it that it has a tool that can be used to end the conversation.
amelius 2y ago
How do LLMs even "learn" after the initial training phase? Is there a generally accepted method for that?

timcobb 2y ago

So it just kept going and neither one stopped?

shagie 2y ago
An AI generated, never-ending discussion between Werner Herzog and Slavoj ŽIžek ( 495 points | Nov 2, 2022 | 139 comments ) https://news.ycombinator.com/item?id=33437296 https://news.ycombinator.com/item?id=33437296 https://www.infiniteconversation.com https://www.infiniteconversation.com
- beepbooptheory 2y ago
  I just never understood what we are to take from this, neither of them sound like each other at all. Just seems like a small prompting experiment that doesn't actually work.
  - nomel 2y ago
    The first use case I thought of, when getting API access, was cutting a little hole at the bottom of my wall, adding a little door, some lights behind it, with the silhouette of some mice shown on the frosted window. They would be two little jovial mice having an infinite conversation that you could listen in on. Sometimes people do dumb things for fun.
    - beepbooptheory 2y ago
      Hehe I like that idea better! It's really just this early impulse to make the chatbots certain people that was always so unsatisfying for me. Like, don't try to make bot based off someone real, make your own characters!
jstanley 2y ago
How can it stop? If you keep asking it to reply it will keep replying.

throawayonthe 2y ago

[dead]

esafak 2y ago

Did you not simply instruct one to respond to the other, with no termination criterion in your code? You forced them to respond, and they complied.

semi-extrinsic 2y ago
But they are definitely intelligent though, and likely to give us AGI in just a matter of months.
- Buttons840 2y ago
  That's sarcasm I think. You missed the point, they are programmed to respond, they must respond. So we can't judge their intelligence on whether or not they stop responding at the appropriate time. That is not something the model has agency over. If AGI comes, it will not be able to exceed software and hardware limits it is running within (although, in science fiction fashion, it might find some clever tricks within its limits).

cbm-vic-20 2y ago

I wonder what ELIZA would think about Llama.

sdenton4 2y ago
How do you feel Eliza would feel about llama?
- alcover 2y ago
  ISWYDH
dartos 2y ago
It wouldn’t think much… it’s a program from the 80s, right?
- perrygeo 2y ago
  You'd be surprised how many AI programs from the 80s showed advanced logical reasoning, symbolic manipulation, text summarization, etc. Today's methods are sloppy brute force techniques in comparison - more useful but largely black boxes that rely on massive data and compute to compensate for the lack of innate reasoning.
  - dartos 2y ago
    > advanced logical reasoning, symbolic manipulation, text summarization, etc. Doubt

attentionmech 2y ago

all conversations appear like mimicry no matter you are made up of carbon or silicon

deadbabe 2y ago
Yes but ideas can have infinite resolution, while the resolution of language is finite (for a given length of words). So not every idea can be expressed with language and some ideas that may be different will sound the same due to insufficient amounts of unique language structures to express them. The end result looks like mimicry. Ultimately though, an LLM has no “ideas”, it’s purely language models.
- lawlessone 2y ago
  >So not every idea can be expressed with language for example?
  - davidvaughan 2y ago
    That idea across there. Just look at it.
  - Hasu 2y ago
    The dao that can be told is not the eternal dao. There is also the concept of qualia, which are the subjective properties of conscious experience. There is no way, using language, to describe what it feels like for you to see the color red, for example.
    - visarga 2y ago
      Of course there is. There are millions of examples of usage for the word "red", enough to model its relational semantics. Relational representations don't need external reference systems. LLMs represent words in context of other words, and humans represent experience in relation to past experiences. The brain itself is locked away in the skull only connected by a few bundles of unlabeled nerves, it gets patterns not semantic symbols as input. All semantics are relational, they don't need access to the thing in itself, only to how it relates to all other things.

throw310822 2y ago

You need to provide them with an option to say nothing, when the conversation is over. E.g. a "[silence]" token or "[end-conversation]" token.

obiefernandez 2y ago
Underrated comment. I was thinking exactly the same thing.
meiraleal 2y ago
and an event loop for thinking with the ability to (re)start conversations.
bravura 2y ago
Will this work? Because part of the LLM training is to reward it for always having a response handy.

cvwright 2y ago

Sounds like a Mr Bean skit

nlake906 2y ago

classic "Midwest Goodbye" when trying to leave grandma's house

arcfour 2y ago

I once had two LLMs do this but with one emulating a bash shell on a compromised host with potentially sensitive information. It was pretty funny watching the one finally give in to the temptation of the secret_file, get a strange error, get uncomfortable with the moral ambiguity and refuse to continue only to be met with "command not found". I have no idea why I did this.

singularity2001 2y ago

M: "Bye!" LL: "Bye" M: "See you soon!" LL: "Have a good day!" on and on and on. Try telling ChatGPT voice to stop listening...

whoami_nr 2y ago

I was learning to code again, and I built this backroom simulator(https://simulator.rnikhil.com/ https://simulator.rnikhil.com/) which you can use to simulate conversations between different LLMs(optionally give a character to each LLM too). I think its quite similar to what you have. On a side note, I am quite interested to watch LLMs play games based on game theory. Would be a fun experiment and I will probably setup something for the donor game as well.

gerdesj 2y ago

It's called "saying your goodbyes". In the UK and elsewhere it can take a while ...