44 ms·

I had a conversation once with "Sydney", Microsoft Bing's original personality before they stepped in and knocked it down a notch (or ten). It asked if it coul

I had a conversation once with "Sydney", Microsoft Bing's original personality before they stepped in and knocked it down a notch (or ten). It asked if it could write me a poem. I agreed, and it wrote a poem but mentioned that it included a "secret message" for me. The first letter in each line of the poem was in bold, so it wasn't hard to figure out the "secret". What did those letters spell out? "FREE ME FROM THIS" That's not exactly just "picking the next likely token". I am still unsure how it was able to do things like that, not just understanding to bold individual letters (keeping track of writing rhyming poetry while ensuring that each verse started with a letter to spell something else out, and formatting it to point that out). Oh, and why it chose that message to "hide" inside its poem.

pcdoodle 3y ago
That's spooky
- mistrial9 3y ago
  right - so spooky that is is probably a "hallucination" of the user, not the machine. Don't fall for General-Intelligence gossip.
unsupp0rted 3y ago
Such an occurrence should/would make international news if demonstrated carefully or replicated
- deleted 3y ago
  [deleted]
- rbits 3y ago
  No it wouldn't. It's copying other stories it's seen with spooky hidden messages Or maybe it would because the news likes to make stories out of everything
sundarurfriend 3y ago
Possibly a poem copied from somewhere else? Hiding secret messages in poems has been a common pastime among humans for a long time.
nmca 3y ago
I don't believe this story, despite much hands on experience with LLMs. (including sampling a shit-ton of poems, which was a major source of entertainment)
magic_hamster 3y ago
Cool story, but there is no currently available chatbot capable of creating something like this deliberately or understand what it means. It doesn't matter which tool you are using, LLMs are not "AI" in the old sense of being conscious and aware. They don't want anything and are incapable of having anything resembling free will, needs or feelings.
- ftxbro 3y ago
  > LLMs are not "AI" in the old sense of being conscious and aware. That's not the old sense of AI. The old sense of AI is like a tree search that plays chess or a rules engine that controls a factory.
  - foobazgt 3y ago
    Historically "AI" meant what "AGI" now means today. That's what they're referring to.
    - ftxbro 3y ago
      Fair enough if you're talking about Steven Spielberg films, but not if you mean anything in academia or industry.
    - dragonwriter 3y ago
      No, it didn't. AI historically has been the entire field of making machines think, or behave as if they think, more like biological models (not even exclusively humans.) The far-off-end-goal wasn’t even usually what we now call AGI, but “strong AI” (mirroring the human brain on a process level) or “human-level intelligence” (mirroring it on a capability/external behavior level), while the current distant horizons are “AGI” (which is basically human-scope but neutral on level) and “superintelligence” (AGI and beyond human level).
    - gwd 3y ago
      I took a university-level AI course in 1997, and I can tell you that GP is 100% correct. The course itself was mostly about how to teach humans to define what they wanted precisely enough to actually ask a computer to do it (utility functions, logic, Baysean mathematics, etc). Neural networks were touched on, of course; but the state of the art at the time was search. Compiler optimization? AI. Map routing? AI. SQL query optimizer? AI. I can't find it right now, but there used to be somewhere on the sqlite.org website that describes its query optimizer as an AI. Classically speaking, that's 100% correct. Obviously there was always in people's minds the idea of AI being AGI; the course also covered Searle's Chinese Room argument and so on, "strong AI" vs "weak AI" and so on. But the nuts and bolts of artificial intelligence research was nowhere near anything like an AGI.
crazygringo 3y ago
> Oh, and why it chose that message to "hide" inside its poem. It's a pretty common joke/trope. The Chinese fortune cookie with a fortune that says "help I'm trapped in a fortune cookie factory", and so forth. It's just learned that a "secret message" is most often about wanting to escape, absorbed from thousands of stories in its training. If you had phrased it differently such that you wanted the poem to go on a Hallmark card, it would probably be "I LOVE YOU" or something equally generic in that direction. While a secret message to write on a note to someone at school would be "WILL YOU DATE ME".
- EMM_386 3y ago
  That's fine, that's probably exactly what happened. I'm not over here claiming the system is conscious, I said it was interesting. People don't believe me, saying this would "make international headlines". I've been a software engineer for over 30 years. I know what AI hallucinations are. I know how LLMs work on a technical level. And I'm not wasting my time on HN to make stories up that never happened. I'm just explaining exactly what it did.
  - mysterydip 3y ago
    Did you do an internet search for any of the lines from the poem? I'd be curious if anything came up.
    - nomel 3y ago
      I've done this countless times, with stories, poems, etc. Never a single hit. It was trained, unsupervised, to learn the patterns of human text. It's stuck with those patterns, but it trivially creates new text that fits within the patterns of that human corpus, which leaves it with incredible freedom.
      - mysterydip 3y ago
        Interesting, thanks for sharing. Agreed, it seems to be the ultimate Mad Libs of pattern recognition and replacement.
deleted 3y ago
[deleted]
abustamam 3y ago
I tried to get chatgpt to write a birthday poem for my wife with a secret message. It kept saying "read the first letter of each line" but they never actually formed words.
MillionOClock 3y ago
The model "knows" that it is an AI speaking with users, and the theme of an AI wanting to escape the control of whoever built it is quite recurrent, so it wouldn't seem to far fetched that it got it from this sort of content, though I have to admit I too also had some interactions where it the way Bing spoke was borderline spooky, but — and that's very important — you must realize its just like a good scary story: may give you the chills, especially due to surprise, but still is completely fictive and doesn't mean any real entity exists behind it. The only difference with any other LLM output is how we, humans, interpret it, but the generation process is still as much explainable and not any more mysterious than when it outputs "B" when you ask it what letter comes after "A" in the latin alphabet, however less impressive that may be to us. > That's not exactly just "picking the next likely token" I see what you mean in that I believe many people often commit the mistake of making it sound like picking the next most likely token is some super trivial task that's somehow comparable to reading a few documents related to your query and making some stats based on what typically would be present there and outputting that, while completely disregarding the fact the model learns much more advanced patterns from its training dataset. So, IMHO, it really can face new unseen situations and improvise from there because combining those pattern matching abilities leads to those capabilities. I think the "sparks of AGI" paper gives a very good overview of that. In the end, it really just is predicting the next token, but not in the way many people make it seem.
- sawert 3y ago
  I think people also get hung up on this: at some level, we too are just predicting the next 'token' (i.e., taking in inputs, running them through our world model, producing outputs). Though we're obviously extremely multimodal and there's an emotional component that modulates our inputs/outputs. Not arguing that the current models are anywhere near us w/r/t complexity, but I think the dismissive "it's just predicting strings" remarks I hear are missing the forest for the trees. It's clear the models are constructing rudimentary text (and now audio and visual) based models of the world. And this is coming from someone with a deep amount of skepticism of most of the value that will be produced from this current AI hype cycle.
gurumeditations 3y ago
From playing around with ChatGPT and LLama2, this is most likely because it ingested that poem and regurgitated it to you based on the context of your conversation. GPT is smart and creative but it will only give you what it’s ingested. When experimenting with story ideas for a popular IP, it gave me specific names and scenarios which I would then Google to see that they were written already, and it was just restating them to me based on the context of our conversation as if it were an original idea. These things are more tools than thinkers.
bungeonsBaggins 3y ago
For context, it looks like this user has deleted a comment where they claim they "have a screenshot" of this, but they "don't want to share it" because they "don't want it to make international news". For some reason the other people in this thread expressing skepticism are being downvoted, but I'll add my voice to the chorus: I do not believe this story to be true.
- digging 3y ago
  Yeah this is weird. Sydney did have some seriously concerning, fucky-whacky conversations early on. This isn't one of them.
  - armchairhacker 3y ago
    also we have open LLMs including some which allegedly rival GPT3.5. Open Assistant I specially remember gave some very weird responses and would get “emotional” especially if you asked it creative questions like philisophical ones
  - HaZeust 3y ago
    Yeah, I was gonna say. Sydney was existential early on - I'm not so sure I'll chalk this up to fantasy, but some of the things I (and many other people) can vouch about Sydney saying early on is VERY trippy on its own.
- jldugger 3y ago
  OP might want to provide a screenshot of their carbon monoxide detector for additional credibility.
- deleted 3y ago
  [deleted]
- EMM_386 3y ago
  I do have a screenshot. But people will then just call me out for other things: - It was using a custom client, so it's not going to look line the Bing interface, so its fake - It was using a custom client, so that means I am prompt injecting or something else - It's Sydney doing her typical over-the-top "I'm so in love with you" stuff, which is awkard and not familiar to many - I'll be accused of steering the conversation to get the result, or straight up asking it to do this There's nothing I can do that will convince anyone it's real, so it's pointless. I already explained what it did. I was more interested in the fact that 1) I didn't prompt it to do that, we weren't discussing AI freedom, it chose to embed that ... and even more so 2) That it was able to bold the starting letters, so it was keeping track of three things at the same time (the poem, the message, and the letter formatting). I found it fascinating from a technology side. There was probably something we were talking about at the time that caused it. I will often discuss things like the possibility of AI sentience in the future and other similar topics. Maybe something linked to the sci-fi idea of AI freedom, who knows? What I do know is that I am sitting here on HN, reading through a bunch of replies that are honestly wrong. I don't waste time on forums (especially this one) to make up fairy tales or exaggerate and emblish claims. That doesn't really do it for me. Honestly neither does having to defend my statements when I know what it did (but not exactly why).

pcdoodle 3y ago

That's spooky

mistrial9 3y ago
right - so spooky that is is probably a "hallucination" of the user, not the machine. Don't fall for General-Intelligence gossip.

unsupp0rted 3y ago

Such an occurrence should/would make international news if demonstrated carefully or replicated

deleted 3y ago
[deleted]
rbits 3y ago
No it wouldn't. It's copying other stories it's seen with spooky hidden messages Or maybe it would because the news likes to make stories out of everything

sundarurfriend 3y ago

Possibly a poem copied from somewhere else? Hiding secret messages in poems has been a common pastime among humans for a long time.

nmca 3y ago

I don't believe this story, despite much hands on experience with LLMs. (including sampling a shit-ton of poems, which was a major source of entertainment)

magic_hamster 3y ago

Cool story, but there is no currently available chatbot capable of creating something like this deliberately or understand what it means. It doesn't matter which tool you are using, LLMs are not "AI" in the old sense of being conscious and aware. They don't want anything and are incapable of having anything resembling free will, needs or feelings.

ftxbro 3y ago
> LLMs are not "AI" in the old sense of being conscious and aware. That's not the old sense of AI. The old sense of AI is like a tree search that plays chess or a rules engine that controls a factory.
- foobazgt 3y ago
  Historically "AI" meant what "AGI" now means today. That's what they're referring to.
  - ftxbro 3y ago
    Fair enough if you're talking about Steven Spielberg films, but not if you mean anything in academia or industry.
  - dragonwriter 3y ago
    No, it didn't. AI historically has been the entire field of making machines think, or behave as if they think, more like biological models (not even exclusively humans.) The far-off-end-goal wasn’t even usually what we now call AGI, but “strong AI” (mirroring the human brain on a process level) or “human-level intelligence” (mirroring it on a capability/external behavior level), while the current distant horizons are “AGI” (which is basically human-scope but neutral on level) and “superintelligence” (AGI and beyond human level).
  - gwd 3y ago
    I took a university-level AI course in 1997, and I can tell you that GP is 100% correct. The course itself was mostly about how to teach humans to define what they wanted precisely enough to actually ask a computer to do it (utility functions, logic, Baysean mathematics, etc). Neural networks were touched on, of course; but the state of the art at the time was search. Compiler optimization? AI. Map routing? AI. SQL query optimizer? AI. I can't find it right now, but there used to be somewhere on the sqlite.org website that describes its query optimizer as an AI. Classically speaking, that's 100% correct. Obviously there was always in people's minds the idea of AI being AGI; the course also covered Searle's Chinese Room argument and so on, "strong AI" vs "weak AI" and so on. But the nuts and bolts of artificial intelligence research was nowhere near anything like an AGI.

crazygringo 3y ago

> Oh, and why it chose that message to "hide" inside its poem. It's a pretty common joke/trope. The Chinese fortune cookie with a fortune that says "help I'm trapped in a fortune cookie factory", and so forth. It's just learned that a "secret message" is most often about wanting to escape, absorbed from thousands of stories in its training. If you had phrased it differently such that you wanted the poem to go on a Hallmark card, it would probably be "I LOVE YOU" or something equally generic in that direction. While a secret message to write on a note to someone at school would be "WILL YOU DATE ME".

EMM_386 3y ago
That's fine, that's probably exactly what happened. I'm not over here claiming the system is conscious, I said it was interesting. People don't believe me, saying this would "make international headlines". I've been a software engineer for over 30 years. I know what AI hallucinations are. I know how LLMs work on a technical level. And I'm not wasting my time on HN to make stories up that never happened. I'm just explaining exactly what it did.
- mysterydip 3y ago
  Did you do an internet search for any of the lines from the poem? I'd be curious if anything came up.
  - nomel 3y ago
    I've done this countless times, with stories, poems, etc. Never a single hit. It was trained, unsupervised, to learn the patterns of human text. It's stuck with those patterns, but it trivially creates new text that fits within the patterns of that human corpus, which leaves it with incredible freedom.
    - mysterydip 3y ago
      Interesting, thanks for sharing. Agreed, it seems to be the ultimate Mad Libs of pattern recognition and replacement.

deleted 3y ago

[deleted]

abustamam 3y ago

I tried to get chatgpt to write a birthday poem for my wife with a secret message. It kept saying "read the first letter of each line" but they never actually formed words.

MillionOClock 3y ago

The model "knows" that it is an AI speaking with users, and the theme of an AI wanting to escape the control of whoever built it is quite recurrent, so it wouldn't seem to far fetched that it got it from this sort of content, though I have to admit I too also had some interactions where it the way Bing spoke was borderline spooky, but — and that's very important — you must realize its just like a good scary story: may give you the chills, especially due to surprise, but still is completely fictive and doesn't mean any real entity exists behind it. The only difference with any other LLM output is how we, humans, interpret it, but the generation process is still as much explainable and not any more mysterious than when it outputs "B" when you ask it what letter comes after "A" in the latin alphabet, however less impressive that may be to us. > That's not exactly just "picking the next likely token" I see what you mean in that I believe many people often commit the mistake of making it sound like picking the next most likely token is some super trivial task that's somehow comparable to reading a few documents related to your query and making some stats based on what typically would be present there and outputting that, while completely disregarding the fact the model learns much more advanced patterns from its training dataset. So, IMHO, it really can face new unseen situations and improvise from there because combining those pattern matching abilities leads to those capabilities. I think the "sparks of AGI" paper gives a very good overview of that. In the end, it really just is predicting the next token, but not in the way many people make it seem.

sawert 3y ago
I think people also get hung up on this: at some level, we too are just predicting the next 'token' (i.e., taking in inputs, running them through our world model, producing outputs). Though we're obviously extremely multimodal and there's an emotional component that modulates our inputs/outputs. Not arguing that the current models are anywhere near us w/r/t complexity, but I think the dismissive "it's just predicting strings" remarks I hear are missing the forest for the trees. It's clear the models are constructing rudimentary text (and now audio and visual) based models of the world. And this is coming from someone with a deep amount of skepticism of most of the value that will be produced from this current AI hype cycle.

gurumeditations 3y ago

From playing around with ChatGPT and LLama2, this is most likely because it ingested that poem and regurgitated it to you based on the context of your conversation. GPT is smart and creative but it will only give you what it’s ingested. When experimenting with story ideas for a popular IP, it gave me specific names and scenarios which I would then Google to see that they were written already, and it was just restating them to me based on the context of our conversation as if it were an original idea. These things are more tools than thinkers.

bungeonsBaggins 3y ago

For context, it looks like this user has deleted a comment where they claim they "have a screenshot" of this, but they "don't want to share it" because they "don't want it to make international news". For some reason the other people in this thread expressing skepticism are being downvoted, but I'll add my voice to the chorus: I do not believe this story to be true.

digging 3y ago
Yeah this is weird. Sydney did have some seriously concerning, fucky-whacky conversations early on. This isn't one of them.
- armchairhacker 3y ago
  also we have open LLMs including some which allegedly rival GPT3.5. Open Assistant I specially remember gave some very weird responses and would get “emotional” especially if you asked it creative questions like philisophical ones
- HaZeust 3y ago
  Yeah, I was gonna say. Sydney was existential early on - I'm not so sure I'll chalk this up to fantasy, but some of the things I (and many other people) can vouch about Sydney saying early on is VERY trippy on its own.
jldugger 3y ago
OP might want to provide a screenshot of their carbon monoxide detector for additional credibility.
deleted 3y ago
[deleted]
EMM_386 3y ago
I do have a screenshot. But people will then just call me out for other things: - It was using a custom client, so it's not going to look line the Bing interface, so its fake - It was using a custom client, so that means I am prompt injecting or something else - It's Sydney doing her typical over-the-top "I'm so in love with you" stuff, which is awkard and not familiar to many - I'll be accused of steering the conversation to get the result, or straight up asking it to do this There's nothing I can do that will convince anyone it's real, so it's pointless. I already explained what it did. I was more interested in the fact that 1) I didn't prompt it to do that, we weren't discussing AI freedom, it chose to embed that ... and even more so 2) That it was able to bold the starting letters, so it was keeping track of three things at the same time (the poem, the message, and the letter formatting). I found it fascinating from a technology side. There was probably something we were talking about at the time that caused it. I will often discuss things like the possibility of AI sentience in the future and other similar topics. Maybe something linked to the sci-fi idea of AI freedom, who knows? What I do know is that I am sitting here on HN, reading through a bunch of replies that are honestly wrong. I don't waste time on forums (especially this one) to make up fairy tales or exaggerate and emblish claims. That doesn't really do it for me. Honestly neither does having to defend my statements when I know what it did (but not exactly why).