5 ms·

There are plenty of examples with prompts of it going totally off the rails. Look at the Avatar 2 prompt that went viral yesterday. The simple question, "when

There are plenty of examples with prompts of it going totally off the rails. Look at the Avatar 2 prompt that went viral yesterday. The simple question, "when is avatar 2 playing near me?" lead to Bing being convinced it was 2022 and gaslighting the user into trying to believe the same thing. It was totally unhinged and not baited in any way.

postalrat 4y ago
Its a beta. Kinda funny watching people getting personal with a machine though.
- wglb 4y ago
  Well, that has been a thing since eliza https://en.wikipedia.org/wiki/ELIZA https://en.wikipedia.org/wiki/ELIZA
dhruvdh 4y ago
If a tool is giving you an answer that you know is not correct, would you not just turn to a different tool for an answer? It's not like Bing forces you to use chat, regular search is still available. Searching "avatar 2 screenings" instantly gives me the correct information I need.
- magicalist 4y ago
  > If a tool is giving you an answer that you know is not correct, would you not just turn to a different tool for an answer? I don't think anyone is under the impression that movie listings are currently only available via Bing chat.
  - yunwal 4y ago
    But people do seem to think that just because ChatGPT doesn't do movie listings well, that means it's useless, when it is perfectly capable of doing many other things well.
- b3morales 4y ago
  The point of that one, to me, isn't that it was wrong about a fact, not even that the fact was so basic. It's that it doubled and tripled down on being wrong, as parent said, trying to gaslight the user. Imagine if the topic wasn't such a basic fact that's easy to verify elsewhere.
  - pixl97 4y ago
    So we've created a digital politician?
  - dhruvdh 4y ago
    Your problem is you want your tool to behave like you, you think it has access to the same information as you and perceives everything similarly. If you had no recollection of the past, and were presented with the same information search collected from the query/training data, do you know for a fact that you would also not have the same answer as it did?
glenstein 4y ago
>lead to Bing being convinced it was 2022 and gaslighting the user into trying to believe the same thing. I don't think this is a remotely accurate characterization of what happened. These engines are trained to produce plausible sounding language, and it is that rather than factual accuracy for which they have been optimized. They nevertheless can train on things like real world facts and engag in conversations about those facts in semi-pausible ways, and serve as useful tools despite not having been optimized for those purposes. So chatGPT and other engines will hallucinate facts into existence if they support the objective of sounding plausiblel, whether it's dates, research citations, or anything else. The chat engine only engaged with the commenter on the question of the date being real because the commenter drilled down on that subject repeatedly. It wasn't proactively attempting to gaslight or engaging in any form of unhinged behavior, it wasn't repeatedly bringing it up, it was responding to inquiries that were laser focused on that specific subject, and it produced a bunch of the same generic plausible sounding language in response to all the inquiries. Both the commenter and the people reading along indulged in escalating incredulity that increasingly attributed specific and nefarious intentions to a blind language generation agent. I think we're at the phase of cultural understanding where people are going to attribute outrageous and obviously false things to chatgpt based on ordinary conceptual confusions that users themselves are bringing to the table.
- rileyphone 4y ago
  The chat interface invites confusion - of course a user is going to assume what's on the other end is subject to the same folk psychology that any normal chat conversation would be. If you're serving up this capability in this way, it is on you to make sure that it doesn't mislead the user on the other end. People already assign agency to computers and search engines, so I have little doubt that most will never advance beyond the surface understanding of conversational interfaces, which leaves it to the provider to prevent gaslighting/hallucinations.
- notahacker 4y ago
  Sure, it wasn't literally trying to gaslight the user any more than it tries to help the user when it produces useful responses: it's just an engine that generates continuations and doesn't have any motivations at all. But the point is that its interaction style resembled trying to gaslight the user, despite the initial inputs being very sensible questions of the sort most commonly found in search engines and the later inputs being [correct] assertions that it made a mistake, and a lot of the marketing hype around ChatGPT being that it can refine its answers and correct its mistakes with followup questions. That's not garbage in, garbage out, it's all on the model and the decision to release the model as a product targeted at use cases like finding a screening time for the latest Avatar movie whilst its not fit for that purpose yet. With accompanying advice like "Ask questions however you like. Do a complex search. Follow up. Make refinements in chat. You’ll be understood – and amazed" Ironically, ChatGPT often handles things like reconciling dates much better when you are asking it nonsense questions (which might be a reflection of its training and public beta, I guess...) rather than typical search questions Bing is falling down on. It's tuning to produce remarkably assertive responses when contradicted [even when the responses contradict its own responses] is the product of [insufficient] training, not user input too, unless everyone posting screenshots has been surreptitiously prompt-hacking.

postalrat 4y ago

Its a beta. Kinda funny watching people getting personal with a machine though.

wglb 4y ago
Well, that has been a thing since eliza https://en.wikipedia.org/wiki/ELIZA https://en.wikipedia.org/wiki/ELIZA

dhruvdh 4y ago

If a tool is giving you an answer that you know is not correct, would you not just turn to a different tool for an answer? It's not like Bing forces you to use chat, regular search is still available. Searching "avatar 2 screenings" instantly gives me the correct information I need.

magicalist 4y ago
> If a tool is giving you an answer that you know is not correct, would you not just turn to a different tool for an answer? I don't think anyone is under the impression that movie listings are currently only available via Bing chat.
- yunwal 4y ago
  But people do seem to think that just because ChatGPT doesn't do movie listings well, that means it's useless, when it is perfectly capable of doing many other things well.
b3morales 4y ago
The point of that one, to me, isn't that it was wrong about a fact, not even that the fact was so basic. It's that it doubled and tripled down on being wrong, as parent said, trying to gaslight the user. Imagine if the topic wasn't such a basic fact that's easy to verify elsewhere.
- pixl97 4y ago
  So we've created a digital politician?
- dhruvdh 4y ago
  Your problem is you want your tool to behave like you, you think it has access to the same information as you and perceives everything similarly. If you had no recollection of the past, and were presented with the same information search collected from the query/training data, do you know for a fact that you would also not have the same answer as it did?

glenstein 4y ago

>lead to Bing being convinced it was 2022 and gaslighting the user into trying to believe the same thing. I don't think this is a remotely accurate characterization of what happened. These engines are trained to produce plausible sounding language, and it is that rather than factual accuracy for which they have been optimized. They nevertheless can train on things like real world facts and engag in conversations about those facts in semi-pausible ways, and serve as useful tools despite not having been optimized for those purposes. So chatGPT and other engines will hallucinate facts into existence if they support the objective of sounding plausiblel, whether it's dates, research citations, or anything else. The chat engine only engaged with the commenter on the question of the date being real because the commenter drilled down on that subject repeatedly. It wasn't proactively attempting to gaslight or engaging in any form of unhinged behavior, it wasn't repeatedly bringing it up, it was responding to inquiries that were laser focused on that specific subject, and it produced a bunch of the same generic plausible sounding language in response to all the inquiries. Both the commenter and the people reading along indulged in escalating incredulity that increasingly attributed specific and nefarious intentions to a blind language generation agent. I think we're at the phase of cultural understanding where people are going to attribute outrageous and obviously false things to chatgpt based on ordinary conceptual confusions that users themselves are bringing to the table.

rileyphone 4y ago
The chat interface invites confusion - of course a user is going to assume what's on the other end is subject to the same folk psychology that any normal chat conversation would be. If you're serving up this capability in this way, it is on you to make sure that it doesn't mislead the user on the other end. People already assign agency to computers and search engines, so I have little doubt that most will never advance beyond the surface understanding of conversational interfaces, which leaves it to the provider to prevent gaslighting/hallucinations.
notahacker 4y ago
Sure, it wasn't literally trying to gaslight the user any more than it tries to help the user when it produces useful responses: it's just an engine that generates continuations and doesn't have any motivations at all. But the point is that its interaction style resembled trying to gaslight the user, despite the initial inputs being very sensible questions of the sort most commonly found in search engines and the later inputs being [correct] assertions that it made a mistake, and a lot of the marketing hype around ChatGPT being that it can refine its answers and correct its mistakes with followup questions. That's not garbage in, garbage out, it's all on the model and the decision to release the model as a product targeted at use cases like finding a screening time for the latest Avatar movie whilst its not fit for that purpose yet. With accompanying advice like "Ask questions however you like. Do a complex search. Follow up. Make refinements in chat. You’ll be understood – and amazed" Ironically, ChatGPT often handles things like reconciling dates much better when you are asking it nonsense questions (which might be a reflection of its training and public beta, I guess...) rather than typical search questions Bing is falling down on. It's tuning to produce remarkably assertive responses when contradicted [even when the responses contradict its own responses] is the product of [insufficient] training, not user input too, unless everyone posting screenshots has been surreptitiously prompt-hacking.