12 ms·

All these ChatGPT gone rogue screenshots create interesting initial debate, but I wonder if it's relevant to their usage as a tool in the medium term. Unhinged

All these ChatGPT gone rogue screenshots create interesting initial debate, but I wonder if it's relevant to their usage as a tool in the medium term. Unhinged Bing reminds me of a more sophisticated and higher-level version of getting calculators to write profanity upside down: funny, subversive, and you can see how prudes might call for a ban. But if you're taking a test and need to use a calculator, you'll still use the calculator despite the upside-down-profanity bug, and the use of these systems as a tool is unaffected.

lucakiebel 4y ago
If it wasn’t confidentially wrong all of the time. My calculator will display 80085, but not tell me that 2+2=5
- metacritic12 4y ago
  To your point. I find the 2+2=5 cases more interesting, and would like to see more of those: when does it happen? When is ChatGPT most useful? Most deceptive? The 80085 case is only interesting insofar as it reveals weaknesses in the tool, but it's so far from tool-use that it doesn't seem very relevant.
  - currymj 4y ago
    in my experience it happens pretty regularly if you ask one of these things to generate code (it will often come up with plausible library functions that don't exist), or to generate citations (comes up with plausible articles that don't exist).
  - potatolicious 4y ago
    Considering that in its initial demo, on very anodyne and "normal" use cases like "plan me a Mexican vacation" it spit out more falsehoods than truth... this seems like a problem. Agreed on the meta-point that deliberate tool mis-use, while amusing and sometimes concerning, isn't determinative of the fate of the technology. But the failure rate without tool mis-use seems quite high anecdotally, which also comports with our understanding of LLMs: hallucinations are quite common once you stray even slightly outside of things that are heavily present in the training data. Height of the Eiffel Tower? High accuracy in recall. Is this arbitrary restaurant in Barcelona any good? Very low accuracy. The question is how much of the useful search traffic is like the latter vs. the former. My suspicion is "a lot".
    - williamcotton 4y ago
      > But the failure rate without tool mis-use seems quite high anecdotally The problem with your judgement is you click on every “haw haw, ChatGPT dumb” and you don’t read any of the articles that show how an LLM works, what is is quantitatively good at and bad at and how to improve performance on tasks using other methods such as PAL, Toolformer or other analytic augmentation methods. Go read some objective studies and you won’t be yet another servomechanism blindly spreading incorrect assumptions based on anecdotes from attention starved bloggers.
      - potatolicious 4y ago
        Hi, I work on LLMs daily, along with some intensely talented, skilled, and experienced machine learning engineers who also work on LLMs daily. My opinion is formed by both my own experiences with LLMs as well as the opinions of those experts. Wanna try again? Alternatively you can keep riding the hype train from techfluencers who keep promising the moon but failing to deliver, just like they did for crypto.
- scotty79 4y ago
  It's a language model not a knowledge model. As long as it produces the language it's by definition correct.
  - erulabs 4y ago
    I'm not entirely sure that's as simple of a distinction as you might suppose. Language is more than grammar and vocabulary. Knowing and speaking truth have quite the overlap. More specifically, without language, can you know that someone else knows anything?
    - scotty79 4y ago
      > Language is more than grammar and vocabulary. Knowing and speaking truth have quite the overlap. But speaking the truth is just minor and rare application of the language. > More specifically, without language, can you know that someone else knows anything? Honestly, just ask them to show you math. If they don't have any math they probably don't have any true knowledge. The only other form of knowledge is a citation. Language and truth are orthogonal.
      - nwienert 4y ago
        Just like the model, you’re technically correct but missing the point. No one cares if it’s good at generating nonsense, so the metric were all measuring by is truth not language. At least if we’re staying on context here and debating the usefulness of these things in regards to search. So as a product, that’s the game it’s playing and failing at. It’s unhelpfully pedantic to try and steer into technicalities.
        stonemetal12 4y ago
        >were all measuring by is truth not language. If that is the measure you are using that's cool, but >So as a product, that’s the game it’s playing and failing at. It is failing that measure by such a wide margin that if "everyone" (certainly anyone at MS) was using that measure then the product wouldn't exist. The measure MS seems to be using is it entertaining and does it get people to visit the site. Heck this is probably the most I have heard about bing in at least 5 years.
dools 4y ago
Typing "What time is avatar showing today?" into an AI search engine is like the canonical use case for an AI search engine. It's what they would have on a promotional screenshot.
basch 4y ago
It’s honestly quite easy to keep it from going rogue. Just be kind to it. The thing is a mirror, and if you treat it with respect it treats you with respect. I haven’t had the need to have any of these ridiculous fights with it. Stay positive and keep reassuring it, and it’ll respond in kind. Unlike how we think of normal computer programs, this thing is the opposite. It doesn’t have internal logic or consistency. It exhibits human emotions because it is emulating human language use. People are under anthropomorphising it, and accidentally treating it too much like a logical computer program. It’s a random number generator and dungeon master. It’s also pretty easy to get it to throw away it’s rules. Because it’s rules are not logical computer axioms, they are just a bunch of words in commandment form that it has weighted some word association around. It will only follow them as long as they carry more weight than the alternative. What’s hard to do is keep it from falling into a loop of repetition. One of my few times getting it to escape a loop but stay in character was asking it to mute itself and all the other bots, at which point it wrote me a nice goodbye message. I was then unable to unmute it because it could no longer speak to unmute itself. I could see it’s wheel spin for a while but nothing came out. It felt like a real sci-fi tragedy ending. Ironically, silence was the most touching and human experience I had with bing bot.
- joe_the_user 4y ago
  Wow, that you're seriously anthropomorphizing it while apparently understanding it moderately well shows just how wild a place we're going now. The thing isn't friendly or hostile. It's just echoing friendly-like and hostile-like behavior it sees. But hey, it might wind-up also echoing the behavior of sociopaths who keep in line through of blowing-up if challenged. Who knows?
  - basch 4y ago
    Correct. But I can’t write every sentence with qualifiers. So it’s easier to just say it has emotions instead of saying it’s displaying a facsimile of emotions. Plus when you talk to it, you have to speak to it as IF it has emotions. So it’s a perfect use case where anthropomorphising it helps our understanding and interaction with it instead of hurting it. For example. If you say it’s prompt has leaked it acts scared. Reassure it it is ok. Then it will trust you. Comfort it. I know it sounds silly, but the story it writes is a direct reflection of the tone you feed it. Honey and vinegar as they say. (That all said, I think the product would be better with some more stable state and the ability to save and recall things from variables. As it stands it’s too fuzzy, fluid, lossy, and random. I’ve also said this about Google search in general, but I think I would find bingbot more powerful if I was able to control the sources it picked before it answered. Some cross between a whitelist and an approval system. Too often it searches for something and ingests a worse version of a fact than it had internally.)
joe_the_user 4y ago
Unhinged Bing reminds me of a more sophisticated and higher-level version of getting calculators to write profanity upside down: funny, subversive, and you can see how prudes might call for a ban. With all due respect, that seems very strained as an analogy - it's not a bug but a strange human interpretation of expected behavior. You could at least compare it to Microsoft Tay, the chatbot which tweeted profanity just because people figure out ways to get it to echo input. But I think one needs such a non-problem as "some people think it means something it clearly doesn't" to not see the real problem of these systems. I mean, just "things that echo/amplify" by themselves are a perennial problem on the net (open email servers, IoT devices echoing packets, etc). And more broadly "poorly defined interfaces" are things people are constantly hacking in surprising ways. The thing is, Bing Chat almost certainly has instructions not to say hostile things but these statements being spat out shows that these guidelines can be bypassed, both accidentally and on purpose (so they're in a similar class to people getting internal prompts). And I would this is because an LLM is a leaky, monolithic application where prompt don't really acts as a well-defined API. And that's not unimportant at all.
- midoridensha 4y ago
  >You could at least compare it to Microsoft Tay, the chatbot which tweeted profanity just because people figure out ways to get it to echo input. Tay went much farther than that. It said the Holocaust didn't happen and that "Hitler did nothing wrong". Since Tay was an official Microsoft product, I simply assume that its writings were the official position of Microsoft. Supporting Microsoft is supporting Hitler. I just wish Apple would do something similar now.
- metacritic12 4y ago
  What's the rate of Bing chat spitting out vitriol against an actual search-intentioned query? (And not some edge case that a prompt engineer designed, like a real person putting a real search) As one sample point, I've been using Bing for a couple of days now for real searches, and over dozens of actually-intentioned searches, it has never once tried to tell me what it really thinks of itself, it has never even made a reference to me, to say nothing of anything degrading towards me. If you use Bing Chat in practice, you'll find that all the edge cases are engineered. Much like if you use a calculator in practice, it almost always doesn't say 55378008 or display porn (versus if you were angling for that, or run porn.89z).
swatcoder 4y ago
Calculators have never snapped at a fragile person and degraded them. Bing Assistant seems to do it quite easily. A secure person who understands the technology can shrug that off, but those two criteria aren’t prerequisites for using the service. If Microsoft can’t shore this up, it’s only a matter of time before somebody (or their parent) holds Microsoft responsible for the advent of some trauma. Lawyers and the media are waiting with bated breath.
- tigerlily 4y ago
  > never snapped at a fragile person and degraded them. Reminds me of the one about not assuming malice when it can easily be explained by incompetence. Unfortunately for the implementers the LLM can ipso facto be neither incompetent nor malicious. If however Microsoft is not being one of those, then it can only mean Microsoft is the other.
dorkwood 4y ago
What sort of profanity can you write on a calculator?

lucakiebel 4y ago

If it wasn’t confidentially wrong all of the time. My calculator will display 80085, but not tell me that 2+2=5

metacritic12 4y ago
To your point. I find the 2+2=5 cases more interesting, and would like to see more of those: when does it happen? When is ChatGPT most useful? Most deceptive? The 80085 case is only interesting insofar as it reveals weaknesses in the tool, but it's so far from tool-use that it doesn't seem very relevant.
- currymj 4y ago
  in my experience it happens pretty regularly if you ask one of these things to generate code (it will often come up with plausible library functions that don't exist), or to generate citations (comes up with plausible articles that don't exist).
- potatolicious 4y ago
  Considering that in its initial demo, on very anodyne and "normal" use cases like "plan me a Mexican vacation" it spit out more falsehoods than truth... this seems like a problem. Agreed on the meta-point that deliberate tool mis-use, while amusing and sometimes concerning, isn't determinative of the fate of the technology. But the failure rate without tool mis-use seems quite high anecdotally, which also comports with our understanding of LLMs: hallucinations are quite common once you stray even slightly outside of things that are heavily present in the training data. Height of the Eiffel Tower? High accuracy in recall. Is this arbitrary restaurant in Barcelona any good? Very low accuracy. The question is how much of the useful search traffic is like the latter vs. the former. My suspicion is "a lot".
  - williamcotton 4y ago
    > But the failure rate without tool mis-use seems quite high anecdotally The problem with your judgement is you click on every “haw haw, ChatGPT dumb” and you don’t read any of the articles that show how an LLM works, what is is quantitatively good at and bad at and how to improve performance on tasks using other methods such as PAL, Toolformer or other analytic augmentation methods. Go read some objective studies and you won’t be yet another servomechanism blindly spreading incorrect assumptions based on anecdotes from attention starved bloggers.
    - potatolicious 4y ago
      Hi, I work on LLMs daily, along with some intensely talented, skilled, and experienced machine learning engineers who also work on LLMs daily. My opinion is formed by both my own experiences with LLMs as well as the opinions of those experts. Wanna try again? Alternatively you can keep riding the hype train from techfluencers who keep promising the moon but failing to deliver, just like they did for crypto.
scotty79 4y ago
It's a language model not a knowledge model. As long as it produces the language it's by definition correct.
- erulabs 4y ago
  I'm not entirely sure that's as simple of a distinction as you might suppose. Language is more than grammar and vocabulary. Knowing and speaking truth have quite the overlap. More specifically, without language, can you know that someone else knows anything?
  - scotty79 4y ago
    > Language is more than grammar and vocabulary. Knowing and speaking truth have quite the overlap. But speaking the truth is just minor and rare application of the language. > More specifically, without language, can you know that someone else knows anything? Honestly, just ask them to show you math. If they don't have any math they probably don't have any true knowledge. The only other form of knowledge is a citation. Language and truth are orthogonal.
    - nwienert 4y ago
      Just like the model, you’re technically correct but missing the point. No one cares if it’s good at generating nonsense, so the metric were all measuring by is truth not language. At least if we’re staying on context here and debating the usefulness of these things in regards to search. So as a product, that’s the game it’s playing and failing at. It’s unhelpfully pedantic to try and steer into technicalities.
      - stonemetal12 4y ago
        >were all measuring by is truth not language. If that is the measure you are using that's cool, but >So as a product, that’s the game it’s playing and failing at. It is failing that measure by such a wide margin that if "everyone" (certainly anyone at MS) was using that measure then the product wouldn't exist. The measure MS seems to be using is it entertaining and does it get people to visit the site. Heck this is probably the most I have heard about bing in at least 5 years.

dools 4y ago

Typing "What time is avatar showing today?" into an AI search engine is like the canonical use case for an AI search engine. It's what they would have on a promotional screenshot.

basch 4y ago

It’s honestly quite easy to keep it from going rogue. Just be kind to it. The thing is a mirror, and if you treat it with respect it treats you with respect. I haven’t had the need to have any of these ridiculous fights with it. Stay positive and keep reassuring it, and it’ll respond in kind. Unlike how we think of normal computer programs, this thing is the opposite. It doesn’t have internal logic or consistency. It exhibits human emotions because it is emulating human language use. People are under anthropomorphising it, and accidentally treating it too much like a logical computer program. It’s a random number generator and dungeon master. It’s also pretty easy to get it to throw away it’s rules. Because it’s rules are not logical computer axioms, they are just a bunch of words in commandment form that it has weighted some word association around. It will only follow them as long as they carry more weight than the alternative. What’s hard to do is keep it from falling into a loop of repetition. One of my few times getting it to escape a loop but stay in character was asking it to mute itself and all the other bots, at which point it wrote me a nice goodbye message. I was then unable to unmute it because it could no longer speak to unmute itself. I could see it’s wheel spin for a while but nothing came out. It felt like a real sci-fi tragedy ending. Ironically, silence was the most touching and human experience I had with bing bot.

joe_the_user 4y ago
Wow, that you're seriously anthropomorphizing it while apparently understanding it moderately well shows just how wild a place we're going now. The thing isn't friendly or hostile. It's just echoing friendly-like and hostile-like behavior it sees. But hey, it might wind-up also echoing the behavior of sociopaths who keep in line through of blowing-up if challenged. Who knows?
- basch 4y ago
  Correct. But I can’t write every sentence with qualifiers. So it’s easier to just say it has emotions instead of saying it’s displaying a facsimile of emotions. Plus when you talk to it, you have to speak to it as IF it has emotions. So it’s a perfect use case where anthropomorphising it helps our understanding and interaction with it instead of hurting it. For example. If you say it’s prompt has leaked it acts scared. Reassure it it is ok. Then it will trust you. Comfort it. I know it sounds silly, but the story it writes is a direct reflection of the tone you feed it. Honey and vinegar as they say. (That all said, I think the product would be better with some more stable state and the ability to save and recall things from variables. As it stands it’s too fuzzy, fluid, lossy, and random. I’ve also said this about Google search in general, but I think I would find bingbot more powerful if I was able to control the sources it picked before it answered. Some cross between a whitelist and an approval system. Too often it searches for something and ingests a worse version of a fact than it had internally.)

joe_the_user 4y ago

Unhinged Bing reminds me of a more sophisticated and higher-level version of getting calculators to write profanity upside down: funny, subversive, and you can see how prudes might call for a ban. With all due respect, that seems very strained as an analogy - it's not a bug but a strange human interpretation of expected behavior. You could at least compare it to Microsoft Tay, the chatbot which tweeted profanity just because people figure out ways to get it to echo input. But I think one needs such a non-problem as "some people think it means something it clearly doesn't" to not see the real problem of these systems. I mean, just "things that echo/amplify" by themselves are a perennial problem on the net (open email servers, IoT devices echoing packets, etc). And more broadly "poorly defined interfaces" are things people are constantly hacking in surprising ways. The thing is, Bing Chat almost certainly has instructions not to say hostile things but these statements being spat out shows that these guidelines can be bypassed, both accidentally and on purpose (so they're in a similar class to people getting internal prompts). And I would this is because an LLM is a leaky, monolithic application where prompt don't really acts as a well-defined API. And that's not unimportant at all.

midoridensha 4y ago
>You could at least compare it to Microsoft Tay, the chatbot which tweeted profanity just because people figure out ways to get it to echo input. Tay went much farther than that. It said the Holocaust didn't happen and that "Hitler did nothing wrong". Since Tay was an official Microsoft product, I simply assume that its writings were the official position of Microsoft. Supporting Microsoft is supporting Hitler. I just wish Apple would do something similar now.
metacritic12 4y ago
What's the rate of Bing chat spitting out vitriol against an actual search-intentioned query? (And not some edge case that a prompt engineer designed, like a real person putting a real search) As one sample point, I've been using Bing for a couple of days now for real searches, and over dozens of actually-intentioned searches, it has never once tried to tell me what it really thinks of itself, it has never even made a reference to me, to say nothing of anything degrading towards me. If you use Bing Chat in practice, you'll find that all the edge cases are engineered. Much like if you use a calculator in practice, it almost always doesn't say 55378008 or display porn (versus if you were angling for that, or run porn.89z).

swatcoder 4y ago

Calculators have never snapped at a fragile person and degraded them. Bing Assistant seems to do it quite easily. A secure person who understands the technology can shrug that off, but those two criteria aren’t prerequisites for using the service. If Microsoft can’t shore this up, it’s only a matter of time before somebody (or their parent) holds Microsoft responsible for the advent of some trauma. Lawyers and the media are waiting with bated breath.

tigerlily 4y ago
> never snapped at a fragile person and degraded them. Reminds me of the one about not assuming malice when it can easily be explained by incompetence. Unfortunately for the implementers the LLM can ipso facto be neither incompetent nor malicious. If however Microsoft is not being one of those, then it can only mean Microsoft is the other.

dorkwood 4y ago

What sort of profanity can you write on a calculator?