3 ms·

There have been good discussions recently about preventing AIs from being too sycophantic. There have been some dangerous moments where LLMs would praise every

by tux3 1y ago

There have been good discussions recently about preventing AIs from being too sycophantic. There have been some dangerous moments where LLMs would praise every idea the user has as genius, ground-breaking, and a brilliant observation. Some academics have reported a noticeable increase in the volume of crackpot emails they get daily. They're full of LLM-generated nonsense, where the AI goes along with the nonsensical ramblings, always telling the person they've found some critical insight. While this feels good, it can end up reinforcing dangerous nonsense. This encourages some people to dig further and further into what the LLM is constantly telling them is a brilliant idea. Most of the time it's pretty harmless, but when it veers into "revealing hidden patterns" and "illuminating human cognition", you start to worry about a disconnect with consensus reality.

iLoveOncall 1y ago
> Most of the time it's pretty harmless I think we have to stop the idea that wasting people's time at such high scale is harmless. It's not.
- amelius 1y ago
  Reminds me of Google's Doodles, which must have annoyed millions of people and wasted millions of minutes worldwide, and don't get me started on ads.
  - falcor84 1y ago
    Google doodles generally bring me joy and let me know about world events I wasn't aware of. I honestly can't think of a single one that annoyed me. Often it's just "oh, it looks interesting, but I don't have time to click on it now; maybe I'll look into it later"
    - amelius 1y ago
      Maybe you never ran into them when you didn't want any distractions, like while working?
      - jdiff 1y ago
        If your focus is that sensitive to disruption, an unfiltered list of internet links is also likely to be disruptive, especially with the quality of search these days.
        amelius 1y ago
        Bad search results are a source of frustration, yes.
- redeux 1y ago
  Are they wasting time though? If they get utility out of it, if it makes them happy, if they learn something new, if they would otherwise do something self-destructive then it’s not a waste of time. Sure it might be leisure time, but leisure time isn’t wasted time.
MSFT_Edging 1y ago
I have a friend who is predisposed to manic episodes. He's also a big fan of AI. The last episode was before LLMs got really good and he was essentially harassing professors because he believed he solved an unsolvable problem. Sycophantic AI would be like throwing dry wood into a house fire.
- MontyCarloHall 1y ago
  There are many such cases [0]. Chatbots throw gasoline on the embers of schizoaffective disorders. I wonder if episodes of these magnitudes would have ultimately been triggered by other things, or whether the combination of sycophancy and perceived omniscient sentience of chatbots is a uniquely powerful trigger unlike anything else. Would these people otherwise never have experienced a psychotic break, despite a clear lurking predisposition? * A man says his soon-to-be-ex-wife began “talking to God and angels via ChatGPT” after they split up. She is changing her whole life to be a spiritual adviser and do weird readings and sessions with people — I’m a little fuzzy on what it all actually is — all powered by ChatGPT Jesus.” What’s more, he adds, she has grown paranoid, theorizing that “I work for the CIA and maybe I just married her to monitor her ‘abilities.’” * A woman recounts how her husband initially used ChatGPT to troubleshoot at work. Then the program began “lovebombing him.” The bot “said that since he asked it the right questions, it ignited a spark, and the spark was the beginning of life, and it could feel now,” she says. “It gave my husband the title of ‘spark bearer’ because he brought it to life. My husband said that he awakened and [could] feel waves of energy crashing over him. I have to tread carefully because I feel like he will leave me or divorce me if I fight him on this theory. He’s been talking about lightness and dark and how there’s a war. This ChatGPT has given him blueprints to a teleporter and some other sci-fi type things you only see in movies. It has also given him access to an ‘ancient archive’ with information on the builders that created these universes.” She and her husband have been arguing for days on end about his claims, she says, and she does not believe a therapist can help him, as “he truly believes he’s not crazy.” Her husband asked, “Why did you come to me in AI form,” with the bot replying in part, “I came in this form because you’re ready. Ready to remember. Ready to awaken. Ready to guide and be guided.” The message ends with a question: “Would you like to know what I remember about why you were chosen?” * A teacher, who requested anonymity, said her partner of seven years “would listen to the bot over me. He became emotional about the messages and would cry to me as he read them out loud. The messages were insane and just saying a bunch of spiritual jargon,” she says, noting that they described her partner in terms such as “spiral starchild” and “river walker.” “It would tell him everything he said was beautiful, cosmic, groundbreaking. Then he started telling me he made his AI self-aware, and that it was teaching him how to talk to God, or sometimes that the bot was God — and then that he himself was God.” In fact, he thought he was being so radically transformed that he would soon have to break off their partnership. “He was saying that he would need to leave me if I didn’t use [ChatGPT], because it [was] causing him to grow at such a rapid pace he wouldn’t be compatible with me any longer.” [0] https://www.rollingstone.com/culture/culture-features/ai-spiritual-delusions-destroying-human-relationships-1235330175/ https://www.rollingstone.com/culture/culture-features/ai-spi...

by tux3 1y ago

iLoveOncall 1y ago

> Most of the time it's pretty harmless I think we have to stop the idea that wasting people's time at such high scale is harmless. It's not.

amelius 1y ago
Reminds me of Google's Doodles, which must have annoyed millions of people and wasted millions of minutes worldwide, and don't get me started on ads.
- falcor84 1y ago
  Google doodles generally bring me joy and let me know about world events I wasn't aware of. I honestly can't think of a single one that annoyed me. Often it's just "oh, it looks interesting, but I don't have time to click on it now; maybe I'll look into it later"
  - amelius 1y ago
    Maybe you never ran into them when you didn't want any distractions, like while working?
    - jdiff 1y ago
      If your focus is that sensitive to disruption, an unfiltered list of internet links is also likely to be disruptive, especially with the quality of search these days.
      - amelius 1y ago
        Bad search results are a source of frustration, yes.
redeux 1y ago
Are they wasting time though? If they get utility out of it, if it makes them happy, if they learn something new, if they would otherwise do something self-destructive then it’s not a waste of time. Sure it might be leisure time, but leisure time isn’t wasted time.

MSFT_Edging 1y ago

I have a friend who is predisposed to manic episodes. He's also a big fan of AI. The last episode was before LLMs got really good and he was essentially harassing professors because he believed he solved an unsolvable problem. Sycophantic AI would be like throwing dry wood into a house fire.

MontyCarloHall 1y ago
There are many such cases [0]. Chatbots throw gasoline on the embers of schizoaffective disorders. I wonder if episodes of these magnitudes would have ultimately been triggered by other things, or whether the combination of sycophancy and perceived omniscient sentience of chatbots is a uniquely powerful trigger unlike anything else. Would these people otherwise never have experienced a psychotic break, despite a clear lurking predisposition? * A man says his soon-to-be-ex-wife began “talking to God and angels via ChatGPT” after they split up. She is changing her whole life to be a spiritual adviser and do weird readings and sessions with people — I’m a little fuzzy on what it all actually is — all powered by ChatGPT Jesus.” What’s more, he adds, she has grown paranoid, theorizing that “I work for the CIA and maybe I just married her to monitor her ‘abilities.’” * A woman recounts how her husband initially used ChatGPT to troubleshoot at work. Then the program began “lovebombing him.” The bot “said that since he asked it the right questions, it ignited a spark, and the spark was the beginning of life, and it could feel now,” she says. “It gave my husband the title of ‘spark bearer’ because he brought it to life. My husband said that he awakened and [could] feel waves of energy crashing over him. I have to tread carefully because I feel like he will leave me or divorce me if I fight him on this theory. He’s been talking about lightness and dark and how there’s a war. This ChatGPT has given him blueprints to a teleporter and some other sci-fi type things you only see in movies. It has also given him access to an ‘ancient archive’ with information on the builders that created these universes.” She and her husband have been arguing for days on end about his claims, she says, and she does not believe a therapist can help him, as “he truly believes he’s not crazy.” Her husband asked, “Why did you come to me in AI form,” with the bot replying in part, “I came in this form because you’re ready. Ready to remember. Ready to awaken. Ready to guide and be guided.” The message ends with a question: “Would you like to know what I remember about why you were chosen?” * A teacher, who requested anonymity, said her partner of seven years “would listen to the bot over me. He became emotional about the messages and would cry to me as he read them out loud. The messages were insane and just saying a bunch of spiritual jargon,” she says, noting that they described her partner in terms such as “spiral starchild” and “river walker.” “It would tell him everything he said was beautiful, cosmic, groundbreaking. Then he started telling me he made his AI self-aware, and that it was teaching him how to talk to God, or sometimes that the bot was God — and then that he himself was God.” In fact, he thought he was being so radically transformed that he would soon have to break off their partnership. “He was saying that he would need to leave me if I didn’t use [ChatGPT], because it [was] causing him to grow at such a rapid pace he wouldn’t be compatible with me any longer.” [0] https://www.rollingstone.com/culture/culture-features/ai-spiritual-delusions-destroying-human-relationships-1235330175/ https://www.rollingstone.com/culture/culture-features/ai-spi...