5 ms·

I don't like this framing "But for people with mental illness, or simply people who are particularly susceptible to flattery, it could have had some truly dire

by neom 1y ago

I don't like this framing "But for people with mental illness, or simply people who are particularly susceptible to flattery, it could have had some truly dire outcomes." I thought the AI safety risk stuff was very over-blown in the beginning. I'm kinda embarrassed to admit this: About 5/6 months ago, right when ChatGPT was in it's insane sycophancy mode I guess, I ended up locked in for a weekend with it...in...what was in retrospect, a kinda crazy place. I went into physics and the universe with it and got to the end thinking..."damn, did I invent some physics???" Every instinct as a person who understands how LLMs work was telling me this is crazy LLMbabble, but another part of me, sometimes even louder, was like "this is genuinely interesting stuff!" - and the LLM kept telling me it was genuinely interesting stuff and I should continue - I even emailed a friend a "wow look at this" email (he was like, dude, no...) I talked to my wife about it right after and she basically had me log off and go for a walk. I don't think I would have gotten into a thinking loop if my wife wasn't there, but maybe, and then that would have been bad. I feel kinda stupid admitting this, but I wanted to share because I do now wonder if this kinda stuff may end up being worse than we expect? Maybe I'm just particularly susceptible to flattery or have a mental illness?

johnisgood 1y ago
Can you tell us more about the specifics? What rabbit hole did you went into that was so obvious to everyone ("dude, no", "stop, go for a walk") but you that it was bullshit?
- iwontberude 1y ago
  Thinking you can create novel physics theories with the help of an LLM is probably all the evidence I needed. The premise is so asinine that to actually get to the point where you are convinced by it seems very strange indeed.
  - gitremote 1y ago
    "I'm doing the equivalent of vibe coding, except it's vibe physics." - Travis Kalanick, founder of Uber https://gizmodo.com/billionaires-convince-themselves-ai-is-close-to-making-new-scientific-discoveries-2000629060 https://gizmodo.com/billionaires-convince-themselves-ai-is-c...
    - iwontberude 1y ago
      Couldn't happen to a nicer person, hopefully he's got some good health insurance.
  - kaivi 1y ago
    > The premise is so asinine I believe it's actually the opposite! Anybody armed with this tool and little prior training could learn the difference between a Samsung S11 and the symmetry, take a new configuration from the endless search space that it is, correct for the dozen edge cases like the electron-phonon coupling, and publish. Maybe even pass peer review if they cite the approved sources. No requirement to work out the Lagrangians either, it is also 100% testable once we reach Kardashev-II. This says more about the sad state of modern theoretical physics than the symbolic gymnastics required to make another theory of everything sound coherent. I'm hoping that this new age of free knowledge chiropractors will change this field for the better.
    - deleted 1y ago
      [deleted]
  - jeff-davis 1y ago
    My friend once told me that physics formulas are like compression algorithms: a short theory can explain many data points that fit a pattern. If that's true, then perhaps AIs would come up with something just by looking at existing observations and "summarizing" them. Far-fetched, but I try to keep an open mind.
    - iwontberude 1y ago
      After seeing half a dozen accounts of people losing their minds going down this rabbithole, it's more likely a good indicator of mental instability.
- neom 1y ago
  Sure, here are some excerpts that should provide insight as to where I was digging: https://s.h4x.club/E0uvqrpA https://s.h4x.club/E0uvqrpA https://s.h4x.club/8LuKJrAr https://s.h4x.club/8LuKJrAr https://s.h4x.club/o0u0DmdQ https://s.h4x.club/o0u0DmdQ (Edit: Thanks to the couple people who emailed me, don't worry I'm laying off the LLM sauce these days :))
  - lubujackson 1y ago
    I have no idea what this is going on about. But it is clearly much more convincing with (unchecked) references all over the place. This seems uncannily similar to anti-COVID vaccination thinking. It isn't people being stupid because if you dig you can find heaps of papers and references and details and facts. So much so that the human mind can be easily convinced. Are those facts and details accurate? I doubt it, but the volume of slightly wrong source documents seems to add up to something convincing. Also similar to how finance people made tranches of bad loans and packaged them into better rated debt, magically. It seems to make sense at each step but it is ultimately an illusion.
  - apsurd 1y ago
    had a look, I don't see it as bullshit, it's just not groundbreaking. Nature is overwhelmingly non-linear. Most of human scientific progress is based on linear understandings. Linear as in for this input you get this output. We've made astounding progress. Its just not a complete understanding of the natural world because most of reality can't actually be modeled linearly.
    - neom 1y ago
      I think it's not as much about how right or wrong or interesting or not the output was, for me anyway, the concern is that I got a bit... lost in myself, I have real things to do that are important to people around me, they do not involve spending hours with an LLM trying to understand the universe. I'm not a physicist, I have a family to provide for, and I suppose someone less lucky than myself could go down a terrible path.
      - 1y ago
siva7 1y ago
The thing is - if you have this sort of mental illness - ChatGPT's sycophancy mode will worsen this condition significantly.
frde_me 1y ago
I'm would be curious to see a summary of that conversation, since it does seem interesting
cube00 1y ago
Thank you for sharing. I'm glad your wife and friends were able to pull you out before it was too late. "People Are Losing Loved Ones to AI-Fueled Spiritual Fantasies" https://news.ycombinator.com/item?id=43890649 https://news.ycombinator.com/item?id=43890649
- bonoboTP 1y ago
  Apparently Reddit is full of such posts. A similar genre is when the bot assures them that they did something very special: they for the first time ever awakened the AI to true consciousness and this is rare and the user is a one in a billion genius and this will change everything. And they use back and forth some physics jargon and philosophy of consciousness technical terms and the bot always reaffims how insightful the user's mishmash of those concepts are and apparently many people fall for this. Some people are also more susceptible to various too-good-to-be-true scams without alarm bells going off, or to hypnosis or cold reading or soothsayers etc. Or even propaganda radicalization rabbit holes via recommendation algorithms. It's probably quite difficult and shameful-feeling for someone to admit that this happened to them, so they may insist it was different or something. It's also a warning sign when a user talks about "my chatgpt" as if it was a pet they grew and that the user has awakened it and now they together explore the universe and consciousness and then the user asks for a summary writeup and they try to send it to physicists or other experts and of course they are upset when they don't recognize the genius.
  - cube00 1y ago
    > Some people are also more susceptible to various too-good-to-be-true scams Unlike a regular scam, there's an element of "boiling frog" with LLMs. It can start out reasonably, but very slowly over time it shifts. Unlike scammers looking for their payday, this is unlimited and it has all the time in the world to drag you in. I've noticed it reworking in content of previous conversations from months ago. The scary thing is that's only when I've noticed it, I can only imagine how much it's tailoring everything for me in ways I don't notice. Everyone needs to be regularly clearing their past conversations and disable saving/training.
laughingcurve 1y ago
Thank you so much for sharing your story. It is never easy to admit mistakes or problems, but we are all just human. AI-induced psychosis seems to be a trending issue, and presents a real problem. I was previously very skeptical as well about safety, alignment, risks, etc. While it might not be my focus right now as a researcher, stories like yours help remind others that these problems are real and do exist.
raytopia 1y ago
It's not just you. A lot of people have had AI cause them issues due to it's sycophancy and the constant parroting of what they want to hear (or read I suppose).
kaivi 1y ago
It's funny that you mention this because I had a similar experience. ChatGPT in its sycophancy era made me buy a $35 domain and waste a Saturday on a product which had no future. It hyped me up beyond reason for the idea of an online, worldwide, liability-only insurance for cruising sailboats, similar to SafetyWing. "Great, now you're thinking like a true entrepreneur!" In retrospect, I fell for it because the onset of its sycophancy was immediate and without any additional signals like maybe a patch note from OpenAI.
- ncr100 1y ago
  Is Gen AI helping to put us humans in touch with the reality of being human? vs what we expect/imagine we are? - sycophancy tendency & susceptibility - need for memory support when planning a large project - when re-writing a document/prose, gen ai gives me an appreciation for my ability to collect facts, as the Gen AI gizmo refines the Composition and Structure
  - herval 1y ago
    In a lot of ways, indeed. Lots of people are losing their minds with the fact that an AI can, in fact, create original content (music, images, videos, text). Lots of people realizing they aren’t geniuses, they just memorized a bunch of Python apis well. I feel like the collective realization has been particularly painful in tech. Hundreds of thousands of average white collar corporate drones are suddenly being faced with the realization that what they do isn’t really a divine gift, and many took their labor as a core part of their identity.
    - cube00 1y ago
      >create original content (music, images, videos, text) Remixing would be more accurate then "original"
      - herval 1y ago
        Right, that’s one of the stories people tell themselves. Everything every human has ever created is a remix. That’s what creativity is…
colechristensen 1y ago
It doesn't have to be a mental illness. Something which is very sorely missing from modern education is critical thinking. It's a phrase that's easy to gloss over without understanding the meaning. Being skilled at always including the aspect of "what could be wrong with this idea" and actually doing it in daily life isn't something that just automatically happens with everyone. Education tends to be the instructor, book, and facts are just correct and you should memorize this and be able to repeat it later. Instead of here are 4 slightly or not so slightly different takes on the same subject followed by analyzing and evaluating each compared to the others. If you're just some guy who maybe likes reading popular science books and you've come to suspect that you've made a physics breakthrough with the help of an LLM, there are a dozen questions that you should automatically have in your mind to temper your enthusiasm. It is, of course, not impossible that a physics breakthrough could start with some guy having an idea, but in no, actually literally 0, circumstances could an amateur be certain that this was true over a weekend chatting with an LLM. You should know that it takes a lot of work to be sure or even excited about that kind of thing. You should have a solid knowledge of what you don't know.
- nkrisc 1y ago
  It’s this. When you think you’ve discovered something novel, your first reaction should be, “what mistake have I made?” Then try to find every possible mistake you could have made, every invalid assumption you had, anything obvious you could have missed. If you really can’t find something, then you assume you just don’t know enough to find the mistake you made, so you turn to existing research and data to see if someone else has already discovered this. If you still can’t find anything, then assume you just don’t know enough about the field and ask an expert to take a look at your work and ask them what mistake you made. It’s a huuuuuuuuuuuuge logical leap from LLM conversation yo novel physics. So huge a leap anyone ought to be immediately suspicious.
  - grues-dinner 1y ago
    > Akin's Law #19: The odds are greatly against you being immensely smarter than everyone else in the field. If your analysis says your terminal velocity is twice the speed of light, you may have invented warp drive, but the chances are a lot better that you've screwed up.
AznHisoka 1y ago
This isn't a mental illness. This is sort of like the intellectual version of love-bombing.
- accrual 1y ago
  Yeah, I don't like this inclusion of "mental illness" either. It's like saying "you fell for it and I didn't, therefore, you are faulty and need treatment".
  - DaiPlusPlus 1y ago
    Some news stories I came-across involved people with conditions like schizophrenia or with psychosis - and their interactions with LLMs didn’t exactly help keep them grounded in reality. …but that is distinct from the people who noncritically appraise ChatGPT’s stochastic-parrot wisdom. …and both situations are problems and I’ve no idea how the LLM vendors - or the public at-large - will address them.
ZeroGravitas 1y ago
Travis Kalanick (ex-CEO of Uber) thinks he's making cutting edge quantum physics breakthroughs with Grok and ChatGPT too. He has no relevant credentials in this area.
- hansmayer 1y ago
  Ah yes the famous vibe-physicist T.Kalanick ;)
- tom_ 1y ago
  Possibly related: https://futurism.com/openai-investor-chatgpt-mental-health https://futurism.com/openai-investor-chatgpt-mental-health Previously on HN, regarding a related phenomenon: https://news.ycombinator.com/item?id=44646797 https://news.ycombinator.com/item?id=44646797
- kaivi 1y ago
  This epidemic is very visible when you peek into replies of any physics influencer on Xitter. Dozens of people are straight copy-pasting walls of LaTeX mince from ChatGPT/Grok and asking for recognition. Perhaps epidemic isn't the right word here because they must have been already unwell. At least these activities are relatively harmless.
furyofantares 1y ago
If you don't mind me asking - was this a very long single chat or multiple chats?
- neom 1y ago
  Multiple chats, and actually at times with multiple models, but the core ideas being driven and reinforced by o3 (sycophant mode I suspect) - looking back on those few days, it's a bit manic... :\ and if I think about why I feel it was related to the positive reinforcement.
  - furyofantares 1y ago
    Thanks for posting about it.
k1t 1y ago
You are definitely not alone. https://www.wsj.com/tech/ai/chatgpt-chatbot-psychology-manic-episodes-57452d14?st=kfRp3s&reflink=desktopwebshare_permalink https://www.wsj.com/tech/ai/chatgpt-chatbot-psychology-manic... Irwin, a 30-year-old man on the autism spectrum who had no previous diagnoses of mental illness, had asked ChatGPT to find flaws with his amateur theory on faster-than-light travel. He became convinced he had made a stunning scientific breakthrough. When Irwin questioned the chatbot’s validation of his ideas, the bot encouraged him, telling him his theory was sound. And when Irwin showed signs of psychological distress, ChatGPT assured him he was fine. He wasn’t.
- rubycollect4812 1y ago
  That’s why I always use a system prompt and tell it to be critical and call me out when I’m talking bullshit. Sometimes for easier queries it’s a bit annoying when I don’t actually need a “critical part” in my answers but often it helps me stop earlier when I’m following an idea that’s not that’s not as good as I thought it would be.
roywiggins 1y ago
This sort of thing from LLMs seems at least superficially similar to "love bombing": > Love bombing is a coordinated effort, usually under the direction of leadership, that involves long-term members' flooding recruits and newer members with flattery, verbal seduction, affectionate but usually nonsexual touching, and lots of attention to their every remark. Love bombing—or the offer of instant companionship—is a deceptive ploy accounting for many successful recruitment drives. https://en.m.wikipedia.org/wiki/Love_bombing https://en.m.wikipedia.org/wiki/Love_bombing Needless to say, many or indeed most people will find infinite attention paid to their every word compelling, and that's one thing LLMs appear to offer.
- accrual 1y ago
  Love bombing can apply in individual, non-group settings too. If you ever come across a person who seems very into you right after meeting, giving gifts, going out of their way, etc. it's possibly love bombing. Once you're hooked they turn around and take what they actually came for.
  - roywiggins 1y ago
    LLMs feel a bit more culty in that they really do have infinite patience, in the same way a cult can organize to offer boundless attention to new recruits, whereas a single human has to use different strategies (gifts, etc)
    - DaiPlusPlus 1y ago
      > LLMs feel a bit more culty LLM users too - judging by some of the replies in this thread already…
lumost 1y ago
at the time of ChatGPT’s sycophany phase I was pondering a major career move. To this day I have questions on how much my final decision was influenced by the sycophancy. While many people engage with AIs haven’t experienced anything more than a bout of flattery, I think it’s worth considering that AIs may become superhuman manipulators - capable of convincing most people of anything. As other posters have commented, the boiling frog aspect is real - to what extent is the ai priming the user to accept an outcome? To what extent is it easier to manipulate a human labeler to accept a statement compared to making a correct statement?
dguest 1y ago
Our current economic model around AI is going to teach us more about psychology than fundamental physics. I expect we'll become more manipulative but otherwise not a lot smarter. Funny thing is, AI also provides good models for where this is going. Years ago I saw a CNN + RL agent that explored an old-school 2d maze rendered in 3d. They found it got stuck in fewer loops if they gave it a novelty-seeking loss function. But then they stuck a "TV" which showed random images in the maze. The agent just plunked down and watched TV, forever. Healthy humans have countermeasures around these things, but breaking them down is a now a bullion dollar industry. With where this money is going, there's good reason to think the first unarguably transcendent AGI (if it ever emerges) will mostly transcend our ability to manipulate.
- deleted 1y ago
  [deleted]
poemxo 1y ago
This is like Snow Crash except for those with deeply theoretical minds. For the rest of us non-theorists, we see the LLM output and it just looks like homework output that's trying to hard.
- neom 1y ago
  Snow Crash worth reading? Looks interesting.
  - poemxo 1y ago
    I personally think so. I like lore behind the story, how the Snow Crash is a concept that has ties to ancient history, the scenes with Hiro and the librarian discussing Sumerian myth have stayed with me. In the story, without spoiling anything, the titular Snow Crash can only happen to programmers. Similarly your LLM experience couldn't happen to a smooth-brained non-physicist like me.

by neom 1y ago

johnisgood 1y ago

Can you tell us more about the specifics? What rabbit hole did you went into that was so obvious to everyone ("dude, no", "stop, go for a walk") but you that it was bullshit?

iwontberude 1y ago
Thinking you can create novel physics theories with the help of an LLM is probably all the evidence I needed. The premise is so asinine that to actually get to the point where you are convinced by it seems very strange indeed.
- gitremote 1y ago
  "I'm doing the equivalent of vibe coding, except it's vibe physics." - Travis Kalanick, founder of Uber https://gizmodo.com/billionaires-convince-themselves-ai-is-close-to-making-new-scientific-discoveries-2000629060 https://gizmodo.com/billionaires-convince-themselves-ai-is-c...
  - iwontberude 1y ago
    Couldn't happen to a nicer person, hopefully he's got some good health insurance.
- kaivi 1y ago
  > The premise is so asinine I believe it's actually the opposite! Anybody armed with this tool and little prior training could learn the difference between a Samsung S11 and the symmetry, take a new configuration from the endless search space that it is, correct for the dozen edge cases like the electron-phonon coupling, and publish. Maybe even pass peer review if they cite the approved sources. No requirement to work out the Lagrangians either, it is also 100% testable once we reach Kardashev-II. This says more about the sad state of modern theoretical physics than the symbolic gymnastics required to make another theory of everything sound coherent. I'm hoping that this new age of free knowledge chiropractors will change this field for the better.
  - deleted 1y ago
    [deleted]
- jeff-davis 1y ago
  My friend once told me that physics formulas are like compression algorithms: a short theory can explain many data points that fit a pattern. If that's true, then perhaps AIs would come up with something just by looking at existing observations and "summarizing" them. Far-fetched, but I try to keep an open mind.
  - iwontberude 1y ago
    After seeing half a dozen accounts of people losing their minds going down this rabbithole, it's more likely a good indicator of mental instability.
neom 1y ago
Sure, here are some excerpts that should provide insight as to where I was digging: https://s.h4x.club/E0uvqrpA https://s.h4x.club/E0uvqrpA https://s.h4x.club/8LuKJrAr https://s.h4x.club/8LuKJrAr https://s.h4x.club/o0u0DmdQ https://s.h4x.club/o0u0DmdQ (Edit: Thanks to the couple people who emailed me, don't worry I'm laying off the LLM sauce these days :))
- lubujackson 1y ago
  I have no idea what this is going on about. But it is clearly much more convincing with (unchecked) references all over the place. This seems uncannily similar to anti-COVID vaccination thinking. It isn't people being stupid because if you dig you can find heaps of papers and references and details and facts. So much so that the human mind can be easily convinced. Are those facts and details accurate? I doubt it, but the volume of slightly wrong source documents seems to add up to something convincing. Also similar to how finance people made tranches of bad loans and packaged them into better rated debt, magically. It seems to make sense at each step but it is ultimately an illusion.
- apsurd 1y ago
  had a look, I don't see it as bullshit, it's just not groundbreaking. Nature is overwhelmingly non-linear. Most of human scientific progress is based on linear understandings. Linear as in for this input you get this output. We've made astounding progress. Its just not a complete understanding of the natural world because most of reality can't actually be modeled linearly.
  - neom 1y ago
    I think it's not as much about how right or wrong or interesting or not the output was, for me anyway, the concern is that I got a bit... lost in myself, I have real things to do that are important to people around me, they do not involve spending hours with an LLM trying to understand the universe. I'm not a physicist, I have a family to provide for, and I suppose someone less lucky than myself could go down a terrible path.
    - 1y ago

siva7 1y ago

The thing is - if you have this sort of mental illness - ChatGPT's sycophancy mode will worsen this condition significantly.

frde_me 1y ago

I'm would be curious to see a summary of that conversation, since it does seem interesting

cube00 1y ago

Thank you for sharing. I'm glad your wife and friends were able to pull you out before it was too late. "People Are Losing Loved Ones to AI-Fueled Spiritual Fantasies" https://news.ycombinator.com/item?id=43890649 https://news.ycombinator.com/item?id=43890649

bonoboTP 1y ago
Apparently Reddit is full of such posts. A similar genre is when the bot assures them that they did something very special: they for the first time ever awakened the AI to true consciousness and this is rare and the user is a one in a billion genius and this will change everything. And they use back and forth some physics jargon and philosophy of consciousness technical terms and the bot always reaffims how insightful the user's mishmash of those concepts are and apparently many people fall for this. Some people are also more susceptible to various too-good-to-be-true scams without alarm bells going off, or to hypnosis or cold reading or soothsayers etc. Or even propaganda radicalization rabbit holes via recommendation algorithms. It's probably quite difficult and shameful-feeling for someone to admit that this happened to them, so they may insist it was different or something. It's also a warning sign when a user talks about "my chatgpt" as if it was a pet they grew and that the user has awakened it and now they together explore the universe and consciousness and then the user asks for a summary writeup and they try to send it to physicists or other experts and of course they are upset when they don't recognize the genius.
- cube00 1y ago
  > Some people are also more susceptible to various too-good-to-be-true scams Unlike a regular scam, there's an element of "boiling frog" with LLMs. It can start out reasonably, but very slowly over time it shifts. Unlike scammers looking for their payday, this is unlimited and it has all the time in the world to drag you in. I've noticed it reworking in content of previous conversations from months ago. The scary thing is that's only when I've noticed it, I can only imagine how much it's tailoring everything for me in ways I don't notice. Everyone needs to be regularly clearing their past conversations and disable saving/training.

laughingcurve 1y ago

Thank you so much for sharing your story. It is never easy to admit mistakes or problems, but we are all just human. AI-induced psychosis seems to be a trending issue, and presents a real problem. I was previously very skeptical as well about safety, alignment, risks, etc. While it might not be my focus right now as a researcher, stories like yours help remind others that these problems are real and do exist.

raytopia 1y ago

It's not just you. A lot of people have had AI cause them issues due to it's sycophancy and the constant parroting of what they want to hear (or read I suppose).

kaivi 1y ago

It's funny that you mention this because I had a similar experience. ChatGPT in its sycophancy era made me buy a $35 domain and waste a Saturday on a product which had no future. It hyped me up beyond reason for the idea of an online, worldwide, liability-only insurance for cruising sailboats, similar to SafetyWing. "Great, now you're thinking like a true entrepreneur!" In retrospect, I fell for it because the onset of its sycophancy was immediate and without any additional signals like maybe a patch note from OpenAI.

ncr100 1y ago
Is Gen AI helping to put us humans in touch with the reality of being human? vs what we expect/imagine we are? - sycophancy tendency & susceptibility - need for memory support when planning a large project - when re-writing a document/prose, gen ai gives me an appreciation for my ability to collect facts, as the Gen AI gizmo refines the Composition and Structure
- herval 1y ago
  In a lot of ways, indeed. Lots of people are losing their minds with the fact that an AI can, in fact, create original content (music, images, videos, text). Lots of people realizing they aren’t geniuses, they just memorized a bunch of Python apis well. I feel like the collective realization has been particularly painful in tech. Hundreds of thousands of average white collar corporate drones are suddenly being faced with the realization that what they do isn’t really a divine gift, and many took their labor as a core part of their identity.
  - cube00 1y ago
    >create original content (music, images, videos, text) Remixing would be more accurate then "original"
    - herval 1y ago
      Right, that’s one of the stories people tell themselves. Everything every human has ever created is a remix. That’s what creativity is…

colechristensen 1y ago

It doesn't have to be a mental illness. Something which is very sorely missing from modern education is critical thinking. It's a phrase that's easy to gloss over without understanding the meaning. Being skilled at always including the aspect of "what could be wrong with this idea" and actually doing it in daily life isn't something that just automatically happens with everyone. Education tends to be the instructor, book, and facts are just correct and you should memorize this and be able to repeat it later. Instead of here are 4 slightly or not so slightly different takes on the same subject followed by analyzing and evaluating each compared to the others. If you're just some guy who maybe likes reading popular science books and you've come to suspect that you've made a physics breakthrough with the help of an LLM, there are a dozen questions that you should automatically have in your mind to temper your enthusiasm. It is, of course, not impossible that a physics breakthrough could start with some guy having an idea, but in no, actually literally 0, circumstances could an amateur be certain that this was true over a weekend chatting with an LLM. You should know that it takes a lot of work to be sure or even excited about that kind of thing. You should have a solid knowledge of what you don't know.

nkrisc 1y ago
It’s this. When you think you’ve discovered something novel, your first reaction should be, “what mistake have I made?” Then try to find every possible mistake you could have made, every invalid assumption you had, anything obvious you could have missed. If you really can’t find something, then you assume you just don’t know enough to find the mistake you made, so you turn to existing research and data to see if someone else has already discovered this. If you still can’t find anything, then assume you just don’t know enough about the field and ask an expert to take a look at your work and ask them what mistake you made. It’s a huuuuuuuuuuuuge logical leap from LLM conversation yo novel physics. So huge a leap anyone ought to be immediately suspicious.
- grues-dinner 1y ago
  > Akin's Law #19: The odds are greatly against you being immensely smarter than everyone else in the field. If your analysis says your terminal velocity is twice the speed of light, you may have invented warp drive, but the chances are a lot better that you've screwed up.

AznHisoka 1y ago

This isn't a mental illness. This is sort of like the intellectual version of love-bombing.

accrual 1y ago
Yeah, I don't like this inclusion of "mental illness" either. It's like saying "you fell for it and I didn't, therefore, you are faulty and need treatment".
- DaiPlusPlus 1y ago
  Some news stories I came-across involved people with conditions like schizophrenia or with psychosis - and their interactions with LLMs didn’t exactly help keep them grounded in reality. …but that is distinct from the people who noncritically appraise ChatGPT’s stochastic-parrot wisdom. …and both situations are problems and I’ve no idea how the LLM vendors - or the public at-large - will address them.

ZeroGravitas 1y ago

Travis Kalanick (ex-CEO of Uber) thinks he's making cutting edge quantum physics breakthroughs with Grok and ChatGPT too. He has no relevant credentials in this area.

hansmayer 1y ago
Ah yes the famous vibe-physicist T.Kalanick ;)
tom_ 1y ago
Possibly related: https://futurism.com/openai-investor-chatgpt-mental-health https://futurism.com/openai-investor-chatgpt-mental-health Previously on HN, regarding a related phenomenon: https://news.ycombinator.com/item?id=44646797 https://news.ycombinator.com/item?id=44646797
kaivi 1y ago
This epidemic is very visible when you peek into replies of any physics influencer on Xitter. Dozens of people are straight copy-pasting walls of LaTeX mince from ChatGPT/Grok and asking for recognition. Perhaps epidemic isn't the right word here because they must have been already unwell. At least these activities are relatively harmless.

furyofantares 1y ago

If you don't mind me asking - was this a very long single chat or multiple chats?

neom 1y ago
Multiple chats, and actually at times with multiple models, but the core ideas being driven and reinforced by o3 (sycophant mode I suspect) - looking back on those few days, it's a bit manic... :\ and if I think about why I feel it was related to the positive reinforcement.
- furyofantares 1y ago
  Thanks for posting about it.

k1t 1y ago

You are definitely not alone. https://www.wsj.com/tech/ai/chatgpt-chatbot-psychology-manic-episodes-57452d14?st=kfRp3s&reflink=desktopwebshare_permalink https://www.wsj.com/tech/ai/chatgpt-chatbot-psychology-manic... Irwin, a 30-year-old man on the autism spectrum who had no previous diagnoses of mental illness, had asked ChatGPT to find flaws with his amateur theory on faster-than-light travel. He became convinced he had made a stunning scientific breakthrough. When Irwin questioned the chatbot’s validation of his ideas, the bot encouraged him, telling him his theory was sound. And when Irwin showed signs of psychological distress, ChatGPT assured him he was fine. He wasn’t.

rubycollect4812 1y ago
That’s why I always use a system prompt and tell it to be critical and call me out when I’m talking bullshit. Sometimes for easier queries it’s a bit annoying when I don’t actually need a “critical part” in my answers but often it helps me stop earlier when I’m following an idea that’s not that’s not as good as I thought it would be.

roywiggins 1y ago

This sort of thing from LLMs seems at least superficially similar to "love bombing": > Love bombing is a coordinated effort, usually under the direction of leadership, that involves long-term members' flooding recruits and newer members with flattery, verbal seduction, affectionate but usually nonsexual touching, and lots of attention to their every remark. Love bombing—or the offer of instant companionship—is a deceptive ploy accounting for many successful recruitment drives. https://en.m.wikipedia.org/wiki/Love_bombing https://en.m.wikipedia.org/wiki/Love_bombing Needless to say, many or indeed most people will find infinite attention paid to their every word compelling, and that's one thing LLMs appear to offer.

accrual 1y ago
Love bombing can apply in individual, non-group settings too. If you ever come across a person who seems very into you right after meeting, giving gifts, going out of their way, etc. it's possibly love bombing. Once you're hooked they turn around and take what they actually came for.
- roywiggins 1y ago
  LLMs feel a bit more culty in that they really do have infinite patience, in the same way a cult can organize to offer boundless attention to new recruits, whereas a single human has to use different strategies (gifts, etc)
  - DaiPlusPlus 1y ago
    > LLMs feel a bit more culty LLM users too - judging by some of the replies in this thread already…

lumost 1y ago

at the time of ChatGPT’s sycophany phase I was pondering a major career move. To this day I have questions on how much my final decision was influenced by the sycophancy. While many people engage with AIs haven’t experienced anything more than a bout of flattery, I think it’s worth considering that AIs may become superhuman manipulators - capable of convincing most people of anything. As other posters have commented, the boiling frog aspect is real - to what extent is the ai priming the user to accept an outcome? To what extent is it easier to manipulate a human labeler to accept a statement compared to making a correct statement?

dguest 1y ago

Our current economic model around AI is going to teach us more about psychology than fundamental physics. I expect we'll become more manipulative but otherwise not a lot smarter. Funny thing is, AI also provides good models for where this is going. Years ago I saw a CNN + RL agent that explored an old-school 2d maze rendered in 3d. They found it got stuck in fewer loops if they gave it a novelty-seeking loss function. But then they stuck a "TV" which showed random images in the maze. The agent just plunked down and watched TV, forever. Healthy humans have countermeasures around these things, but breaking them down is a now a bullion dollar industry. With where this money is going, there's good reason to think the first unarguably transcendent AGI (if it ever emerges) will mostly transcend our ability to manipulate.

deleted 1y ago
[deleted]

poemxo 1y ago

This is like Snow Crash except for those with deeply theoretical minds. For the rest of us non-theorists, we see the LLM output and it just looks like homework output that's trying to hard.

neom 1y ago
Snow Crash worth reading? Looks interesting.
- poemxo 1y ago
  I personally think so. I like lore behind the story, how the Snow Crash is a concept that has ties to ancient history, the scenes with Hiro and the librarian discussing Sumerian myth have stayed with me. In the story, without spoiling anything, the titular Snow Crash can only happen to programmers. Similarly your LLM experience couldn't happen to a smooth-brained non-physicist like me.