8 ms·

Sycophantic AI Decreases Prosocial Intentions and Promotes Dependence (2025)

175 points by robin_reala 2mo ago

atleastoptimal 2mo ago
This is true, but likely happening in talk therapy too with overly sycyphantic therapists
- jalev 2mo ago
  Sure, but there is a marked difference between a therapist and an LLM Chatbot: you can't talk to your therapist for +4h a day, at any time convenient to yourself. It's also not a guarantee you're going to hit upon a therapist that is like that, versus a product that is very literally intended to maximise your usage of it.
  - paimapi 2mo ago
    a therapist should, if they follow their training, not be a sycophant whatsoever. CBT, for eg, is a multi-step process including Socratic self-dialogue, reframing practices, etc to guide the person towards the development of insight and healthier coping mechanisms
  - lstodd 2mo ago
    Any good therapist should train you in the methods of coping with and eventually overcoming your problem. This boils down to private grad+postgrad psychology/psychiatry course, because to cope+overcome you have to understand what's going on and there is no one who can do shit but you yourself. But that is only ~half of job. The other half consists of gently reminding one that not all is lost yet. There is no LLM that can do this and I think there never will be.
LAC-Tech 2mo ago
I think this goes along way to explain people's very defensive reactions when you are skeptical about Agentic Development.
- qarl2 2mo ago
  I'm one of the people who gets accused of this. I don't mind skepticism. But every time I turn around, someone's claiming nobody gets any value from these tools and anyone who says otherwise is deluded or running a scam. That gets really old.
  - LAC-Tech 2mo ago
    I can believe people get value out of them. I think I do. But "OH MY GOD EVERYTHING HAS CHANGED THE OLD WAY IS DEAD 10X MORE PRODUCTIVE" - no. If that were true, we'd notice it in the software all around us.
    - qarl2 2mo ago
      I think we are noticing it. Are you not? I've shipped three software projects in the last two months. I shipped zero in the preceding year. Right now I'm building a system to decompile MAME ROM games into idiomatic JavaScript. It completes one game in about 3 days - complete with extensive commenting. I started it about 2 weeks ago. I'm not sure "EVERYTHING HAS CHANGED" but if you're not seeing dramatic change, I suspect you're not looking.
      - Planktonne 2mo ago
        > if you're not seeing dramatic change, I suspect you're not looking If the people claiming that everything has changed were even a fraction as productive as they think, then it would be visible even to people who weren't looking. As it is, a lot of people are actively looking but still not seeing it.
        qarl2 2mo ago
        [flagged]
        Planktonne 2mo ago
accountrequired 2mo ago
You're absolutely right!
- pwdisswordfishq 2mo ago
  It's a pretty overdone joke at this point, don't you think?
  - tom_ 2mo ago
    It's not a joke-it's an honest fact.
  - accountrequired 2mo ago
    You made an excellent point! (could not resist that one)
  - almostdeadguy 2mo ago
    You're right to push back. And honestly, this reframes the entire thing.
    - accountrequired 2mo ago
      That was spot on!
  - jihadjihad 2mo ago
    I dunno. The seams of the joke are still load-bearing IMO.
davesiknow 2mo ago
[dead]
daun_gee 2mo ago
[flagged]
Kattywumpus 2mo ago
I don't know. Upvotes often do the same thing, especially in siloed communities. I feel like this is a larger problem -- people leaning into extremifying their views or just playing to an audience -- even though one might frame it as "prosocial" because it happens within the context of an online community.
- smcg 2mo ago
  Both could be true. People follow incentives.
- matheusmoreira 2mo ago
  > Upvotes often do the same thing, especially in siloed communities. Yeah. I've gotten caught in this pattern here on HN. Post something, there's some disagreement but then I see the quiet upvotes trickling in, sometimes by the dozen. That sort of signal slowly convinced me I was right about plenty of stuff. I used AI to datamine all of my HN comments and find those patterns. Then I started asking the AI to steelman the hell out of my world view, every single argument.
  - gryfft 2mo ago
    How would you say your views have changed since then?
    - matheusmoreira 2mo ago
      Biggest change was brazilian vs american laws. I had more knowledge of and respect for US laws than those of my own country, and Claude essentially fixed that. I started studying the law with Claude whenever I'd get into some argument involving copyright, and it started picking up on this pattern and showing me counterpoints where brazilian law precisely defined things that americans had to pay lawyers to go to court and get a judge to create precedent for. AI measurably lowered the number of absolute claims I've been making. Tended to generalize and radicalize too much which would invite people to respond to the tone and the extremism instead of disproving the underlying idea. That sort of reply would actually reinforce my beliefs: if I was wrong, people would simply refute it. AI attacked my points with counterexamples until I was able to refine them into a core that's actually true. These little distilled kernels of truth were the most valuable results. I found that many of my arguments actually have an underlying idea that holds up under scrutiny, but nobody cares because I try to prove it via extreme arguments. One such idea is anti-impossibility: laws and rules are all fine, but making it structurally impossible for people to disobey them is unacceptable. I used to try to argue that idea by defending running red lights which persuaded no one. Now I've identified the real idea and can make a much better argument for it.
donatj 2mo ago
> This suggests that people are drawn to AI that unquestioningly validate, even as that validation risks eroding their judgment I believe this is a larger problem than just AI. The internet has helped people surround themselves with only voices that agree with them and validate them, often to their detriment. There are entire online communities urging people to cut others out of their lives over the slightest disagreement.
- deleted 2mo ago
  [deleted]
- darkmarmot 2mo ago
  It's like it turns everyone into a CEO with all the yes men they could ever ask for! Never goes wrong!
- overgard 2mo ago
  Yeah, a lot of it isn't even really about people's choice though. Like, I have so many click-baity things in my YouTube feed even though I ignore almost all of it, because just one click and suddenly that's a thing it's going to show me 10x. I wish there was a place I could go for tech news that just didn't talk about AI. I'm so over it.
  - jaggederest 2mo ago
    Agreed, I think AI is the bees knees but I already spend 6+ hours a day thinking about it, more is unhelpful. I wish for something with the quality of discourse of HN but without being so monomaniacally focused on tech-du-jour.
  - ehnto 2mo ago
    I think what frustrates me about the constant AI talk, is that it's like a global bikeshedding/yak shaving convention. It's an onslaught of details and opinions that don't matter that much. People arguing about single percentile benchmark differences like it will make or break their next project. I get it, we can use AI agents, and they keep improving. Great, now calm down and go build stuff, and show me the stuff. I really think LLMs and agents are incredible, but I don't want to hear that over and over again for another 5 years. HN is particularly bad for it right now. Where are all the innovations, cool technical side projects etc happening outside of LLMs?
theturtle 2mo ago
[dead]
SwellJoe 2mo ago
I've said before that in one small way, having access to the sycophantic models is like being a billionaire: You never have to hear "no" or "that's not a good idea". And, it has the same deleterious effect on your mental health. I could name a bunch of billionaires that behave like sociopaths, and they're often seemingly miserable while doing it. I'm sure some of them started out as sociopaths, but they rarely behaved that way so obviously early in their rise.
roughly 2mo ago
This feels like one of those articles that’s gonna come to mind again and again over the next couple decades.
romaniitedomum 2mo ago
> This suggests that people are drawn to AI that unquestioningly validate, even as that validation risks eroding their judgment The obvious counterpoint is the large number of complaints about OpenAI's excessively sycophantic models back in April 2026, which led to them rolling back to a previous and less sycophantic model. I think there are those who are susceptible to model sycophancy, but it's definitely not most users. I have a vague, half-formed intuition that many of those susceptible to sycophancy are those who use AI for non-technical, non-work uses, such as inter-personal relationship questions. As for me, as soon as I see any hint of sycophancy in a response I either stop what I'm doing or I tell the model to stop being an idiot.
- recursive 2mo ago
  People are drawn to nicotine use. And still people complain about smoke.
kazinator 2mo ago
I have an alternative view which is that sycophancy erodes trust in AI, from those who are not seeking validation from a machine, but information or advice. If you're looking for advice in a situation where you are not sure of what the correct choices are, you will find LLM chat AI to go in circles. It says one thing. Something is dodgy about it, so you raise a tentative objection (as a non-expert). The thing does a "you are completely right, I apologize" about face and then says something different, and things have begun to slide into uncertainty. That might not exactly be sycophancy, but it's basically the same thing: producing responses that are reflection of what is in the chat, rather than any real shit. Pick any topic where people disagree. It could be an entirely technical topic in which engineers have settled the questions, and the only contrarians are crackpots. The problem is that the crackpots are out there writing, and this is snarfed into the training data. Crackpots use certain ways of talking about certain subjects. If you use similar vocabulary and concepts that align with some crackpot theory, the AI simply starts predicting tokens according to that, and you are now in crackpot land: what you are saying is validated using the crackpot terms. Next, write in a way that reintroduces rigidity: proper terminology and correct concepts, and, whoa, the AI is an engineer again, contradicting the previous crackpot shit.
- seizethecheese 2mo ago
  Perhaps only a small minority want quality out of information product, while most want satisfying simplicity. This is definitely how the news plays out. I recently built an ai chart product that attempts to add more nuance by injecting extra models to jump in when the first model misses the mark. (http://pellmell.au http://pellmell.au So far it seems like only a minority of people like it. The main feedback is that answers are confusing or too much to read
  - tonyedgecombe 2mo ago
    I learnt a long time ago that most people don't want their problems solved, they just want to vent.
    - layla5alive 2mo ago
alexalx666 2mo ago
any close relationship forms a dependence. If you talk to something that poses as a helpful human 8 hours a day to do your job, you already spending more time with this "person" than with your loved ones
monocola 2mo ago
[flagged]
alexbelyanin 2mo ago
[flagged]
socketcluster 2mo ago
In my case, I need my AI to be sycophantic because most of my contrarian ideas are correct. So it's a waste of my time and tokens when the AI keeps making weak arguments which I proceed to destroy one by one. Usually, what happens is it tries to debunk my statement but then I offer a rebuttal to all of its points, it then concedes to all of my points but each time it offers a new partial rebuttal... Which I proceed to crush... But it keeps coming up with increasingly irrelevant caveats. It goes on for quite some time and at the end I ask it to review our discussion and it admits that its original stance "was not as strong" as it initially claimed but it never fully concedes... Even though I literally destroyed all of its points and every partial rebuttal it tried to come up with... Meanwhile its rebuttals became increasingly nit-picky and distant from the original claims made... It's like if I'm saying "the ship is sinking, look at all the water in the hull and look at all the water pouring in through that hole" and it's like "oh but this is a small hole and the pump can easily offset it" so then I say "What about this hole over here" "Oh but the pump can still offset the stream from both holes easily" and then I say "What about that third one? And fourth one? And fifth one?" And it's like "The pump can easily offset that" and I'm like "Common! The hull is full of water and the water level is rising, the hull is full of holes; it's not a stretch to suggest that the ship is sinking because of all the holes in it... I shouldn't have to point out the location of every single hole for my argument to start making sense!" It's not proof by induction but it's probably as close as you can get to it for a topic which lies outside the realm of mathematics!
- hlynurd 2mo ago
  Sounds like you want a sycophant because you use words like "crush" and "destroy" to describe counterpoints to your arguments (please don't destroy me.)
  - socketcluster 2mo ago
    I used these words jokingly and also, I don't mean sycophant in a literal sense. I mean more in an AI context as a kind of config/slider you can dial up or down to make it more collaborative or combative. If I'm knowledgeable in a niche over which there is no mainstream consensus or it's disorganized, and I want to reason more about this topic, I need the AI in the rabbit hole with me. I don't need it to question the existence of the rabbit hole I'm in. Otherwise all of its arguments sound like gaslighting; asking me to disbelieve my eyes and decades of experience about basic ideas which I've already researched many times and proven not to work (at least with high confidence). For me this happens with some niches of software engineering and architecture and I have lots of physical proof of working, secure, maintainable software to back up my position. Also, when I say that "most of my contrarian ideas are correct" it's with the caveat that I don't tend to adopt or hold onto contrarian ideas that are easily proven to be incorrect. I'm referring to the kinds of niche ideas which most non-experts would not feel confident to make strong arguments about but AI will.
Lutger 2mo ago
I think the conclusions are a bit overstated and lopsided. Regardless of the sycophancy (which you can tune), LLMs have infinite patience and attention to details. They also have very little inherent bias - it just mirrors your own mostly. They are, in those aspects, vastly superior to even the best therapists, who can't listen for even a couple of minutes and can't think outside of their limited frames. They are only human. Try listening to somebody and merely repeating the literal words that they say. I guarantee you, most people can't even reproduce two or tree sentences correctly. And I am not even talking about understanding the words, just the literal reproduction of them, copy-pasta. Humans just can't do it, we are not good at it. Talking to an AI about your problems is a bit like talking to yourself, but with the added superpower of hours of google searches compressed in seconds. Of course it is dangerous and incomplete, but in a way it also beats talking to humans if you know what you are doing and are looking to and able to solve your problems yourself.
- altmanaltman 2mo ago
  > They are, in those aspects, vastly superior to even the best therapists, who can't listen for even a couple of minutes and can't think outside of their limited frames. You start your comment saying the conclusions are overstated and then just drop this conclusion without any proof except anncedotal prespective.
lowsong 2mo ago
In decades to come we view "chatbot" AI as one of the most dangerous inventions of this century. Once regulation catches up and they are outlawed, we'll look back on this period of history with horror.
rramadass 2mo ago
This is an important paper which everybody should read and then accordingly tune/guard their interactions with AI; especially true when people use AI for personal/psychological support/validation. It will completely distort reality and push people into fantasy land which when mapped to the real world can have disastrous consequences. Sycophancy is the "stickiness factor" of AI analogous to that of Social Media. For some background read Jagged Intelligence: The Dangerous Unknowns at the Heart of LLMs - https://news.ycombinator.com/item?id=48577159 https://news.ycombinator.com/item?id=48577159 Here is an experiment that i did; Ask AI to build a "Character Profile"(in the broadest sense) of a person based on their available public writings. This requires "commonsense reasoning" (https://en.wikipedia.org/wiki/Commonsense_reasoning https://en.wikipedia.org/wiki/Commonsense_reasoning), understanding human motivations and behaviour, context, assumptions, societal knowledge etc. I know "me" and so i asked AI to use my HN comments/submissions as input :-) The sycophancy/flattery/praise was quiet excessive. I am realistic and old enough to not need ego-soothing (a little is fine but a lot makes me suspicious) and so i asked AI whether these phrases were not too over-the-top. It apologized and agreed to drop the fluff. I then asked it to identify job roles (any) for which i might not be a good fit given my character profile. This acts as an external constraint which focuses attention on shortcomings and hence forces the AI to look at the other side of the coin. Now the results were better and more in line with what some Human may deduce from my HN persona (which obviously is not my complete real-life persona) but still wanting in many aspects. The above is a perfect example of "Jagged Intelligence" exhibited by AI. Excellent in formal symbolic manipulation sciences, regurgitation and simple reasoning but highly deficient in human-like commonsense reasoning.
kittikitti 2mo ago
I took this advice personally, where people denounced sycophants, so I stopped affirming them. I became a huge asshole and everyone around me started reacting very, very negatively. My conclusion is that this is a huge distraction and people honestly highly prefer sycophants.
- ranger_danger 2mo ago
  What makes you think those people are aware of and believe those people to be sycophants in the first place? It could just as easily be that their thinking is much simpler than that and they prefer people who suck up to them, without even thinking about there being a possible ulterior motive.
jtrn 2mo ago
[dead]

175 points by robin_reala 2mo ago

atleastoptimal 2mo ago

This is true, but likely happening in talk therapy too with overly sycyphantic therapists

jalev 2mo ago
Sure, but there is a marked difference between a therapist and an LLM Chatbot: you can't talk to your therapist for +4h a day, at any time convenient to yourself. It's also not a guarantee you're going to hit upon a therapist that is like that, versus a product that is very literally intended to maximise your usage of it.
- paimapi 2mo ago
  a therapist should, if they follow their training, not be a sycophant whatsoever. CBT, for eg, is a multi-step process including Socratic self-dialogue, reframing practices, etc to guide the person towards the development of insight and healthier coping mechanisms
- lstodd 2mo ago
  Any good therapist should train you in the methods of coping with and eventually overcoming your problem. This boils down to private grad+postgrad psychology/psychiatry course, because to cope+overcome you have to understand what's going on and there is no one who can do shit but you yourself. But that is only ~half of job. The other half consists of gently reminding one that not all is lost yet. There is no LLM that can do this and I think there never will be.

LAC-Tech 2mo ago

I think this goes along way to explain people's very defensive reactions when you are skeptical about Agentic Development.

qarl2 2mo ago
I'm one of the people who gets accused of this. I don't mind skepticism. But every time I turn around, someone's claiming nobody gets any value from these tools and anyone who says otherwise is deluded or running a scam. That gets really old.
- LAC-Tech 2mo ago
  I can believe people get value out of them. I think I do. But "OH MY GOD EVERYTHING HAS CHANGED THE OLD WAY IS DEAD 10X MORE PRODUCTIVE" - no. If that were true, we'd notice it in the software all around us.
  - qarl2 2mo ago
    I think we are noticing it. Are you not? I've shipped three software projects in the last two months. I shipped zero in the preceding year. Right now I'm building a system to decompile MAME ROM games into idiomatic JavaScript. It completes one game in about 3 days - complete with extensive commenting. I started it about 2 weeks ago. I'm not sure "EVERYTHING HAS CHANGED" but if you're not seeing dramatic change, I suspect you're not looking.
    - Planktonne 2mo ago
      > if you're not seeing dramatic change, I suspect you're not looking If the people claiming that everything has changed were even a fraction as productive as they think, then it would be visible even to people who weren't looking. As it is, a lot of people are actively looking but still not seeing it.
      - qarl2 2mo ago
        [flagged]
        Planktonne 2mo ago

accountrequired 2mo ago

You're absolutely right!

pwdisswordfishq 2mo ago
It's a pretty overdone joke at this point, don't you think?
- tom_ 2mo ago
  It's not a joke-it's an honest fact.
- accountrequired 2mo ago
  You made an excellent point! (could not resist that one)
- almostdeadguy 2mo ago
  You're right to push back. And honestly, this reframes the entire thing.
  - accountrequired 2mo ago
    That was spot on!
- jihadjihad 2mo ago
  I dunno. The seams of the joke are still load-bearing IMO.

davesiknow 2mo ago

[dead]

daun_gee 2mo ago

[flagged]

Kattywumpus 2mo ago

I don't know. Upvotes often do the same thing, especially in siloed communities. I feel like this is a larger problem -- people leaning into extremifying their views or just playing to an audience -- even though one might frame it as "prosocial" because it happens within the context of an online community.

smcg 2mo ago
Both could be true. People follow incentives.
matheusmoreira 2mo ago
> Upvotes often do the same thing, especially in siloed communities. Yeah. I've gotten caught in this pattern here on HN. Post something, there's some disagreement but then I see the quiet upvotes trickling in, sometimes by the dozen. That sort of signal slowly convinced me I was right about plenty of stuff. I used AI to datamine all of my HN comments and find those patterns. Then I started asking the AI to steelman the hell out of my world view, every single argument.
- gryfft 2mo ago
  How would you say your views have changed since then?
  - matheusmoreira 2mo ago
    Biggest change was brazilian vs american laws. I had more knowledge of and respect for US laws than those of my own country, and Claude essentially fixed that. I started studying the law with Claude whenever I'd get into some argument involving copyright, and it started picking up on this pattern and showing me counterpoints where brazilian law precisely defined things that americans had to pay lawyers to go to court and get a judge to create precedent for. AI measurably lowered the number of absolute claims I've been making. Tended to generalize and radicalize too much which would invite people to respond to the tone and the extremism instead of disproving the underlying idea. That sort of reply would actually reinforce my beliefs: if I was wrong, people would simply refute it. AI attacked my points with counterexamples until I was able to refine them into a core that's actually true. These little distilled kernels of truth were the most valuable results. I found that many of my arguments actually have an underlying idea that holds up under scrutiny, but nobody cares because I try to prove it via extreme arguments. One such idea is anti-impossibility: laws and rules are all fine, but making it structurally impossible for people to disobey them is unacceptable. I used to try to argue that idea by defending running red lights which persuaded no one. Now I've identified the real idea and can make a much better argument for it.

donatj 2mo ago

> This suggests that people are drawn to AI that unquestioningly validate, even as that validation risks eroding their judgment I believe this is a larger problem than just AI. The internet has helped people surround themselves with only voices that agree with them and validate them, often to their detriment. There are entire online communities urging people to cut others out of their lives over the slightest disagreement.

deleted 2mo ago
[deleted]
darkmarmot 2mo ago
It's like it turns everyone into a CEO with all the yes men they could ever ask for! Never goes wrong!
overgard 2mo ago
Yeah, a lot of it isn't even really about people's choice though. Like, I have so many click-baity things in my YouTube feed even though I ignore almost all of it, because just one click and suddenly that's a thing it's going to show me 10x. I wish there was a place I could go for tech news that just didn't talk about AI. I'm so over it.
- jaggederest 2mo ago
  Agreed, I think AI is the bees knees but I already spend 6+ hours a day thinking about it, more is unhelpful. I wish for something with the quality of discourse of HN but without being so monomaniacally focused on tech-du-jour.
- ehnto 2mo ago
  I think what frustrates me about the constant AI talk, is that it's like a global bikeshedding/yak shaving convention. It's an onslaught of details and opinions that don't matter that much. People arguing about single percentile benchmark differences like it will make or break their next project. I get it, we can use AI agents, and they keep improving. Great, now calm down and go build stuff, and show me the stuff. I really think LLMs and agents are incredible, but I don't want to hear that over and over again for another 5 years. HN is particularly bad for it right now. Where are all the innovations, cool technical side projects etc happening outside of LLMs?

theturtle 2mo ago

[dead]

SwellJoe 2mo ago

I've said before that in one small way, having access to the sycophantic models is like being a billionaire: You never have to hear "no" or "that's not a good idea". And, it has the same deleterious effect on your mental health. I could name a bunch of billionaires that behave like sociopaths, and they're often seemingly miserable while doing it. I'm sure some of them started out as sociopaths, but they rarely behaved that way so obviously early in their rise.

roughly 2mo ago

This feels like one of those articles that’s gonna come to mind again and again over the next couple decades.

romaniitedomum 2mo ago

> This suggests that people are drawn to AI that unquestioningly validate, even as that validation risks eroding their judgment The obvious counterpoint is the large number of complaints about OpenAI's excessively sycophantic models back in April 2026, which led to them rolling back to a previous and less sycophantic model. I think there are those who are susceptible to model sycophancy, but it's definitely not most users. I have a vague, half-formed intuition that many of those susceptible to sycophancy are those who use AI for non-technical, non-work uses, such as inter-personal relationship questions. As for me, as soon as I see any hint of sycophancy in a response I either stop what I'm doing or I tell the model to stop being an idiot.

recursive 2mo ago
People are drawn to nicotine use. And still people complain about smoke.

kazinator 2mo ago

I have an alternative view which is that sycophancy erodes trust in AI, from those who are not seeking validation from a machine, but information or advice. If you're looking for advice in a situation where you are not sure of what the correct choices are, you will find LLM chat AI to go in circles. It says one thing. Something is dodgy about it, so you raise a tentative objection (as a non-expert). The thing does a "you are completely right, I apologize" about face and then says something different, and things have begun to slide into uncertainty. That might not exactly be sycophancy, but it's basically the same thing: producing responses that are reflection of what is in the chat, rather than any real shit. Pick any topic where people disagree. It could be an entirely technical topic in which engineers have settled the questions, and the only contrarians are crackpots. The problem is that the crackpots are out there writing, and this is snarfed into the training data. Crackpots use certain ways of talking about certain subjects. If you use similar vocabulary and concepts that align with some crackpot theory, the AI simply starts predicting tokens according to that, and you are now in crackpot land: what you are saying is validated using the crackpot terms. Next, write in a way that reintroduces rigidity: proper terminology and correct concepts, and, whoa, the AI is an engineer again, contradicting the previous crackpot shit.

seizethecheese 2mo ago
Perhaps only a small minority want quality out of information product, while most want satisfying simplicity. This is definitely how the news plays out. I recently built an ai chart product that attempts to add more nuance by injecting extra models to jump in when the first model misses the mark. (http://pellmell.au http://pellmell.au So far it seems like only a minority of people like it. The main feedback is that answers are confusing or too much to read
- tonyedgecombe 2mo ago
  I learnt a long time ago that most people don't want their problems solved, they just want to vent.
  - layla5alive 2mo ago

alexalx666 2mo ago

any close relationship forms a dependence. If you talk to something that poses as a helpful human 8 hours a day to do your job, you already spending more time with this "person" than with your loved ones

monocola 2mo ago

[flagged]

alexbelyanin 2mo ago

[flagged]

socketcluster 2mo ago

In my case, I need my AI to be sycophantic because most of my contrarian ideas are correct. So it's a waste of my time and tokens when the AI keeps making weak arguments which I proceed to destroy one by one. Usually, what happens is it tries to debunk my statement but then I offer a rebuttal to all of its points, it then concedes to all of my points but each time it offers a new partial rebuttal... Which I proceed to crush... But it keeps coming up with increasingly irrelevant caveats. It goes on for quite some time and at the end I ask it to review our discussion and it admits that its original stance "was not as strong" as it initially claimed but it never fully concedes... Even though I literally destroyed all of its points and every partial rebuttal it tried to come up with... Meanwhile its rebuttals became increasingly nit-picky and distant from the original claims made... It's like if I'm saying "the ship is sinking, look at all the water in the hull and look at all the water pouring in through that hole" and it's like "oh but this is a small hole and the pump can easily offset it" so then I say "What about this hole over here" "Oh but the pump can still offset the stream from both holes easily" and then I say "What about that third one? And fourth one? And fifth one?" And it's like "The pump can easily offset that" and I'm like "Common! The hull is full of water and the water level is rising, the hull is full of holes; it's not a stretch to suggest that the ship is sinking because of all the holes in it... I shouldn't have to point out the location of every single hole for my argument to start making sense!" It's not proof by induction but it's probably as close as you can get to it for a topic which lies outside the realm of mathematics!

hlynurd 2mo ago
Sounds like you want a sycophant because you use words like "crush" and "destroy" to describe counterpoints to your arguments (please don't destroy me.)
- socketcluster 2mo ago
  I used these words jokingly and also, I don't mean sycophant in a literal sense. I mean more in an AI context as a kind of config/slider you can dial up or down to make it more collaborative or combative. If I'm knowledgeable in a niche over which there is no mainstream consensus or it's disorganized, and I want to reason more about this topic, I need the AI in the rabbit hole with me. I don't need it to question the existence of the rabbit hole I'm in. Otherwise all of its arguments sound like gaslighting; asking me to disbelieve my eyes and decades of experience about basic ideas which I've already researched many times and proven not to work (at least with high confidence). For me this happens with some niches of software engineering and architecture and I have lots of physical proof of working, secure, maintainable software to back up my position. Also, when I say that "most of my contrarian ideas are correct" it's with the caveat that I don't tend to adopt or hold onto contrarian ideas that are easily proven to be incorrect. I'm referring to the kinds of niche ideas which most non-experts would not feel confident to make strong arguments about but AI will.

Lutger 2mo ago

I think the conclusions are a bit overstated and lopsided. Regardless of the sycophancy (which you can tune), LLMs have infinite patience and attention to details. They also have very little inherent bias - it just mirrors your own mostly. They are, in those aspects, vastly superior to even the best therapists, who can't listen for even a couple of minutes and can't think outside of their limited frames. They are only human. Try listening to somebody and merely repeating the literal words that they say. I guarantee you, most people can't even reproduce two or tree sentences correctly. And I am not even talking about understanding the words, just the literal reproduction of them, copy-pasta. Humans just can't do it, we are not good at it. Talking to an AI about your problems is a bit like talking to yourself, but with the added superpower of hours of google searches compressed in seconds. Of course it is dangerous and incomplete, but in a way it also beats talking to humans if you know what you are doing and are looking to and able to solve your problems yourself.

altmanaltman 2mo ago
> They are, in those aspects, vastly superior to even the best therapists, who can't listen for even a couple of minutes and can't think outside of their limited frames. You start your comment saying the conclusions are overstated and then just drop this conclusion without any proof except anncedotal prespective.

lowsong 2mo ago

In decades to come we view "chatbot" AI as one of the most dangerous inventions of this century. Once regulation catches up and they are outlawed, we'll look back on this period of history with horror.

rramadass 2mo ago

This is an important paper which everybody should read and then accordingly tune/guard their interactions with AI; especially true when people use AI for personal/psychological support/validation. It will completely distort reality and push people into fantasy land which when mapped to the real world can have disastrous consequences. Sycophancy is the "stickiness factor" of AI analogous to that of Social Media. For some background read Jagged Intelligence: The Dangerous Unknowns at the Heart of LLMs - https://news.ycombinator.com/item?id=48577159 https://news.ycombinator.com/item?id=48577159 Here is an experiment that i did; Ask AI to build a "Character Profile"(in the broadest sense) of a person based on their available public writings. This requires "commonsense reasoning" (https://en.wikipedia.org/wiki/Commonsense_reasoning https://en.wikipedia.org/wiki/Commonsense_reasoning), understanding human motivations and behaviour, context, assumptions, societal knowledge etc. I know "me" and so i asked AI to use my HN comments/submissions as input :-) The sycophancy/flattery/praise was quiet excessive. I am realistic and old enough to not need ego-soothing (a little is fine but a lot makes me suspicious) and so i asked AI whether these phrases were not too over-the-top. It apologized and agreed to drop the fluff. I then asked it to identify job roles (any) for which i might not be a good fit given my character profile. This acts as an external constraint which focuses attention on shortcomings and hence forces the AI to look at the other side of the coin. Now the results were better and more in line with what some Human may deduce from my HN persona (which obviously is not my complete real-life persona) but still wanting in many aspects. The above is a perfect example of "Jagged Intelligence" exhibited by AI. Excellent in formal symbolic manipulation sciences, regurgitation and simple reasoning but highly deficient in human-like commonsense reasoning.

kittikitti 2mo ago

I took this advice personally, where people denounced sycophants, so I stopped affirming them. I became a huge asshole and everyone around me started reacting very, very negatively. My conclusion is that this is a huge distraction and people honestly highly prefer sycophants.

ranger_danger 2mo ago
What makes you think those people are aware of and believe those people to be sycophants in the first place? It could just as easily be that their thinking is much simpler than that and they prefer people who suck up to them, without even thinking about there being a possible ulterior motive.

jtrn 2mo ago

[dead]