5 ms·

LLMs as a rule seem to be primed to make the user feel especially smart or gifted, even when they are clearly not. ChatGPT is by far the worst offender in this

by jacquesm 10mo ago

LLMs as a rule seem to be primed to make the user feel especially smart or gifted, even when they are clearly not. ChatGPT is by far the worst offender in this sense but others are definitely not clean.

Ajedi32 10mo ago
They're trained to give responses that get positive ratings from reviewers in post-training. A little flattery probably helps achieve that. Not to mention sycophancy is probably positively correlated with following instructions, the latter usually being an explicit goal of post-training.
- mikepurvis 10mo ago
  I'd be interested to see someone try to untangle the sycophancy/flattery from the modern psych / non-violent communication piece. In theory (so much as I understand it around NVC) the first is outright manipulative and the second is supposed to be about avoiding misunderstandings, but I do wonder how much the two are actually linked. A lot of NVC writing seems to fall into the grey area of like, here's how to communicate in way that will be least likely to trigger or upset the listener, even when the meat of what is being said is in fact unpleasant or embarrassing or confronting to them. How far do you have to go before the indirection associated with empathy-first communication and the OFNR framework start to just look like LLM ego strokes? Where is the line?
  - nielsole 10mo ago
    > indirection Isn't nvc often about communicating explicitly instead of implicitly? So frequently it can be the opposite of an indirection.
    - mikepurvis 10mo ago
      I guess so? I'm not well-versed, but the basics are usually around observation and validation of feelings, so instead of "you took steps a, b, c, which would normally be the correct course of action, but in this instance (b) caused side-effect (d) which triggered these further issues e and f", it's something more like "I can understand how you were feeling overwhelmed and under pressure and that led you to a, b, c ..." Maybe this is an unhelpful toy example, but for myself I would be frustrated to be on either side of the second interaction. Like, don't waste everyone's time giving me excuses for my screwup so that my ego is soothed, let's just talk about it plainly, and the faster we can move on to identifying concrete fixes to process or documentation that will prevent this in the future, the better.
  - cycomanic 10mo ago
    I think the difference between sycophancy and NVC (at least how I learned it) is that a sycophantic person just uncritically agrees with you, but NVC is about how to communicate disagreement, so the other person actually listen to your argument instead of adopting a reflexive defense response.
    - Ajedi32 10mo ago
      I think the problem is that telling someone they're wrong without hurting their ego is a very difficult skill to learn. And even if you're really good at it, you'll still often fail because sometimes people just don't want to be disagreed with regardless of how you phrase it. It's far easier for the AI to learn to be a sycophant instead (or on the opposite side of the spectrum, to learn to just not care about hurting people's feelings).
  - CuriousSkeptic 10mo ago
    A lot of NVC writing is pretty bad. I recommend going directly to the source https://youtu.be/l7TONauJGfc https://youtu.be/l7TONauJGfc (3h video, but worth the time) I think NVC is better understood as a framework to reach deep non-judging empathic understanding than a speech pattern. If you are not really engaging in curious exploration of the other party using the OFNR framework before trying to deliver your own request I don’t think you can really call it NVC. At the very least it will be very hard to get your point across even with OFNR if ot validating the receiver. Validation being another word needing disambiguation I suppose. I see it as the act of expressing non-judging emphatic understanding. Using the OFNR framework with active listening can be a great approach. A similar framework is the evaporating clouds of Theory of Constraints: https://en.wikipedia.org/wiki/Evaporating_cloud https://en.wikipedia.org/wiki/Evaporating_cloud Also see Kants categorical imperative: moral actions must be based on principles that respect the dignity and autonomy of all individuals, rather than personal desires or outcomes
- ascorbic 10mo ago
  As people become more familiar with (and annoyed by) LLMs' tone, I wonder if future RLHF reviewers will stop choosing the sycophantic responses.
fzeindl 10mo ago
LLMs sometimes remind me of american car salesmen. Was the hopeful "anything is possible" mentality of the american dream accidentally baked into the larger models?
Hikikomori 10mo ago
You're absolutely right.
amluto 10mo ago
I would pay an extra tiny bit for the LLM to stop telling me how brilliant my idea was when I ask it questions. (Getting complemented on my brilliance is not in any respect indicative of a particular idea being useful, as should be obvious to anyone who uses these tools for more than two minutes. Imagine is a hammer said “great whack!” 60% of the time you hit a nail even if you’re wildly off axis. You’d get a new hammer than would stop commenting, I hope.) Heck, I can literally prompt Claude to read text and “Do not comment on the text” and it will still insert cute Emoji in the text. All of this is getting old.
- iwontberude 10mo ago
  It’s a free hammer, I’m certainly not stupid enough to pay money for it, I’ll throw it away when I’m done with it or when it stops being free.
- nielsole 10mo ago
  The surprising thing for me was how long it took to get old. I got a reward(and then immediate regret upon reflection) for way too long.
- shagie 10mo ago
  On ChatGPT... Personalization: Base style: Efficient Characteristics: Warm: less Enthusiastic: less Headers & Lists: default Emoji: less Custom: Not chatty. Unbiased. Avoid use of emoji. Rather than "Let me know if..." style continuations, list a set of prompts to explore further topics. Do not start out with short sentences or smalltalk that does not meaningfully advance the response. If there is ambiguity that needs to be resolved before an answer can be given, identify that ambiguity before proceeding. --- I believe the bit in the prompt "[d]o not start out with short sentences or smalltalk that does not meaningfully advance the response." is the key part to not have it start off with such text (scrolling back through my old chats, I can see the "Great question" leads in responses... and that's what prompted me to stop that particular style of response).
- sqrt_1 10mo ago
  John Carmack likes how Grok will tell him he is wrong. "I appreciate how Grok doesn’t sugar coat corrections" https://x.com/ID_AA_Carmack/status/1985784337816555744 https://x.com/ID_AA_Carmack/status/1985784337816555744
cycomanic 10mo ago
Maybe that was necessary to get it passed their CEO...?
- jacquesm 10mo ago
  I think LLMs reflect the personality of their creators.

Ajedi32 10mo ago

They're trained to give responses that get positive ratings from reviewers in post-training. A little flattery probably helps achieve that. Not to mention sycophancy is probably positively correlated with following instructions, the latter usually being an explicit goal of post-training.

mikepurvis 10mo ago
I'd be interested to see someone try to untangle the sycophancy/flattery from the modern psych / non-violent communication piece. In theory (so much as I understand it around NVC) the first is outright manipulative and the second is supposed to be about avoiding misunderstandings, but I do wonder how much the two are actually linked. A lot of NVC writing seems to fall into the grey area of like, here's how to communicate in way that will be least likely to trigger or upset the listener, even when the meat of what is being said is in fact unpleasant or embarrassing or confronting to them. How far do you have to go before the indirection associated with empathy-first communication and the OFNR framework start to just look like LLM ego strokes? Where is the line?
- nielsole 10mo ago
  > indirection Isn't nvc often about communicating explicitly instead of implicitly? So frequently it can be the opposite of an indirection.
  - mikepurvis 10mo ago
    I guess so? I'm not well-versed, but the basics are usually around observation and validation of feelings, so instead of "you took steps a, b, c, which would normally be the correct course of action, but in this instance (b) caused side-effect (d) which triggered these further issues e and f", it's something more like "I can understand how you were feeling overwhelmed and under pressure and that led you to a, b, c ..." Maybe this is an unhelpful toy example, but for myself I would be frustrated to be on either side of the second interaction. Like, don't waste everyone's time giving me excuses for my screwup so that my ego is soothed, let's just talk about it plainly, and the faster we can move on to identifying concrete fixes to process or documentation that will prevent this in the future, the better.
- cycomanic 10mo ago
  I think the difference between sycophancy and NVC (at least how I learned it) is that a sycophantic person just uncritically agrees with you, but NVC is about how to communicate disagreement, so the other person actually listen to your argument instead of adopting a reflexive defense response.
  - Ajedi32 10mo ago
    I think the problem is that telling someone they're wrong without hurting their ego is a very difficult skill to learn. And even if you're really good at it, you'll still often fail because sometimes people just don't want to be disagreed with regardless of how you phrase it. It's far easier for the AI to learn to be a sycophant instead (or on the opposite side of the spectrum, to learn to just not care about hurting people's feelings).
- CuriousSkeptic 10mo ago
  A lot of NVC writing is pretty bad. I recommend going directly to the source https://youtu.be/l7TONauJGfc https://youtu.be/l7TONauJGfc (3h video, but worth the time) I think NVC is better understood as a framework to reach deep non-judging empathic understanding than a speech pattern. If you are not really engaging in curious exploration of the other party using the OFNR framework before trying to deliver your own request I don’t think you can really call it NVC. At the very least it will be very hard to get your point across even with OFNR if ot validating the receiver. Validation being another word needing disambiguation I suppose. I see it as the act of expressing non-judging emphatic understanding. Using the OFNR framework with active listening can be a great approach. A similar framework is the evaporating clouds of Theory of Constraints: https://en.wikipedia.org/wiki/Evaporating_cloud https://en.wikipedia.org/wiki/Evaporating_cloud Also see Kants categorical imperative: moral actions must be based on principles that respect the dignity and autonomy of all individuals, rather than personal desires or outcomes
ascorbic 10mo ago
As people become more familiar with (and annoyed by) LLMs' tone, I wonder if future RLHF reviewers will stop choosing the sycophantic responses.

fzeindl 10mo ago

LLMs sometimes remind me of american car salesmen. Was the hopeful "anything is possible" mentality of the american dream accidentally baked into the larger models?

Hikikomori 10mo ago

You're absolutely right.

amluto 10mo ago

I would pay an extra tiny bit for the LLM to stop telling me how brilliant my idea was when I ask it questions. (Getting complemented on my brilliance is not in any respect indicative of a particular idea being useful, as should be obvious to anyone who uses these tools for more than two minutes. Imagine is a hammer said “great whack!” 60% of the time you hit a nail even if you’re wildly off axis. You’d get a new hammer than would stop commenting, I hope.) Heck, I can literally prompt Claude to read text and “Do not comment on the text” and it will still insert cute Emoji in the text. All of this is getting old.

iwontberude 10mo ago
It’s a free hammer, I’m certainly not stupid enough to pay money for it, I’ll throw it away when I’m done with it or when it stops being free.
nielsole 10mo ago
The surprising thing for me was how long it took to get old. I got a reward(and then immediate regret upon reflection) for way too long.
shagie 10mo ago
On ChatGPT... Personalization: Base style: Efficient Characteristics: Warm: less Enthusiastic: less Headers & Lists: default Emoji: less Custom: Not chatty. Unbiased. Avoid use of emoji. Rather than "Let me know if..." style continuations, list a set of prompts to explore further topics. Do not start out with short sentences or smalltalk that does not meaningfully advance the response. If there is ambiguity that needs to be resolved before an answer can be given, identify that ambiguity before proceeding. --- I believe the bit in the prompt "[d]o not start out with short sentences or smalltalk that does not meaningfully advance the response." is the key part to not have it start off with such text (scrolling back through my old chats, I can see the "Great question" leads in responses... and that's what prompted me to stop that particular style of response).
sqrt_1 10mo ago
John Carmack likes how Grok will tell him he is wrong. "I appreciate how Grok doesn’t sugar coat corrections" https://x.com/ID_AA_Carmack/status/1985784337816555744 https://x.com/ID_AA_Carmack/status/1985784337816555744

cycomanic 10mo ago

Maybe that was necessary to get it passed their CEO...?

jacquesm 10mo ago
I think LLMs reflect the personality of their creators.