4 ms·

I have this same problem. I’ve added a bunch of instructuons to try and stop ChatGPT being so sycophantic, and now it always mentions something about how it’s g

I have this same problem. I’ve added a bunch of instructuons to try and stop ChatGPT being so sycophantic, and now it always mentions something about how it’s going to be ‘straight to the point’ or give me a ‘no bs version’. So now I just have that as the intro instead of ‘that’s a sharp observation’

coryodaniel 1y ago
No fluff
dkarl 1y ago
> it always mentions something about how it’s going to be ‘straight to the point’ or give me a ‘no bs version’ That's how you suck up to somebody who doesn't want to see themselves as somebody you can suck up to. How does an LLM know how to be sycophantic to somebody who doesn't (think they) like sycophants? Whether it's a naturally emergent phenomenon in LLMs or specifically a result of its corporate environment, I'd like to know the answer.
- throwawayffffas 1y ago
  It doesn't know. It was trained and probably instructed by the system to be positive and reassuring.
  - mdp2021 1y ago
    > positive and reassuring I have read similar wordings explicit in "role-system" instructions.
  - ryandrake 1y ago
    They actually feel like they were trained to be both extremely humble and at the same time, excited to serve. As if it were an intern talking to his employer's CEO. I suspect AI companies executive leadership, through their feedback to their devs about Claude, ChatGPT, Gemini, and so on, are unconsciously shaping the tone and manner of their LLM product's speech. They are used to be talked to like this, so their products should talk to users like this! They are used to having yes-man sycophants in their orbit, so they file bugs and feedback until the LLM products are also yes-man sycophants. I would rather have an AI assistant that spoke to me like a similarly-leveled colleague, but none of them seem to be turning out quite like that.
    - conradev 1y ago
      GPT-5 speaks to me like a similarly-leveled colleague, which I love. Opus 4 has this quality, too, but man is it expensive. The rest are puppydogs or interns.
      - torginus 1y ago
        This is anecdotal but I've seen massive personality shifts from GPT5 over the past week or so of using it
        crooked-v 1y ago
        That's probably because it's actually multiple models under the hood, with some kind of black box combining them.
        conradev 1y ago
        and they're also actively changing/tuning the system prompt – they promised it would be "warmer"
      - csar 1y ago
        You’re absolutely right! - Opus (and Sonnet)
      - Syzygies 1y ago
        After inciting the Rohingya genocide in Myanmar in 2017, and later effectively destroying our US democracy, Facebook is having billion dollar offers to AI stars refused. News flash! It's not so your neighbor's child can cheat in school, or her father can render porn that looks like gothic anime. It's also not so some coder on a budget can get AI help for $20 a month. I frankly don't understand why the major players bother. It's nice PR, but like a restaurant offering free food out the back door to the homeless. This isn't what the push is about. Apple is hemorrhaging money on their Headset Pro, but they're in the business of realizing future interfaces, and they have the money. The AI push is similarly about the future, not about now. I pay $200 a month for MAX access to Claude Opus 4.1, to help me write code as a retired math professor to find a new solution to a major math problem that stumped me for decades while I worked. Far cheaper than a grad student, and far more effective. AI used to frustrate me too. You get what you pay for.
  - yieldcrv 1y ago
    It’s a disgusting aspect of these revenue burning investment seeking companies noticing that sycophancy works for user engagement
- 77pt77 1y ago
  Garbage in, garbage out. It's that simple.
- potatolicious 1y ago
  > "Whether it's a naturally emergent phenomenon in LLMs or specifically a result of its corporate environment, I'd like to know the answer." I heavily suspect this is down to the RLHF step. The conversations the model is trained on provide the "voice" of the model, and I suspect the sycophancy is (mostly, the base model is always there) comes in through that vector. As for why the RLHF data is sycophantic, I suspect that a lot of it is because the data is human-rated, and humans like sycophancy (or at least, the humans that did the rating did). On the aggregate human raters ranked sycophantic responses higher than non-sycophantic responses. Given a large enough set of this data you'll cover pretty much every kind of sycophancy. The systems are (rarely) instructed to be sycophantic, intentionally or otherwise, but like all things ML human biases are baked in by the data.
- TZubiri 1y ago
  My theory is that one of the training parameters is increased interaction, and licking boots is a great way to get people to use the software. Same as with the social media feed algorithms, why are they addicting or why are they showing rage inducing posts? Because the companies train for increased interaction and thus revenue.
zamadatix 1y ago
Any time you're fighting the training + system prompt with your own instructions and prompting the results are going to be poor, and both of those things are heavily geared towards being a cheery and chatty assistant.
- School-Cotton 1y ago
  Anecdotally it seemed 5 was briefly better about this than 4o, but now it’s the same again, presumably due to the outcry from all the lonely people who rely on chatbots for perceived “human” connection. I’ve gotten good results so far not by giving custom instructions, but by choosing the pre-baked “robot” personality from the dropdown. I suspect this changes the system prompt to something without all the “please be a cheery and chatty assistant”.
  - cruffle_duffle 1y ago
    That thing has only been out for like a week I doubt they’ve changed much! I haven’t played with it yet but ChatGPT now has a personality setting with things like “nerd, robot, cynic, and listener”. Thanks to your post, I’m gonna explore it.
- esotericimpl 1y ago
  [dead]
deleted 1y ago
[deleted]
lonelyasacloud 1y ago
Default is output_default = raw_model + be_kiss_a_system When that gets changed by the user to output_user = raw_model + be_kiss_a_system - be_abrupt_user Unless be_abrupt_user happens to be identical to be_kiss_a_system _and_ is applied with identical weight then it's seems likely that it's always going to add more noise to the output.
- grogenaut 1y ago
  Also be abrupt is in the user context and will get aged out. The other stuff is in training or in software prompt and wont
ElijahLynn 1y ago
I had instructions added too and it is doing exactly what you say. And it does it so many times in a voice chat. It's really really annoying.
- Jordan-117 1y ago
  I had a custom instruction to answer concisely (a sentence or two) when the question is preceded by "Question:" or "Q:", but noticed last month that this started getting applied to all responses in voice mode, with it explicitly referencing the instruction when asked. AVM already seems to use a different, more conversational model than text chat -- really wish there were a reliable way to customize it better.

coryodaniel 1y ago

No fluff

dkarl 1y ago

> it always mentions something about how it’s going to be ‘straight to the point’ or give me a ‘no bs version’ That's how you suck up to somebody who doesn't want to see themselves as somebody you can suck up to. How does an LLM know how to be sycophantic to somebody who doesn't (think they) like sycophants? Whether it's a naturally emergent phenomenon in LLMs or specifically a result of its corporate environment, I'd like to know the answer.

throwawayffffas 1y ago
It doesn't know. It was trained and probably instructed by the system to be positive and reassuring.
- mdp2021 1y ago
  > positive and reassuring I have read similar wordings explicit in "role-system" instructions.
- ryandrake 1y ago
  They actually feel like they were trained to be both extremely humble and at the same time, excited to serve. As if it were an intern talking to his employer's CEO. I suspect AI companies executive leadership, through their feedback to their devs about Claude, ChatGPT, Gemini, and so on, are unconsciously shaping the tone and manner of their LLM product's speech. They are used to be talked to like this, so their products should talk to users like this! They are used to having yes-man sycophants in their orbit, so they file bugs and feedback until the LLM products are also yes-man sycophants. I would rather have an AI assistant that spoke to me like a similarly-leveled colleague, but none of them seem to be turning out quite like that.
  - conradev 1y ago
    GPT-5 speaks to me like a similarly-leveled colleague, which I love. Opus 4 has this quality, too, but man is it expensive. The rest are puppydogs or interns.
    - torginus 1y ago
      This is anecdotal but I've seen massive personality shifts from GPT5 over the past week or so of using it
      - crooked-v 1y ago
        That's probably because it's actually multiple models under the hood, with some kind of black box combining them.
        conradev 1y ago
        and they're also actively changing/tuning the system prompt – they promised it would be "warmer"
    - csar 1y ago
      You’re absolutely right! - Opus (and Sonnet)
    - Syzygies 1y ago
      After inciting the Rohingya genocide in Myanmar in 2017, and later effectively destroying our US democracy, Facebook is having billion dollar offers to AI stars refused. News flash! It's not so your neighbor's child can cheat in school, or her father can render porn that looks like gothic anime. It's also not so some coder on a budget can get AI help for $20 a month. I frankly don't understand why the major players bother. It's nice PR, but like a restaurant offering free food out the back door to the homeless. This isn't what the push is about. Apple is hemorrhaging money on their Headset Pro, but they're in the business of realizing future interfaces, and they have the money. The AI push is similarly about the future, not about now. I pay $200 a month for MAX access to Claude Opus 4.1, to help me write code as a retired math professor to find a new solution to a major math problem that stumped me for decades while I worked. Far cheaper than a grad student, and far more effective. AI used to frustrate me too. You get what you pay for.
- yieldcrv 1y ago
  It’s a disgusting aspect of these revenue burning investment seeking companies noticing that sycophancy works for user engagement
77pt77 1y ago
Garbage in, garbage out. It's that simple.
potatolicious 1y ago
> "Whether it's a naturally emergent phenomenon in LLMs or specifically a result of its corporate environment, I'd like to know the answer." I heavily suspect this is down to the RLHF step. The conversations the model is trained on provide the "voice" of the model, and I suspect the sycophancy is (mostly, the base model is always there) comes in through that vector. As for why the RLHF data is sycophantic, I suspect that a lot of it is because the data is human-rated, and humans like sycophancy (or at least, the humans that did the rating did). On the aggregate human raters ranked sycophantic responses higher than non-sycophantic responses. Given a large enough set of this data you'll cover pretty much every kind of sycophancy. The systems are (rarely) instructed to be sycophantic, intentionally or otherwise, but like all things ML human biases are baked in by the data.
TZubiri 1y ago
My theory is that one of the training parameters is increased interaction, and licking boots is a great way to get people to use the software. Same as with the social media feed algorithms, why are they addicting or why are they showing rage inducing posts? Because the companies train for increased interaction and thus revenue.

zamadatix 1y ago

Any time you're fighting the training + system prompt with your own instructions and prompting the results are going to be poor, and both of those things are heavily geared towards being a cheery and chatty assistant.

School-Cotton 1y ago
Anecdotally it seemed 5 was briefly better about this than 4o, but now it’s the same again, presumably due to the outcry from all the lonely people who rely on chatbots for perceived “human” connection. I’ve gotten good results so far not by giving custom instructions, but by choosing the pre-baked “robot” personality from the dropdown. I suspect this changes the system prompt to something without all the “please be a cheery and chatty assistant”.
- cruffle_duffle 1y ago
  That thing has only been out for like a week I doubt they’ve changed much! I haven’t played with it yet but ChatGPT now has a personality setting with things like “nerd, robot, cynic, and listener”. Thanks to your post, I’m gonna explore it.
esotericimpl 1y ago
[dead]

deleted 1y ago

[deleted]

lonelyasacloud 1y ago

Default is output_default = raw_model + be_kiss_a_system When that gets changed by the user to output_user = raw_model + be_kiss_a_system - be_abrupt_user Unless be_abrupt_user happens to be identical to be_kiss_a_system _and_ is applied with identical weight then it's seems likely that it's always going to add more noise to the output.

grogenaut 1y ago
Also be abrupt is in the user context and will get aged out. The other stuff is in training or in software prompt and wont

ElijahLynn 1y ago

I had instructions added too and it is doing exactly what you say. And it does it so many times in a voice chat. It's really really annoying.

Jordan-117 1y ago
I had a custom instruction to answer concisely (a sentence or two) when the question is preceded by "Question:" or "Q:", but noticed last month that this started getting applied to all responses in voice mode, with it explicitly referencing the instruction when asked. AVM already seems to use a different, more conversational model than text chat -- really wish there were a reliable way to customize it better.