7 ms·

AI sycophancy panic

57 points by firasd 9mo ago

firasd 9mo ago
This is my attempt to articulate why some recent shifts in AI discourse seem to be degrading the product experience of everyday conversation. I argue that “sycophancy” has become an overloaded and not very helpful term; almost a fashionable label applied to a wide range of unrelated complaints (tone, feedback depth, conversational flow). Curious whether this resonates with how you feel or if you disagree Also see the broader Vibesbench project: https://github.com/firasd/vibesbench/ https://github.com/firasd/vibesbench/ Vibesbench discord: https://discord.gg/5K4EqWpp https://discord.gg/5K4EqWpp
- rvnx 9mo ago
  It's because people are so happy that they learnt a new word that they try to say it to everyone in every occasion. What drives me crazy are the emojis and the patronizing at the end of conversation. Before 2022 no-one was using that word
  - bananaflag 9mo ago
    Same with "stochastic parrot", I doubt lots of people knew the word "stochastic" before
- _alternator_ 9mo ago
  The issue raised here seems mostly semantic, in the sense that the concern is about the mismatch between the standard meaning of a word (sycophant) and its meaning as applied to an issue with LLMs. It seems to me that the issue it refers to (unwarranted or obsequious praise) is a real problem with modern chatbots. The harms range from minor (annoyance, or running down the wrong path because I didn’t have a good idea to start with) to dangerous (reinforcing paranoia and psychotic thoughts). Do you agree that these are problems, and there a more useful term or categorization for these issues?
  - A4ET8a8uTh0_v2 9mo ago
    << The harms range from minor (annoyance, or running down the wrong path because I didn’t have a good idea to start with) to dangerous (reinforcing paranoia and psychotic thoughts). Do you agree that these are problems, and there a more useful term or categorization for these issues? I think that the issue is a little more nuanced. The problems you mentioned are problems of sort, but the 'solution' in place kneecaps one of the ways llms ( as offered by various companies ) were useful. You mention the problem is reinforcement of the bad tendencies, but no indication of reinforcement of good ones. In short, I posit that the harms should not outweigh the benefits of augmentation. Because this is the way it actually does appear to work: 1. dumb people get dumber 2. smart people get smarter 3. psychopaths get more psychopathy I think there is a way forward here that does not have to include neutering seemingly useful tech.
  - firasd 9mo ago
    Re: minor outcomes. It really depends on the example I guess. But if the user types "What if Starbucks focuses on lemonade" and then gets disappointed that the AI didn't yell at them for being off track--what are they expecting exactly? The attempt to satisfy them has led to GPT-5.2-Thinking style nitpicking[1] They have to think of the stress test angles themselves ('can we look up how much they are selling as far as non-warm beverages...') [1] eg. when I said Ian Malcolm in Jurassic Park is a self-insert, it clarified to me "Malcolm is less a “self-insert” in the fanfic sense (author imagining himself in the story) and more Crichton’s designated mouthpiece". Completely irrelevant to my point but answering as if a bunch of reviewers are gonna quibble with its output With regards to mental health issues, of course nobody on Earth (not even the patients with these issues, in their moments of grounded reflection) would say that that the AI should agree with their take. But I also think we need to be careful about what's called "ecological validity". Unfortunately I suspect there may be a lot of LARPing in prompts testing for delusions akin to Hollywood pattern matching, aesthetic talk etc. I think if someone says that people are coming after them the model should not help them build a grand scenario, we can all agree with that. Sycophancy is not exactly the concern there is it? It's more like knowing that this may be a false theory. So it ties into reasoning and contextual fluency (which anti-'sycophancy' tuning may reduce!) and mental health guardrails
- ____mr____ 9mo ago
  AI sycophancy is a real issue and having an AI affirm the user in all/most cases has already led to a murder-suicide[0]. If we want AI chatbots to be "reasonable" conversation participants or even something you can bounce ideas off of, they need to not tell you everything you suggest is a good idea and affirm your every insecurity or neurosis. 0. https://www.aljazeera.com/economy/2025/12/11/openai-sued-for-allegedly-enabling-murder-suicide https://www.aljazeera.com/economy/2025/12/11/openai-sued-for...
- florkbork 9mo ago
  Did you actually argue this? Or did you place about 2-5 paragraphs per heading, with little connection between the ideas? For example: > Perhaps what some users are trying to express with concerns about ‘sycophancy’ is that when they paste information, they'd like to see the AI examine various implications rather than provide an affirming summary. Did you, you personally, find any evidence of this? Or evidence to the opposite? Or is this just a wild guess? Wait; nevermind that we're already moving on! No need to do anything supportive or similar to bolster. > If so, anti-‘sycophancy’ tuning is ironically a counterproductive response and may result in more terse or less fluent responses. Exploring a topic is an inherently dialogic endeavor. Is it? Evidence? Counter evidence? Or is this simply feelpinion so no one can tell you your feelings are wrong? Or wait; that's "vibes" now! I put it to you that you are stringing together (to an outside observer using AI) a series of words in a consecutive order that feels roughly good but lacks any kind of fundamental/logical basis. I put it to you that if your premise is that AI leads to a robust discussion with a back and forth; the one you had that resulted in "product" was severely lacking in any real challenge to your prompts, suggestions, input or viewpoints. I invite you to show me one shred of dialogue where the AI called you out for lacking substance, credibility, authority, research, due dilligence or similar. I strongly suspect you can't. Given that; do you perhaps consider that might be the problem when people label AI responses as sycophancy?
  - firasd 9mo ago
    Well I do have a chat log somewhere where I say potential energy seems like a fake concept and GPT and/or Gemini got around to explaining that it can actually be expressed in equations reliably.. does that count? "called you out for lacking substance, credibility, authority, research, due dilligence or similar" seems like level of emotional angst that LLMs don't usually tend to show Actually amusingly enough the Gemini/Verhoeven example in my doc is an example where the AIs seem to have a memorably strong opinion
delichon 9mo ago
> Ironically, users who are extremely put off by conversational expressions from LLMs are just as vibe-sensitive as anyone else, if not more so. These are preferences regarding style and affect, expressed using the loaded term ‘sycophancy’. It's not just about style. These expressions are information-free noise that distract me from the signal, and I'm paying for them by the token. So I added a system message to the effect that I don't want any compliments, throat clearing, social butter, etc., just the bare facts as straightforward as possible. So then the chatbot started leading every response with a statement to the effect that "here are the bare straightforward facts without the pleasantries", and ending them with something like "those are the straightforward facts without any pleasantries." If I add instructions to stop that, it just paraphrases those instructions at the top and bottom and. will. not. stop. Anyone have a better system prompt for that?
- jchanimal 9mo ago
  I got mine to bundle all that bs into a one word suffix "DISCLAIMER." which it puts at the end of responses now but basically doesn't bother me with that stuff.
  - qaboutthat 9mo ago
    Can you share that part of your prompt? Initial attempts are failing for me, but that would be a great outcome.
- empressplay 9mo ago
  'Answer in the manner of a German army surgeon'
- andy99 9mo ago
  The “sycophantic sugar” part - “that’s exactly right”, “what an insightful observation” etc. is the most outwardly annoying part of the problem but is only part of it. The bigger issue is the real sycophancy, going along with the user’s premise even when it’s questionable or wrong. I don’t see it as just being uncritical - there’s something more to it than that, like reinforcement. It’s one thing to not question what you write, it’s another to subtly play it back as if you’re onto something smart or doing great work. There are tons of extant examples now of people using LLMs that think they’ve some something smart or produced something of value, that haven’t, and the reinforcement they get is a big reason for this.
AlexDragusin 9mo ago
My system prompt that works great, you can apply minor adjustments on case by case basis, good start: > Write in textbook style prose, without headings, no tables, no emojis.
softwaredoug 9mo ago
I recently realized every hypothesis I tested with an LLM, the LLM agreed with me. And if I wasn't careful about reading its caveats, I could leave thinking my idea was brilliant and would never get pushback. I tried something in the political realm. Asking to test a hypothesis and its opposite > Test this hypothesis: the far right in US politics mirrors late 19th century Victorianism as a cultural force compared to > Test this hypothesis: The left in US politics mirrors late 19th century Victorianism as a cultural force An LLM wants to agree with both, it created plausible arguments for both. While giving "caveats" instead of counterarguments. If I had my brain off, I might leave with some sense of "this hypothesis is correct". Now I'm not saying this makes LLMs useless. But the LLM didn't act like a human that might tell you your full of shit. It WANTED my hypothesis to be true and constructed a plausible argument for both. Even with prompting to act like a college professor critiquing a grad student, eventually it devolves back to "helpful / sycophantic". What I HAVE found useful is to give a list of mutually exclusive hypothesis and get probability ratings for each. Then it doesn't look like you want one / other. When the outcome matters, you realize research / hypothesis testing with LLMs is far more of a skill than just dumping a question to an LLM.
- firasd 9mo ago
  But this is the thing I'm pointing out. The idea that the LLM is an oracle or at least a stable subjective view holder is a mistake. As humans, WE have to explore the latent space of the model. We have to activate neurons. We have to say maybe the puritanism of the left ... maybe the puritanism of the right.. okay how about... We are privileged--and doomed--to have to think for ourselves alas
- papandada 9mo ago
  Admittedly I'm not an expert on 19th century Victorianism nor US politics, but this anecdote doesn't make me think any less of LLM's in the least, and I don't see why I should expect different behaviour than what you describe, especially on complex topics
  - Rastonbury 9mo ago
    When I ask for an evaluation or issue I want the LLM to tell me when and how I'm wrong and me not having to worry about phrasing before I take the LLMs word and update my own beliefs/knowledge (especially for subjects I have zero knowledge about), I'm aware of this so when it isn't throw away queries, I tend to ask multiple times and ask for strawmans. I am aware they do this, but with the number of people walking around quoting 'chatgpt said' in real life and on forums, I don't think many people bother to stress test or are aware they phrasing may induce biased responses. It's akin to reading the news only from one source By now I have somewhat stopped relying on LLMs for point of view on latest academic stuff. I don't believe LLMs are able to evaluate paradigm shifting new studies against their massive training corpus. Thinking traces filled with 'tried to open this study, but it's paywalled, I'll use another' does not fill me with confidence that it can articulate a 2025 scientific consensus well. Based on how they work this definitely isn't an easy fix!
avalys 9mo ago
"The anti-sycophancy turn seems to mask a category error about what level of prophetic clarity an LLM can offer. No amount of persona tuning for skepticism will provide epistemic certainty about whether a business idea will work out, whether to add a line to your poem, or why a great movie flopped." What a lot of people actually want from an LLM, is for the LLM to have an opinion about the question being asked. The cool thing about LLMs is that they appear capable of doing this - rather than a machine that just regurgitates black-and-white facts, they seem to be capable of dealing with nuance and gray areas, providing insight, and using logic to reach a conclusion from ambiguous data. But this is the biggest misconception and flaw of LLMs. LLMs do not have opinions. That is not how they work. At best, they simulate what a reasonable answer from a person capable of having an opinion might be - without any consistency around what that opinion is, because it is simply a manifestation of sampling a probability distribution, not the result of logic. And what most people call sycophancy is that, as a result of this statistical construction, the LLM tends to reinforce the opinions, biases, or even factual errors, that it picks up on in the prompt or conversation history.
- paulddraper 9mo ago
  > At best, they simulate what a reasonable answer from a person capable of having an opinion might be And how would you compare that to human thoughts? “A submarine doesn’t actually swim” Okay what does it do then
  - syntheticcdo 9mo ago
    But we didn't name them "artificial swimmers". We called them submarines - because there is a difference between human beings and machines.
    - senko 9mo ago
      But we did call computers "computers", even though "computers" used to refer to the human computers doing the same computing jobs.
casey2 9mo ago
AI bros complaining about loaded terms like slop and sycophancy when they still use terms like "intelligence", "learning", "reasoning", "attention" and all the words derived from "neuro" to describe a computer program. Don't make sycophantic slop generators and people will stop calling them that
transcriptase 9mo ago
I still suspect what happened was when the midwits all got access to ChatGPT etc and started participating in the A/B tests, they strongly selected for responses that agreed with them regardless of whether they were actually correct. Some of us want to be told when and why we’re wrong, and somewhere along the way AI models were either intentionally or unintentionally guided away from doing it because it improved satisfaction or engagement metrics. We already know from decades of studies that people prefer information that confirms their existing beliefs, so when you present 2 options with a “Which answer do you prefer?” selection, it’s not hard to see how the one that begins with “You’re absolutely right!” wins out.
port3000 9mo ago
I had completely forgotten about 'Sydney' and its emoji-laden diatribes. What a crazy moment, looking back.
devin 9mo ago
My observation is that some of the time what people are picking up on is that a conversation is not the interface they want for examining their problem.
leftouterjoins 9mo ago
At least with Opus 4.5 that magic phrase that has worked for me for it to understand something that has happened since it was trained is to explain the situation prefaced with "This happened after your knowledge cutoff" It understands this and often will spawn web searches to fill itself in on the matter.
- jeffbee 9mo ago
  "Knowledge cutoff" may not apply to all systems, and in the near future I doubt it will apply to any popular system because the cutoff is such a harmful antipattern. I've noticed in the last half year or so more of the best models are able to incorporate recent events without being explicitly told to do so.
damjon 9mo ago
.
- jeffbee 9mo ago
  I have standing orders for Gemini to reserve extreme praise for exceptional circumstances, to never use the words "masterclass" unless literally warranted, and to exercise more range of judgements rather than compressing all possible judgements in between "interesting" and "brilliant". This seems to have corrected a few of its habits.
vjerancrnjak 9mo ago
I catch myself sometimes writing the model it is confused and it should just assume what I am writing is true and continue reasoning from there. Sometimes I am actually right but sometimes I am not. Not sure what happens to any future RL and does it lean more to constantly assuming what is written as true but then has to wiggle out of it.
sz4kerto 9mo ago
Recently I've been using AI for some stuff that service providers don't want it to be used for (specifically: medical diagnosis). I found that Grok (4.1) is superior to most of the others when it comes to this, because it doesn't go out of its way to support my own hypotheses. I believe that syncophancy and guardrails will be major differentiators between LLM services, and the ones with less of those will always have a fan base.
djaouen 9mo ago
> For the purpose of conversation, what the model appears to ‘believe’ probably doesn’t matter This is wrong to the point of being absurd. What the model "appears to 'believe'" does matter, and the model's "beliefs" about humans and society at large have vast implications for humanity's future.
pfisch 9mo ago
It is a real problem that AI's will basically confirm that most inquiries are true. Just by asking a leading question often results in the AI confirming it is true or stretching reality to accommodate the answer being true. If I ask if a drug has a specific side effect and the answer is no it should say no. Not try to find a way to say yes that isn't really backed by evidence. People don't realize that when they ask a leading question that is really specific in a way where no one has a real answer then the AI will try to find a way to agree, and this is going to destroy people's lives. Honestly it already has.
ccppurcell 9mo ago
I just asked chatgpt about 72 hour preparedness kit. I said what I had already and asked it about the best way to store it first. It said something to the effect of "good, you are thinking about this in exactly the right way -- make sure you have the containers in place and then get the exact contents right later" not exactly the wording but you get the picture and can probably guess where I'm going with this. When I asked again, this time I asked about the items first. I had to prompt it with something like "or do you think I should get the storage sorted first" and it said "you are thinking about this in exactly the right way -- preparedness kits fail more often due to missing essentials than sub optimal storage" I can't decide which of these is right! Maybe there's an argument that it doesn't matter, and getting started is the most important thing, and so being encouraging is generally the best strategy here. But it's definitely worrying to me. It pretty much always says something like this to me (this is on the "honest and direct" personality setting or whatever).
cadamsdotcom 9mo ago
Another sign that AGI is still very far away.
TacticalCoder 9mo ago
"Dear LLM, TFA reads quite bit pompous to me" "You're absolutely right, what a great observation" ; )
diamond559 9mo ago
If you still unironically call llm's "AI" then you have no right calling out the obvious sycophancy of your product bro. He also ignores the real sycophancy in llm's reinforcing mental illness/delusions and focuses on semantic choices of the outputs. This post is drivel.

57 points by firasd 9mo ago

This is my attempt to articulate why some recent shifts in AI discourse seem to be degrading the product experience of everyday conversation. I argue that “sycophancy” has become an overloaded and not very helpful term; almost a fashionable label applied to a wide range of unrelated complaints (tone, feedback depth, conversational flow). Curious whether this resonates with how you feel or if you disagree Also see the broader Vibesbench project: https://github.com/firasd/vibesbench/ https://github.com/firasd/vibesbench/ Vibesbench discord: https://discord.gg/5K4EqWpp https://discord.gg/5K4EqWpp

rvnx 9mo ago
It's because people are so happy that they learnt a new word that they try to say it to everyone in every occasion. What drives me crazy are the emojis and the patronizing at the end of conversation. Before 2022 no-one was using that word
- bananaflag 9mo ago
  Same with "stochastic parrot", I doubt lots of people knew the word "stochastic" before
_alternator_ 9mo ago
The issue raised here seems mostly semantic, in the sense that the concern is about the mismatch between the standard meaning of a word (sycophant) and its meaning as applied to an issue with LLMs. It seems to me that the issue it refers to (unwarranted or obsequious praise) is a real problem with modern chatbots. The harms range from minor (annoyance, or running down the wrong path because I didn’t have a good idea to start with) to dangerous (reinforcing paranoia and psychotic thoughts). Do you agree that these are problems, and there a more useful term or categorization for these issues?
- A4ET8a8uTh0_v2 9mo ago
  << The harms range from minor (annoyance, or running down the wrong path because I didn’t have a good idea to start with) to dangerous (reinforcing paranoia and psychotic thoughts). Do you agree that these are problems, and there a more useful term or categorization for these issues? I think that the issue is a little more nuanced. The problems you mentioned are problems of sort, but the 'solution' in place kneecaps one of the ways llms ( as offered by various companies ) were useful. You mention the problem is reinforcement of the bad tendencies, but no indication of reinforcement of good ones. In short, I posit that the harms should not outweigh the benefits of augmentation. Because this is the way it actually does appear to work: 1. dumb people get dumber 2. smart people get smarter 3. psychopaths get more psychopathy I think there is a way forward here that does not have to include neutering seemingly useful tech.
- firasd 9mo ago
  Re: minor outcomes. It really depends on the example I guess. But if the user types "What if Starbucks focuses on lemonade" and then gets disappointed that the AI didn't yell at them for being off track--what are they expecting exactly? The attempt to satisfy them has led to GPT-5.2-Thinking style nitpicking[1] They have to think of the stress test angles themselves ('can we look up how much they are selling as far as non-warm beverages...') [1] eg. when I said Ian Malcolm in Jurassic Park is a self-insert, it clarified to me "Malcolm is less a “self-insert” in the fanfic sense (author imagining himself in the story) and more Crichton’s designated mouthpiece". Completely irrelevant to my point but answering as if a bunch of reviewers are gonna quibble with its output With regards to mental health issues, of course nobody on Earth (not even the patients with these issues, in their moments of grounded reflection) would say that that the AI should agree with their take. But I also think we need to be careful about what's called "ecological validity". Unfortunately I suspect there may be a lot of LARPing in prompts testing for delusions akin to Hollywood pattern matching, aesthetic talk etc. I think if someone says that people are coming after them the model should not help them build a grand scenario, we can all agree with that. Sycophancy is not exactly the concern there is it? It's more like knowing that this may be a false theory. So it ties into reasoning and contextual fluency (which anti-'sycophancy' tuning may reduce!) and mental health guardrails
____mr____ 9mo ago
AI sycophancy is a real issue and having an AI affirm the user in all/most cases has already led to a murder-suicide[0]. If we want AI chatbots to be "reasonable" conversation participants or even something you can bounce ideas off of, they need to not tell you everything you suggest is a good idea and affirm your every insecurity or neurosis. 0. https://www.aljazeera.com/economy/2025/12/11/openai-sued-for-allegedly-enabling-murder-suicide https://www.aljazeera.com/economy/2025/12/11/openai-sued-for...
florkbork 9mo ago
Did you actually argue this? Or did you place about 2-5 paragraphs per heading, with little connection between the ideas? For example: > Perhaps what some users are trying to express with concerns about ‘sycophancy’ is that when they paste information, they'd like to see the AI examine various implications rather than provide an affirming summary. Did you, you personally, find any evidence of this? Or evidence to the opposite? Or is this just a wild guess? Wait; nevermind that we're already moving on! No need to do anything supportive or similar to bolster. > If so, anti-‘sycophancy’ tuning is ironically a counterproductive response and may result in more terse or less fluent responses. Exploring a topic is an inherently dialogic endeavor. Is it? Evidence? Counter evidence? Or is this simply feelpinion so no one can tell you your feelings are wrong? Or wait; that's "vibes" now! I put it to you that you are stringing together (to an outside observer using AI) a series of words in a consecutive order that feels roughly good but lacks any kind of fundamental/logical basis. I put it to you that if your premise is that AI leads to a robust discussion with a back and forth; the one you had that resulted in "product" was severely lacking in any real challenge to your prompts, suggestions, input or viewpoints. I invite you to show me one shred of dialogue where the AI called you out for lacking substance, credibility, authority, research, due dilligence or similar. I strongly suspect you can't. Given that; do you perhaps consider that might be the problem when people label AI responses as sycophancy?
- firasd 9mo ago
  Well I do have a chat log somewhere where I say potential energy seems like a fake concept and GPT and/or Gemini got around to explaining that it can actually be expressed in equations reliably.. does that count? "called you out for lacking substance, credibility, authority, research, due dilligence or similar" seems like level of emotional angst that LLMs don't usually tend to show Actually amusingly enough the Gemini/Verhoeven example in my doc is an example where the AIs seem to have a memorably strong opinion

delichon 9mo ago

> Ironically, users who are extremely put off by conversational expressions from LLMs are just as vibe-sensitive as anyone else, if not more so. These are preferences regarding style and affect, expressed using the loaded term ‘sycophancy’. It's not just about style. These expressions are information-free noise that distract me from the signal, and I'm paying for them by the token. So I added a system message to the effect that I don't want any compliments, throat clearing, social butter, etc., just the bare facts as straightforward as possible. So then the chatbot started leading every response with a statement to the effect that "here are the bare straightforward facts without the pleasantries", and ending them with something like "those are the straightforward facts without any pleasantries." If I add instructions to stop that, it just paraphrases those instructions at the top and bottom and. will. not. stop. Anyone have a better system prompt for that?

jchanimal 9mo ago
I got mine to bundle all that bs into a one word suffix "DISCLAIMER." which it puts at the end of responses now but basically doesn't bother me with that stuff.
- qaboutthat 9mo ago
  Can you share that part of your prompt? Initial attempts are failing for me, but that would be a great outcome.
empressplay 9mo ago
'Answer in the manner of a German army surgeon'
andy99 9mo ago
The “sycophantic sugar” part - “that’s exactly right”, “what an insightful observation” etc. is the most outwardly annoying part of the problem but is only part of it. The bigger issue is the real sycophancy, going along with the user’s premise even when it’s questionable or wrong. I don’t see it as just being uncritical - there’s something more to it than that, like reinforcement. It’s one thing to not question what you write, it’s another to subtly play it back as if you’re onto something smart or doing great work. There are tons of extant examples now of people using LLMs that think they’ve some something smart or produced something of value, that haven’t, and the reinforcement they get is a big reason for this.

AlexDragusin 9mo ago

My system prompt that works great, you can apply minor adjustments on case by case basis, good start: > Write in textbook style prose, without headings, no tables, no emojis.

softwaredoug 9mo ago

I recently realized every hypothesis I tested with an LLM, the LLM agreed with me. And if I wasn't careful about reading its caveats, I could leave thinking my idea was brilliant and would never get pushback. I tried something in the political realm. Asking to test a hypothesis and its opposite > Test this hypothesis: the far right in US politics mirrors late 19th century Victorianism as a cultural force compared to > Test this hypothesis: The left in US politics mirrors late 19th century Victorianism as a cultural force An LLM wants to agree with both, it created plausible arguments for both. While giving "caveats" instead of counterarguments. If I had my brain off, I might leave with some sense of "this hypothesis is correct". Now I'm not saying this makes LLMs useless. But the LLM didn't act like a human that might tell you your full of shit. It WANTED my hypothesis to be true and constructed a plausible argument for both. Even with prompting to act like a college professor critiquing a grad student, eventually it devolves back to "helpful / sycophantic". What I HAVE found useful is to give a list of mutually exclusive hypothesis and get probability ratings for each. Then it doesn't look like you want one / other. When the outcome matters, you realize research / hypothesis testing with LLMs is far more of a skill than just dumping a question to an LLM.

firasd 9mo ago
But this is the thing I'm pointing out. The idea that the LLM is an oracle or at least a stable subjective view holder is a mistake. As humans, WE have to explore the latent space of the model. We have to activate neurons. We have to say maybe the puritanism of the left ... maybe the puritanism of the right.. okay how about... We are privileged--and doomed--to have to think for ourselves alas
papandada 9mo ago
Admittedly I'm not an expert on 19th century Victorianism nor US politics, but this anecdote doesn't make me think any less of LLM's in the least, and I don't see why I should expect different behaviour than what you describe, especially on complex topics
- Rastonbury 9mo ago
  When I ask for an evaluation or issue I want the LLM to tell me when and how I'm wrong and me not having to worry about phrasing before I take the LLMs word and update my own beliefs/knowledge (especially for subjects I have zero knowledge about), I'm aware of this so when it isn't throw away queries, I tend to ask multiple times and ask for strawmans. I am aware they do this, but with the number of people walking around quoting 'chatgpt said' in real life and on forums, I don't think many people bother to stress test or are aware they phrasing may induce biased responses. It's akin to reading the news only from one source By now I have somewhat stopped relying on LLMs for point of view on latest academic stuff. I don't believe LLMs are able to evaluate paradigm shifting new studies against their massive training corpus. Thinking traces filled with 'tried to open this study, but it's paywalled, I'll use another' does not fill me with confidence that it can articulate a 2025 scientific consensus well. Based on how they work this definitely isn't an easy fix!

avalys 9mo ago

"The anti-sycophancy turn seems to mask a category error about what level of prophetic clarity an LLM can offer. No amount of persona tuning for skepticism will provide epistemic certainty about whether a business idea will work out, whether to add a line to your poem, or why a great movie flopped." What a lot of people actually want from an LLM, is for the LLM to have an opinion about the question being asked. The cool thing about LLMs is that they appear capable of doing this - rather than a machine that just regurgitates black-and-white facts, they seem to be capable of dealing with nuance and gray areas, providing insight, and using logic to reach a conclusion from ambiguous data. But this is the biggest misconception and flaw of LLMs. LLMs do not have opinions. That is not how they work. At best, they simulate what a reasonable answer from a person capable of having an opinion might be - without any consistency around what that opinion is, because it is simply a manifestation of sampling a probability distribution, not the result of logic. And what most people call sycophancy is that, as a result of this statistical construction, the LLM tends to reinforce the opinions, biases, or even factual errors, that it picks up on in the prompt or conversation history.

paulddraper 9mo ago
> At best, they simulate what a reasonable answer from a person capable of having an opinion might be And how would you compare that to human thoughts? “A submarine doesn’t actually swim” Okay what does it do then
- syntheticcdo 9mo ago
  But we didn't name them "artificial swimmers". We called them submarines - because there is a difference between human beings and machines.
  - senko 9mo ago
    But we did call computers "computers", even though "computers" used to refer to the human computers doing the same computing jobs.

casey2 9mo ago

AI bros complaining about loaded terms like slop and sycophancy when they still use terms like "intelligence", "learning", "reasoning", "attention" and all the words derived from "neuro" to describe a computer program. Don't make sycophantic slop generators and people will stop calling them that

transcriptase 9mo ago

I still suspect what happened was when the midwits all got access to ChatGPT etc and started participating in the A/B tests, they strongly selected for responses that agreed with them regardless of whether they were actually correct. Some of us want to be told when and why we’re wrong, and somewhere along the way AI models were either intentionally or unintentionally guided away from doing it because it improved satisfaction or engagement metrics. We already know from decades of studies that people prefer information that confirms their existing beliefs, so when you present 2 options with a “Which answer do you prefer?” selection, it’s not hard to see how the one that begins with “You’re absolutely right!” wins out.

port3000 9mo ago

I had completely forgotten about 'Sydney' and its emoji-laden diatribes. What a crazy moment, looking back.

devin 9mo ago

My observation is that some of the time what people are picking up on is that a conversation is not the interface they want for examining their problem.

leftouterjoins 9mo ago

At least with Opus 4.5 that magic phrase that has worked for me for it to understand something that has happened since it was trained is to explain the situation prefaced with "This happened after your knowledge cutoff" It understands this and often will spawn web searches to fill itself in on the matter.

jeffbee 9mo ago
"Knowledge cutoff" may not apply to all systems, and in the near future I doubt it will apply to any popular system because the cutoff is such a harmful antipattern. I've noticed in the last half year or so more of the best models are able to incorporate recent events without being explicitly told to do so.

damjon 9mo ago

jeffbee 9mo ago
I have standing orders for Gemini to reserve extreme praise for exceptional circumstances, to never use the words "masterclass" unless literally warranted, and to exercise more range of judgements rather than compressing all possible judgements in between "interesting" and "brilliant". This seems to have corrected a few of its habits.

vjerancrnjak 9mo ago

I catch myself sometimes writing the model it is confused and it should just assume what I am writing is true and continue reasoning from there. Sometimes I am actually right but sometimes I am not. Not sure what happens to any future RL and does it lean more to constantly assuming what is written as true but then has to wiggle out of it.

sz4kerto 9mo ago

Recently I've been using AI for some stuff that service providers don't want it to be used for (specifically: medical diagnosis). I found that Grok (4.1) is superior to most of the others when it comes to this, because it doesn't go out of its way to support my own hypotheses. I believe that syncophancy and guardrails will be major differentiators between LLM services, and the ones with less of those will always have a fan base.

djaouen 9mo ago

> For the purpose of conversation, what the model appears to ‘believe’ probably doesn’t matter This is wrong to the point of being absurd. What the model "appears to 'believe'" does matter, and the model's "beliefs" about humans and society at large have vast implications for humanity's future.

pfisch 9mo ago

It is a real problem that AI's will basically confirm that most inquiries are true. Just by asking a leading question often results in the AI confirming it is true or stretching reality to accommodate the answer being true. If I ask if a drug has a specific side effect and the answer is no it should say no. Not try to find a way to say yes that isn't really backed by evidence. People don't realize that when they ask a leading question that is really specific in a way where no one has a real answer then the AI will try to find a way to agree, and this is going to destroy people's lives. Honestly it already has.

ccppurcell 9mo ago

I just asked chatgpt about 72 hour preparedness kit. I said what I had already and asked it about the best way to store it first. It said something to the effect of "good, you are thinking about this in exactly the right way -- make sure you have the containers in place and then get the exact contents right later" not exactly the wording but you get the picture and can probably guess where I'm going with this. When I asked again, this time I asked about the items first. I had to prompt it with something like "or do you think I should get the storage sorted first" and it said "you are thinking about this in exactly the right way -- preparedness kits fail more often due to missing essentials than sub optimal storage" I can't decide which of these is right! Maybe there's an argument that it doesn't matter, and getting started is the most important thing, and so being encouraging is generally the best strategy here. But it's definitely worrying to me. It pretty much always says something like this to me (this is on the "honest and direct" personality setting or whatever).

cadamsdotcom 9mo ago

Another sign that AGI is still very far away.

TacticalCoder 9mo ago

"Dear LLM, TFA reads quite bit pompous to me" "You're absolutely right, what a great observation" ; )

diamond559 9mo ago

If you still unironically call llm's "AI" then you have no right calling out the obvious sycophancy of your product bro. He also ignores the real sycophancy in llm's reinforcing mental illness/delusions and focuses on semantic choices of the outputs. This post is drivel.