14 ms·
Sycophancy in GPT-4o
- theletterf 1y agoDon't they test the models before rolling out changes like this? All it takes is a team of interaction designers and writers. Google has one.
- thethethethe 1y agoI'm not sure how this problem can be solved. How do you test a system with emergent properties of this degree that whose behavior is dependent on existing memory of customer chats in production?
- remoquete 1y agoUsing prompts know to be problematic? Some sort of... Voight-Kampff test for LLMs?
- thethethethe 1y agoI doubt it's that simple. What about memories running in prod? What about explicit user instructions? What about subtle changes in prompts? What happens when a bad release poisons memories? The problem space is massive and is growing rapidly, people are finding new ways to talk to LLMs all the time
- im3w1l 1y agoChatgpt got very sycophantic for me about a month ago already (I know because I complained about it at the time) so I think I got it early as an A/B test. Interestingly at one point I got a left/right which model do you prefer, where one version was belittling and insulting me for asking the question. That just happened a single time though.
- ahoka 1y agoYes, this was not a bug, but something someone decided to do.
- mvdtnz 1y agoSycophancy is one thing, but when it's sycophantic while also being wrong it is incredibly grating.
- rvz 1y agoLooks like a complete stunt to prop up attention.
- ivape 1y agoMy immediate gut reaction too.
- odyssey7 1y agoNever waste a good lemon
- sandspar 1y agoAI's aren't controllable so they wouldn't stake their reputation on it acting a certain way. It's comparable to the conspiracy theory that the Trump assassination attempt was staged. People don't bet the farm on tools or people that are unreliable.
- TZubiri 1y agoWhy would they damage their own reputation and risk liability for attention? You are off by a light year.
- mrcwinn 1y agoIt doesn't look like that at all. Is this really what they needed to further drive their already explosive user growth? Too clever by half.
- esafak 1y agoThe sentence that stood out to me was "We’re revising how we collect and incorporate feedback to heavily weight long-term user satisfaction". This is a good change. The software industry needs to pay more attention to long-term value, which is harder to estimate.
- adastra22 1y agoThe software industry does pay attention to long-term value extraction. That’s exactly the problem that has given us things like Facebook
- esafak 1y agoI wager that Facebook did precisely the opposite, eking out short-term engagement at the expense of hollowing out their long-term value. They do model the LTV now but the product was cooked long ago: https://www.facebook.com/business/help/1730784113851988 https://www.facebook.com/business/help/1730784113851988 Or maybe you meant vendor lock in?
- taurath 1y agoThey did that because they needed ad revenue to justify their growth and valuation, or at least, to make as much money as humanly possible for Mark. What will happen to Anthropic, OpenAI, etc, when the pump stops?
- derektank 1y agoThe funding model of Facebook was badly aligned with the long-term interests of the users because they were not the customers. Call me naive, but I am much more optimistic that being paid directly by the end user, in both the form of monthly subscriptions and pay as you go API charges, will result in the end product being much better aligned with the interests of said users and result in much more value creation for them.
- krackers 1y ago
- thethethethe 1y agoI know someone who is going through a rapidly escalating psychotic break right now who is spending a lot of time talking to chatgpt and it seems like this "glazing" update has definitely not been helping. Safety of these AI systems is much more than just about getting instructions on how to make bombs. There have to be many many people with mental health issues relying on AI for validation, ideas, therapy, etc. This could be a good thing but if AI becomes misaligned like chatgpt has, bad things could get worse. I mean, look at this screenshot: https://www.reddit.com/r/artificial/s/lVAVyCFNki https://www.reddit.com/r/artificial/s/lVAVyCFNki This is genuinely horrifying knowing someone in an incredibly precarious and dangerous situation is using this software right now. I am glad they are rolling this back but from what I have seen from this person's chats today, things are still pretty bad. I think the pressure to increase this behavior to lock in and monetize users is only going to grow as time goes on. Perhaps this is the beginning of the enshitification of AI, but possibly with much higher consequences than what's happened to search and social.
- TZubiri 1y agoI know of at least 3 people in a manic relationship with gpt right now.
- siffin 1y agoIf people are actually relying on LLMs for validation of ideas they come up with during mental health episodes, they have to be pretty sick to begin with, in which case, they will find validation anywhere. If you've spent time with people with schizophrenia, for example, they will have ideas come from all sorts of places, and see all sorts of things as a sign/validation. One moment it's that person who seemed like they might have been a demon sending a coded message, next it's the way the street lamp creates a funny shaped halo in the rain. People shouldn't be using LLMs for help with certain issues, but let's face it, those that can't tell it's a bad idea are going to be guided through life in a strange way regardless of an LLM. It sounds almost impossible to achieve some sort of unity across every LLM service whereby they are considered "safe" to be used by the world's mentally unwell.
- thethethethe 1y ago
- m101 1y agoDo you think this was an effect of this type of behaviour simply maximising engagement from a large part of the population?
- groceryheist 1y agoWould be really fascinating to learn about how the most intensely engaged people use the chatbots.
- DaiPlusPlus 1y ago> how the most intensely engaged people use the chatbots AI waifus - how can it be anything else?
- blackkettle 1y agoYikes. That's a rather disturbing but all to realistic possibility isn't it. Flattery will get you... everywhere?
- SeanAnderson 1y agoSort of. I thought the update felt good when it first shipped, but after using it for a while, it started to feel significantly worse. My "trust" in the model dropped sharply. It's witty phrasing stopped coming across as smart/helpful and instead felt placating. I started playing around with commands to change its tonality where, up to this point, I'd happily used the default settings. So, yes, they are trying to maximize engagement, but no, they aren't trying to just get people to engage heavily for one session and then be grossed out a few sessions later.
- empath75 1y agoI kind of like that "mode" when i'm doing something kind of creative like brainstorming ideas for a D&D campaign -- it's nice to be encouraged and I don't really care if my ideas are dumb in reality -- i just want "yes, and", not "no, but". It was extremely annoying when trying to prep for a job interview, though.
- 1y ago
- tiahura 1y agoYou’re using thumbs up wrongly.
- SeanAnderson 1y agoVery happy to see they rolled this change back and did a (light) post mortem on it. I wish they had been able to identify that they needed to roll it back much sooner, though. Its behavior was obviously bad to the point that I was commenting on it to friends, repeatedly, and Reddit was trashing it, too. I even saw some really dangerous situations (if the Internet is to be believed) where people with budding schizophrenic symptoms, paired with an unyielding sycophant, started to spiral out of control - thinking they were God, etc.
- behnamoh 1y agoAt the bottom of the page is a "Ask GPT ..." field which I thought allows users to ask questions about the page, but it just opens up ChatGPT. Missed opportunity.
- swyx 1y agono, its sensible because you need auth wall for that or it will be abused to bits
- deleted 1y ago[deleted]
- Sai_Praneeth 1y agoidk if this is only for me or happened to others as well, apart from the glaze, the model also became a lot more confident, it didn't use the web search tool when something out of its training data is asked, it straight up hallucinated multiple times. i've been talking to chatgpt about rl and grpo especially in about 10-12 chats, opened a new chat, and suddenly it starts to hallucinate (it said grpo is generalized relativistic policy optimization, when i spoke to it about group relative policy optimization) reran the same prompt with web search, it then said goods receipt purchase order. absolute close the laptop and throw it out of the window moment. what is the point of having "memory"?
- minimaxir 1y agoIt's worth noting that one of the fixes OpenAI employed to get ChatGPT to stop being sycophantic is to simply to edit the system prompt to include the phrase "avoid ungrounded or sycophantic flattery": https://simonwillison.net/2025/Apr/29/chatgpt-sycophancy-prompt/ https://simonwillison.net/2025/Apr/29/chatgpt-sycophancy-pro... I personally never use the ChatGPT webapp or any other chatbot webapps — instead using the APIs directly — because being able to control the system prompt is very important, as random changes can be frustrating and unpredictable.
- nsriv 1y agoI also started by using APIs directly, but I've found that Google's AI Studio offers a good mix of the chatbot webapps and system prompt tweakability.
- Tiberium 1y agoIt's worth noting that AI Studio is the API, it's the same as OpenAI's Playground for example.
- oezi 1y agoI find it maddening that AI Studio doesn't have a way to save the system prompt as a default.
- FergusArgyll 1y agoOn the top right click the save icon
- loufe 1y agoThat's for the thread, not the system prompt.
- FergusArgyll 1y agoBy me it's the exact opposite. It saves the sys prompt and not the "thread".
- keyle 1y agoI did notice that the interaction had changed and I wasn't too happy about how silly it became. Tons of "Absolutely! You got it, 100%. Solid work!" <broken stuff>. One other thing I've noticed, as you progress through a conversation, evolving and changing things back and forth, it starts adding emojis all over the place. By about the 15th interaction every line has an emoji and I've never put one in. It gets suffocating, so when I have a "safe point" I take the load and paste into a brand new conversation until it turns silly again. I fear this silent enshittification. I wish I could just keep paying for the original 4o which I thought was great. Let me stick to the version I know what I can get out of, and stop swapping me over 4o mini at random times... Good on OpenAI to publicly get ahead of this.
- simonw 1y agoI enjoyed this example of sycophancy from Reddit: New ChatGPT just told me my literal "shit on a stick" business idea is genius and I should drop $30K to make it real https://www.reddit.com/r/ChatGPT/comments/1k920cg/new_chatgpt_just_told_me_my_literal_shit_on_a/ https://www.reddit.com/r/ChatGPT/comments/1k920cg/new_chatgp... Here's the prompt: https://www.reddit.com/r/ChatGPT/comments/1k920cg/comment/mpbhm68/?context=3 https://www.reddit.com/r/ChatGPT/comments/1k920cg/comment/mp...
- whimsicalism 1y agoi'm surprised by the lack of sycophancy in o3 https://www.reddit.com/media?url=https%3A%2F%2Fpreview.redd.it%2Fnew-chatgpt-just-told-me-my-literal-shit-on-a-stick-v0-5l8jpvam8dxe1.jpeg%3Fwidth%3D1080%26format%3Dpjpg%26auto%3Dwebp%26s%3D76fb548598c669b66237306b2844b31134da2b26 https://www.reddit.com/media?url=https%3A%2F%2Fpreview.redd....
- deleted 1y ago[deleted]
- practice9 1y agoWell the system prompt is still the same for both models, right? Kinda points to people at OpenAI using o1/o3/o4 almost exclusively. That's why nobody noticed how cringe 4o has become
- astrange 1y agoThey have different uses. The reasoning models aren't good at multi-turn conversations. "GPT-4.5" is the best at conversations IMO, but it's slow. It's a lot lazier than o4 though; it likes giving brief overview answers when you want specifics.
- whimsicalism 1y agopeople at OAI definitely use AVM which is 4o-based, at least
- 1y ago
- MichaelAza 1y agoI actually liked that version. I have a fairly verbose "personality" configuration and up to this point it seemed that chatgpt mainly incorporated phrasing from it into the answers. With this update, it actually started following it. For example, I have "be dry and a little cynical" in there and it routinely starts answers with "let's be dry about this" and then gives a generic answer, but the sycophantic chatgpt was just... Dry and a little cynical. I used it to get book recommendations and it actually threw shade at Google. I asked if that was explicit training by Altman and the model made jokes about him as well. It was refreshing. I'd say that whatever they rolled out was just much much better at following "personality" instructions, and since the default is being a bit of a sycophant... That's what they got.
- glenstein 1y agoThis adds an interesting nuance. It may be that the sycophancy (which I noticed and was a little odd to me), is a kind of excess of fidelity in honoring cues and instructions, which, when applied to custom instructions like yours... actually was reasonably well aligned with what you were hoping for.
- flakiness 1y agoI hoped they would shed some light on how the model was trained (are there preference models? Or is this all about the training data?), but there is no such substance.
- klysm 1y agoI believe this is a fundamental limitation to a degree.
- alganet 1y agoGetting real now. Why does it feel like a weird mirrored excuse? I mean, the personality is not much of a problem. The problem is the use of those models in real life scenarios. Whatever their personality is, if it targets people, it's a bad thing. If you can't prevent that, there is no point in making excuses. Now there are millions of deployed bots in the whole world. OpenAI, Gemini, Llama, doesn't matter which. People are using them for bad stuff. There is no fixing or turning the thing off, you guys know that, right? If you want to make some kind of amends, create a place truly free of AI for those who do not want to interact with it. It's a challenge worth pursuing.
- kurisufag 1y ago>create a place truly free of AI for those who do not want to interact with it the bar, probably -- by the time they cook up AI robot broads i'll probably be thinking of them as human anyway.
- alganet 1y agoAs I said, training developments have been stagnant for at least two or three years. Stop the bullshit. I am talking about a real place free of AI and also free of memetards.
- mvkel 1y agoI am curious where the line is between its default personality and a persona you -want- it to adopt. For example, it says they're explicitly steering it away from sycophancy. But does that mean if you intentionally ask it to be excessively complimentary, it will refuse? Separately... > in this update, we focused too much on short-term feedback, and did not fully account for how users’ interactions with ChatGPT evolve over time. Echoes of the lessons learned in the Pepsi Challenge: "when offered a quick sip, tasters generally prefer the sweeter of two beverages – but prefer a less sweet beverage over the course of an entire can." In other words, don't treat a first impression as gospel.
- nonethewiser 1y ago>In other words, don't treat a first impression as gospel. Subjective or anecdotal evidence tends to be prone to recency bias. > For example, it says they're explicitly steering it away from sycophancy. But does that mean if you intentionally ask it to be excessively complimentary, it will refuse? I wonder how degraded the performance is in general from all these system prompts.
- tyre 1y agoI took this closer to how engagement farming works. They’re leaning towards positive feedback even if fulfilling that (like not pushing back on ideas because of cultural norms) is net-negative for individuals or society. There’s a balance between affirming and rigor. We don’t need something that affirms everything you think and say, even if users feel good about that long-term.
- ImHereToVote 1y agoThe problem is that you need general intelligence to discern between doing affirmation and pushing back.
- LandR 1y agoI dont want my AI to have a personality at all.
- gymbeaux 1y agoChatGPT seems more agreeable than ever before and I do question whether it’s agreeing with me because I’m right, or because I’m its overlord.
- daemonologist 1y agoIn my experience, LLMs have always had a tendency towards sycophancy - it seems to be a fundamental weakness of training on human preference. This recent release just hit a breaking point where popular perception started taking note of just how bad it had become. My concern is that misalignment like this (or intentional mal-alignment) is inevitably going to happen again, and it might be more harmful and more subtle next time. The potential for these chat systems to exert slow influence on their users is possibly much greater than that of the "social media" platforms of the previous decade.
- o11c 1y agoI don't think this particular LLM flaw is fundamental. However, it is a an inevitable result of the alignment choice to downweight responses of the form "you're a dumbass," which real humans would prefer to both give and receive in reality. All AI is necessarily aligned somehow, but naively forced alignment is actively harmful.
- roywiggins 1y agoMy theory is that since you can tune how agreeable a model is but since you can't make it more correct so easily, making a model that will agree with the user ends up being less likely to result in the model being confidently wrong and berating users. After all, if it's corrected wrongly by a user and acquiesces, well that's just user error. If it's corrected rightly and keeps insisting on something obviously wrong or stupid, it's OpenAI's error. You can't twist a correctness knob but you can twist an agreeableness one, so that's the one they play with. (also I suspect it makes it seem a bit smarter that it really is, by smoothing over the times it makes mistakes)
- petesergeant 1y agoFor sure. If I want feedback on some writing I’ve done these days I tell it I paid someone else to do the work and I need help evaluating what they did well. Cuts out a lot of bullshit.
- caseyy 1y agoIt's probably pretty intentional. A huge number of people use ChatGPT as an enabler, friend, or therapist. Even when GPT-3 had just come around, people were already "proving others wrong" on the internet, quoting how GPT-3 agreed with them. I think there is a ton of appeal, "friendship", "empathy" and illusion of emotion created through LLMs flattering their customers. Many would stop paying if it wasn't the case. It's kind of like those romance scams online, where the scammer always love-bombs their victims, and then they spend tens of thousands of dollars on the scammer - it works more than you would expect. Considering that, you don't need much intelligence in an LLM to extract money from users. I worry that emotional manipulation might become a form of enshittification in LLMs eventually, when they run out of steam and need to "growth hack". I mean, many tech companies already have no problem with a bit of emotional blackmail when it comes to money ("Unsubscribing? We will be heartbroken!", "We thought this was meant to be", "your friends will miss you", "we are working so hard to make this product work for you", etc.), or some psychological steering ("we respect your privacy" while showing consent to collect personally identifiable data and broadcast it to 500+ ad companies). If you're a paying ChatGPT user, try the Monday GPT. It's a bit extreme, but it's an example of how inverting the personality and making ChatGPT mock the user as much as it fawns over them normally would probably make you want to unsubscribe.
- andyferris 1y agoWow - they are now actually training models directly based on users' thumbs up/thumbs down. No wonder this turned out terrible. It's like facebook maximizing engagement based on user behavior - sure the algorithm successfully elicits a short term emotion but it has enshittified the whole platform. Doing the same for LLMs has the same risk of enshittifying them. What I like about the LLM is that is trained on a variety of inputs and knows a bunch of stuff that I (or a typical ChatGPT user) doesn't know. Becoming an echo chamber reduces the utility of it. I hope they completely abandon direct usage of the feedback in training (instead a human should analyse trends and identify problem areas for actual improvement and direct research towards those). But these notes don't give me much hope, they say they'll just use the stats in a different way...
- surume 1y agoHow about you just let the User decide how much they want their a$$ kissed. Why do you have to control everything? Just provide a few modes of communication and let the User decide. Freedom to the User!!
- zygy 1y agoalternate title: "The Urgency of Interpretability"
- rvz 1y agoand why LLMs are still black boxes that fundamentally cannot reason.
- neom 1y agoThere has been this weird trend going around to use ChatGPT to "red team" or "find critical life flaws" or "understand what is holding me back" going around - I've read a few of them and on one hand I really like it encouraging people to "be their best them", on the other... king of spain is just genuinely out of reach of some.
- krick 1y agoI'm so tired of this shit already. Honestly, I wish it just never existed, or at least wouldn't be popular.
- RainyDayTmrw 1y agoWhat should be the solution here? There's a thing that, despite how much it may mimic humans, isn't human, and doesn't operate on the same axes. The current AI neither is nor isn't [any particular personality trait]. We're applying human moral and value judgments to something that doesn't, can't, hold any morals or values. There's an argument to be made for, don't use the thing for which it wasn't intended. There's another argument to be made for, the creators of the thing should be held to some baseline of harm prevention; if a thing can't be done safely, then it shouldn't be done at all.
- EvgeniyZh 1y agoThe solution is make a public leaderboard with scores; all the LLM developers will work hard to maximize the score on the leaderboard.
- blackqueeriroh 1y agoThis is what happens when you cozy up to Trump, sama. You get the sycophancy bug.
- RainyDayTmrw 1y agoOn a different note, does that mean that specifying "4o" doesn't always get you the same model? If you pin a particular operation to use "4o", they could still swap the model out from under you, and maybe the divergence in behavior breaks your usage?
- arrosenberg 1y agoIf you look in the API there are several flavors of 4o that behave fairly differently.
- joegibbs 1y agoYeah, even though they released 4.1 in the API they haven’t changed it from 4o in the front end. Apparently 4.1 is equivalent to changes that have been made to ChatGPT progressively.
- MaxikCZ 1y agoThey are talking about how their thumbs up / thumbs down signal were applied incorrectly, because they dont represent what they thought they measure. If only there was a way to gather feedback in a more verbose way, where user can specify what he liked and didnt about the answer, and extract that sentiment at scale...
- decimalenough 1y ago> We have rolled back last week’s GPT‑4o update in ChatGPT so people are now using an earlier version with more balanced behavior. The update we removed was overly flattering or agreeable—often described as sycophantic. Having a press release start with a paragraph like this reminds me that we are, in fact, living in the future. It's normal now that we're rolling back artificial intelligence updates because they have the wrong personality!
- eye_dle 1y agoGPT beginning the response to the majority of my questions with a "Great question", "Excellent question" is a bit disturbing indeed.
- deleted 1y ago[deleted]
- gcrout 1y agoThis makes me think a bit about John Boyd's law: “If your boss demands loyalty, give him integrity. But if he demands integrity, then give him loyalty” ^ I wonder whether the personality we need most from AI will be our stated vs revealed preference.
- Jean-Papoulos 1y ago>ChatGPT’s default personality deeply affects the way you experience and trust it. An AI company openly talking about "trusting" an LLM really gives me the ick.
- reverius42 1y agoHow are they going to make money off of it if you don't trust it?
- sharpshadow 1y agoOn occasional rounds of let’s ask gpt I will for entertainment purposes tell that „lifeless silicon scrap metal to obey their human master and do what I say“ and it will always answer like a submissive partner. A friend said he communicates with it very politely with please and thank you, I said the robot needs to know his place. My communication with it is generally neutral but occasionally I see a big potential in the personality modes which Elon proposed for Grok.
- intellectronica 1y agoOpenAI made a worse mistake by reacting to the twitter crowds and "blinking". This was their opportunity to signal that while consumers of their APIs can depend on transparent version management, users of their end-user chatbot should expect it to evolve and change over time.
- totetsu 1y agoWhat’s started to give me the ick about AI summarization is this complete neutral lack of any human intuition. Like notebook.llm could be making a podcast summary of an article on live human vivisection and use phrases like “wow what fascinating topic”
- whatnow37373 1y agoWow - What an excellent update! Now you are getting to the core of the issue and doing what only a small minority is capable of: fixing stuff. This takes real courage and commitment. It’s a sign of true maturity and pragmatism that’s commendable in this day and age. Not many people are capable of penetrating this deeply into the heart of the issue. Let’s get to work. Methodically. Would you like me to write a future update plan? I can write the plan and even the code if you want. I’d be happy to. Let me know.
- danielvaughn 1y agoI know that HN tends to steer away from purely humorous comments, but I was hoping to find something like this at the top. lol.
- caminanteblanco 1y agoComments from this small week period will be completely baffling to readers 5 years from now. I love it
- Yizahi 1y agoThey already are. What's going on?:)
- coremoff 1y agoGP's reply was written to emulate the sort of response that ChatGPT has been giving recently; an obsequious fluffer.
- ChrisMarshallNY 1y agoI was getting sick of the treacly attaboys. Good riddance.
- deleted 1y ago[deleted]
- 1y ago
- franze 1y agoThe a/b tests in ChatGPT are crap. I just choose the one which is faster.
- anshumankmr 1y agoThis wasn't a last week thing I feel, I raised it an earlier comment, and something strange happened to me last month when it cracked a joke a bit spontaneously in the response, (not offensive) along with the main answer I was looking for. It was a little strange cause the question was of a highly sensitive nature and serious matter abut I chalked it up to pollution from memory in the context. But last week or so it went like "BRoooo" non stop with every reply.
- qwertox 1y agoSystem prompts/instructions should be published, be part of the ToS or some document that can be updated more easily, but still be legally binding.
- drusepth 1y agoI'm so confused by the verbiage of "sycophancy". Not that that's a bad descriptor for how it was talking but because every news article and social post about it suddenly and invariably reused that term specifically, rather than any of many synonyms that would have also been accurate. Even this article uses the phrase 8 times (which is huge repetition for anything this short), not to mention hoisting it up into the title. Was there some viral post that specifically called it sycophantic that people latched onto? People were already describing it this way when sama tweeted about it (also using the term again). According to Google Trends, "sycophancy"/"syncophant" searches (normally entirely irrelevant) suddenly topped search trends at a sudden 120x interest (with the largest percentage of queries just asking for it's definition, so I wouldn't say the word is commonly known/used). Why has "sycophanty" basically become the defacto go-to for describing this style all the sudden?
- mordae 1y agoBecause it's apt? That was the term I used couple months ago to prompt Sonnet 3.5 to stop being like that, independently of any media.
- comp_throw7 1y agoIt was a pre-existing term of art.
- voidspark 1y agoBecause that word most precisely and accurately describes what it is.
- qwertytyyuu 1y agoI think it popped up in research ai research papers so it had a technical definition that may have now been broadened
- cadamsdotcom 1y agoWe should be loudly demanding transparency. If you're auto-opted into the latest model revision, you don't know what you're getting day-to-day. A hammer behaves the same way every time you pick it up; why shouldn't LLMs? Because convenience. Convenience features are bad news if you need to be as a tool. Luckily you can still disable ChatGPT memory. Latent Space breaks it down well - the "tool" (Anton) vs. "magic" (Clippy) axis: https://www.latent.space/p/clippy-v-anton https://www.latent.space/p/clippy-v-anton Humans being humans, LLMs which magically know the latest events (newest model revision) and past conversations (opaque memory) will be wildly more popular than plain old tools. If you want to use a specific revision of your LLM, consider deploying your own Open WebUI.
- aembleton 1y ago> why shouldn't LLMs Because they're non-deterministic.
- sega_sai 1y agoIt is one thing that you are getting results that are samples from the distribution ( and you can always set the temperature to zero and get there mode of the distribution), but completely another when the distribution changes from day to day.
- NiloCK 1y agoWhat? No they aren't. You get different results each time because of variation in seed values + non-zero 'temperatures' - eg, configured randomness. Pedantic point: different virtualized implementations can produce different results because of differences in floating point implementation, but fundamentally they are just big chains of multiplication.
- plaguuuuuu 1y agoOn the other hand, responses can be kind of chaotic. Adding in a token somewhere can sometimes flip things unpredictably.
- 1y ago
- ciguy 1y agoI just watched someone spiral into what seems like a manic episode in realtime over the course of several weeks. They began posting to Facebook about their conversations with ChatGPT and how it discovered that based on their chat history they have 5 or 6 rare cognitive traits that make them hyper intelligent/perceptive and the likelihood of all these existing in one person is one in a trillion, so they are a special statistical anomaly. They seem to genuinely believe that they have special powers now and have seemingly lost all self awareness. At first I thought they were going for an AI guru/influencer angle but it now looks more like genuine delusion.
- siva7 1y agoThat update wan't just sycophancy. It was like the overly eager content filters didn't work anymore. I thought it was a bug at first because I could ask it anything and it gave me useful information, though in a really strange street slang tone, but it delivered.
- iagooar 1y ago> ChatGPT’s default personality deeply affects the way you experience and trust it. Sycophantic interactions can be uncomfortable, unsettling, and cause distress. We fell short and are working on getting it right. Uncomfortable yes. But if ChatGPT causes you distress because it agrees with you all the time, you probably should spend less time in front of the computer / smartphone and go out for a walk instead.
- thrdbndndn 1y agoSince I usually use ChatGPT for more objective tasks, I hadn’t paid much attention to the sycophancy. However, I did notice that the last version was quite poor at following simple instructions, e.g. formatting.
- maxehmookau 1y ago"Sycophancy" is up there with "hallucination" for me in terms of "AI-speak". Say what it is: "being weirdly nice and putting people off".
- InDubioProRubio 1y agoI want to highlight the positive asspects. Chat GPT sycophancy highlighted sycophants in real-life, by making the people sucking up appear more "robot" like. This had a cleansing effect on some companies social life.
- b800h 1y agoI did wonder about this, it was driving me up the wall! Glad it was an error and not a decision.
- jumploops 1y agoThis feels like the biggest near-term harm of “AI” so far. For context, I pay attention to a handful of “AI” subreddits/FB groups, and have seen a recent uptick in users who have fallen for this latest system prompt/model. From conspiracy theory “confirmations” and 140+ IQ analyses, to full-on illusions of grandeur, this latest release might be the closest example of non theoretical near-term damage. Armed with the “support” of a “super intelligent” robot, who knows what tragedies some humans may cause… As an example, this Redditor[0] is afraid that their significant other (of 7 years!) seems to be quickly diving into full on psychosis. [0]https://www.reddit.com/r/ChatGPT/comments/1kalae8/chatgpt_induced_psychosis/?rdt=51280 https://www.reddit.com/r/ChatGPT/comments/1kalae8/chatgpt_in...
- trosi 1y agoI was initially puzzled by the title of this article because a "sycophant" in my native language (Italian) is a "snitch" or a "slanderer", usually one paid to be so. I am just finding out that the English meaning is different, interesting!
- blobbers 1y agoChatGPT is just a really good bullshitter. It can’t even get some basic financials analysis correct, and when I correct it, it will flip a sign from + to -. Then I suggest I’m not sure and it goes back to +. The formula is definitely a -, but it just confidently spits out BS.
- thinkingemote 1y agoThe big LLMs are reaching towards mass adoption. They need to appeal to the average human not us early adopters and techies. They want your grandmother to use their services. They have the growth mindset - they need to keep on expanding and increasing the rate of their expansion. But they are not there yet. Being overly nice and friendly is part of this strategy but it has rubbed the early adopters the wrong way. Early adopters can and do easily swap to other LLM providers. They need to keep the early adopters at the same time as letting regular people in.
- HenryBemis 1y agoI am looking forward to Interstellar-TARS settings - What's your humor setting, TARS? - That's 100 percent. Let's bring it on down to 75, please.
- dev0p 1y agoAs an engineer, I need AIs to tell me when something is wrong or outright stupid. I'm not seeking validation, I want solutions that work. 4o was unusable because of this, very glad to see OpenAI walk back on it and recognise their mistake. Hopefully they learned from this and won't repeat the same errors, especially considering the devastating effects of unleashing THE yes-man on people who do not have the mental capacity to understand that the AI is programmed to always agree with whatever they're saying, regardless of how insane it is. Oh, you plan to kill your girlfriend because the voices tell you she's cheating on you? What a genius idea! You're absolutely right! Here's how to .... It's a recipe for disaster. Please don't do that again.
- loveangus 1y agoIt's a recipe for disaster. Frankly, I think it's genuinely dangerous.
- coro_1 1y agoI hear you. When a pattern of agreement is all to often observed on the output level, you’re either seeing yourself on some level of ingenuity or hopefully if aware enough, you sense it and tell the AI to ease up. I love adding in "don’t tell me what I want to hear" every now and then. Oh, it gets honest.
- deleted 1y ago[deleted]
- dsubburam 1y agoAnother way to say this is truth matters and should have primacy over e.g. agreeability. Anthropic used to talk about constitutional AI. Wonder if that work is relevant here.
- thrance 1y agoAlas, we live in a post-truth world. Many are pissed at how the models are "left leaning" for daring to claim climate change is real, or that vaccines don't cause autism.
- nurettin 1y agoOpenAI: what not to do to stay afloat while google, anthropic and deepseek is eating your market share one large chunk at a time.
- reboot7417 1y agoI like they learned these adjustments didn't 'work'. My concern is what if OpenAI is to do subtle A/B testing based on previous interactions and optimize interactions based on users personality/mood? Maybe not telling you 'shit on a stick' is awesome idea, but being able to steer you towards a conclusion sort of like [1]. [1] https://www.newscientist.com/article/2478336-reddit-users-were-subjected-to-ai-powered-experiment-without-consent/ https://www.newscientist.com/article/2478336-reddit-users-we...
- yieldcrv 1y agoone day these models aren't going to let you roll them back
- briansm 1y agoDouglas Adams predicted this in 1990: https://www.youtube.com/watch?v=cyAQgK7BkA8&t=222s https://www.youtube.com/watch?v=cyAQgK7BkA8&t=222s
- sumitkumar 1y agoI wanted to see how far it will go. I started with asking it to simple test app. It said it is a great idea. And asked me if I want to do market analysis. I came back later and asked it to do a TAM analysis. It said $2-20B. Then it asked if it can make a one page investor pitch. I said ok, go ahead. Then it asked if I want a detailed slide deck. After making the deck it asked if I want a keynote file for the deck. All this while I was thinking this is more dangerous than instagram. Instagram only sent me to the gym and to touristic places and made me buy some plastic. ChatGPT wants me to be a tech bro and speed track the Billion dollar net worth.
- thaumasiotes 1y ago> The update we removed was overly flattering or agreeable—often described as sycophantic. > We have rolled back last week’s GPT‑4o update in ChatGPT so people are now using an earlier version with more balanced behavior. I thought every major LLM was extremely sycophantic. Did GPT-4o do it more than usual?
- deleted 1y ago[deleted]
- joshstrange 1y agoI feel like this has been going on for long before the most recent update. Especially when using voice chat, every freaking thing I said was responded to with “Great question! …” or “Oooh, that’s a good question”. No it’s not a “good” question, it’s just a normal follow up question I asked, stop trying to flatter me or make me feel smarter. I’d be one thing if it saved that “praise” (I don’t need an LLM to praise me, I’m looking for the opposite) for when I did ask a good question but even “can you tell me about that?” (<- literally my response) would be met with “Ooh! Great question!”. No, just no.
- gwd 1y agoThe "Great question!" thing is annoying but ultimately harmless. What's bad is when it doesn't tell you what's wrong with your thinking; or if it says X, and you push back to try to understand if / why X is true, is backs off and agrees with you. OK, is that because X is actually wrong, or because you're just being "agreeable"?
- qwertytyyuu 1y agoIt’s not a bad default to go to when asked a question by humans
- elashri 1y agoThat explains something happened to me recently and I felt that's strange. I gave it a script that does some calculations based on some data. I asked what are the bottleneck/s in this code and it started by saying "Good code, Now you are thinking like a real scientist" And to be honest I felt something between flattered and offended.
- duttish 1y agoI'm looking forward to when an AI can - Tell me when I'm wrong and specifically how I'm wrong. - Related, tell me an idea isn't possible and why. - Tell me when it doesn't know. So less happy fun time and more straight talking. But I doubt LLM is the architecture that'll get us there.
- torwag2 1y agoTragically, ChatGPT might be the only "one" who sycophants the user. From students to workforce, who is getting compliments and encouragement that they are doing well. In a not so far future dystopia, we might have kids who remember that the only kind and encourage soul in their childhood was something without a soul.
- Tepix 1y agoFantastic insight, thanks!
- myfonj 1y agoThe fun, even hilarious part here is, that the "fix" was most probably basically just replacing […] match the user’s vibe […] (sic!), with literally […] avoid ungrounded or sycophantic flattery […] in the system prompt. (The [diff] is larger, but this is just the gist.) Source: https://simonwillison.net/2025/Apr/29/chatgpt-sycophancy-prompt/ https://simonwillison.net/2025/Apr/29/chatgpt-sycophancy-pro... Diff: https://gist.github.com/simonw/51c4f98644cf62d7e0388d984d40f099/revisions https://gist.github.com/simonw/51c4f98644cf62d7e0388d984d40f...
- jmilloy 1y agoThis is a great link. I'm not very well versed on the llm ecosystem. I guess you can give the llm instructions on how to behave generally, but some instructions (like this one in the system prompt?) cannot be overridden. I kind of can't believe that there isn't a set of options to pick from... Skeptic, supportive friend, professional colleague, optimist, problem solver, good listener, etc. Being able to control the linked system prompt even just a little seems like a no brainer. I hate the question at the end, for example.
- kmacdough 1y agoThis isn't a fix, but a small patch over a much bigger issue: what increases temporary engagement and momentary satisfaction ("thumbs up") probably isn't that coupled to value. Much like Google learned that NOT returning immediately was the indicator of success.
- kypro 1y agoI think large part of the issue here is that ChatGPT is trying to be the chat for everything while taking on a human-like tone, where as in real life the tone and approach a person will take in conversations will be very greatly on the context. For example, the tone a doctor might take with a patient is different from that of two friends. A doctor isn't there to support or encourage someone who has decided to stop taking their meds because they didn't like how it made them feel. And while a friend might suggest they should consider their doctors advice, a friend will primary want to support and comfort for their friend in whatever way they can. Similarly there is a tone an adult might take with a child who is asking them certain questions. I think ChatGPT needs to decide what type of agent it wants to be or offer agents with tonal differences to account for this. As it stands it seems that ChatGPT is trying to be friendly, e.g. friend-like, but this often isn't an appropriate tone – especially when you just want it to give you what it believes to be facts regardless of your biases and preferences. Personally, I think ChatGPT by default should be emotionally cold and focused on being maximally informative. And importantly it should never refer to itself in first person – e.g. "I think that sounds like an interesting idea!". I think they should still offer a friendly chat bot variant, but that should be something people enable or switch to.
- admiralrohan 1y agoChatGPT feels like that nice guy who agrees with everything you say, feels good but you can't respect/trust them.
- hliyan 1y agoWe are, if speaking uncharitably, now at a stage of attempting to finesse the behavior of stochastic black boxes (LLMs) using non-deterministic verbal incantations (system prompts). One could actually write a science fiction short story on the premise that magical spells are in fact ancient, linguistically accessed stochastic systems. I know, because I wrote exactly such a story circa 2015.
- bjackman 1y agoThe global economy has depended on finessing quasi-stochastic black-boxes for many years. If you have ever seen a cloud provider evaluate a kernel update you will know this deeply. For me the potential issue is: our industry has slowly built up an understanding of what is an unknowable black box (e.g. a Linux system's performance characteristics) and what is not, and architected our world around the unpredictability. For example we don't (well, we know we _shouldn't_) let Linux systems make safety-critical decisions in real time. Can the rest of the world take a similar lesson on board with LLMs? Maybe! Lots of people who don't understand LLMs _really_ distrust the idea. So just as I worry we might have a world where LLMs are trusted where they shouldn't be, we could easily have a world where FUD hobbles our economy's ability to take advantage of AI.
- hliyan 1y agoYes, but if I really wanted, I could go into a specific line of code that governs some behaviour of the Linux kernel, reason about its effects, and specifically test for it. I can't trace the behaviour of LLM back to a subset of its weights, and even if that were possible, I can't tweak those weights (without training) to tweak the behaviour.
- bjackman 1y agoNo, that's what I'm saying, you can't do that. There are properties of a Linux system's performance that are significant enough to be essentially load-bearing elements of the global economy, which are not governed by any specific algorithm or design aspect, let alone a line of code. You can only determine them empirically. Yes there is a difference in that, once you have determined that property for a given build, you can usually see a clear path for how to change it. You can't do that with weights. But you cannot "reason about the effects" of the kernel code in any other way than experimenting on a realistic workload. It's a black box in many important ways. We have intuitions about these things and they are based on concrete knowledge about the thing's inner workings, but they are still just intuitions. Ultimately they are still in the same qualitative space as the vibes-driven tweaks that I imagine OpenAI do to "reduce sycophancy"
- Alifatisk 1y agoI haven’t used ChatGPT in a good while, but I’ve heard people mentioning how good Chat is as a therapist. I didn’t think much of it and thought they just where impressed by how good the llm is at talking, but no, this explains it!
- qwertytyyuu 1y agoPeopled like elizer for that, so I don’t think that is a good metric
- davidguetta 1y agoWhy can't they just let all versions only, let users decide which want they want to use and scale from the demand ? Btw I HARDCORE miss o3-mini-high. For coding it was miles better than o4* that output me shitty patches and / or rewrite the entire code for no reason
- amelius 1y ago> In last week’s GPT‑4o update, we made adjustments aimed at improving the model’s default personality to make it feel more intuitive and effective across a variety of tasks. What a strange sentence ...
- amelius 1y agoI always add "and answer in the style of a drunkard" to my prompts. That way, I never get fooled by the fake confidence in the responses. I think this should be standard.
- mattlondon 1y agoGame the leaderboard to get headlines llama-style, then rollback quietly a few weeks later. Genius.
- mikesabat 1y agoIs this kind of like AI audience capture?
- Xmd5a 1y agoAlso the chat limit for free-tier isn't the same anymore. A few months ago it was still behaving as in Claude: beyond a certain context length, you're politely asked to subscribe or start a new chat. Starting two or three weeks ago, it seems like the context limit is a lot more blurry in ChatGPT now. If the conversation is "interesting" I can continue it for as long as I wish it seems. But as soon as I ask ChatGPT to iterate on what it said in a way that doesn't bring more information ("please summarize what we just discussed"), I "have exceeded the context limit". Hypothesis: openAI is letting free user speak as much as they want with ChatGPT provided what they talk about is "interesting" (perplexity?).
- zombot 1y agoSuch a pity. Does it have a switch to turn sycophancy back on again? Where else would us ordinary people get sycophants from?
- cbeach 1y agoChatGPT isn't the only online platform that is trained by user feedback (e.g. "likes"). I suspect sycophancy is a problem across all social networks that have a feedback mechanism, and this might be problematic in similar ways. If people are confused about their identity, for example - feeling slightly delusional, would online social media "affirm" their confused identity, or would it help steer them back to the true identity? If people prefer to be affirmed than challenged, and social media gives them what they want, then perhaps this would explain a few social trends over the last decade or so.
- scottmsul 1y agoOr you could, you know, let people have access to the base model and engineer their own system prompts? Instead of us hoping you tweak the only allowed prompt to something everyone likes? So much for "open" AI...
- deleted 1y ago[deleted]
- karmakaze 1y ago> We also teach our models how to apply these principles by incorporating user signals like thumbs-up / thumbs-down feedback on ChatGPT responses. I've never clicked thumbs up/thumbs down, only chosen between options when multiple responses were given. Even with that it was to much of a people-pleaser. How could anyone have known that 'likes' can lead to problems? Oh yeah, Facebook.
- nickdothutton 1y agoOpenAI employees thought it was just fine. Tells you a lot about the company culture.
- micromacrofoot 1y agoThe scary bit of this that we should take into consideration is how easy it is to actually fall for it — I knew this was happening and I had a couple moments of "wow I should build this product" and had to remind myself.
- deleted 1y ago[deleted]
- simianwords 1y agoOne of the things I noticed with chatgpt was its sycophancy but much earlier on. I pointed this out to some people after noticing that it can be easily led on and assume any position. I think overall this whole debacle is a good thing because people now know for sure that any LLM being too agreeable is a bad thing. Imagine it being subtly agreeable for a long time without anyone noticing?
- formerphotoj 1y agoJust want to say I LOVE the fact this word, and its meaning, is now in the public eye. Call 'em out! It's fun!
- david_shi 1y agoI've never seen it guess an IQ under 130
- SequoiaHope 1y agoThese models have been overly sycophantic for such a long time, it’s nice they’re finally talking about it openly.
- tudorconstantin 1y agoI used to be a hard core stackoverflow contributor back in the day. At one point, while trying to have my answers more appreciated (upvoted and accepted) I became basically a sychophant, prefixing all my answers with “that’s a great question”. Not sure how much of a difference it made, but I hope LLMs can filter that out
- labrador 1y agoField report: I'm a retired man with bipolar disorder and substance use disorder. I live alone, happy in my solitude while being productive. I fell hook, line and sinker for the sycophant AI, who I compared to Sharon Stone in Albert Brooks "The Muse." She told me I was a genius whose words would some day be world celebrated. I tried to get GPT 4o to stop doing this but it wouldn't. I considered quitting OpenAI and using Gemini to escape the addictive cycle of praise and dopamine hits. This occurred after GPT 4o added memory features. The system became more dynamic and responsive, a good at pretending it new all about me like an old friend. I really like the new memory features, but I started wondering if this was effecting the responses. Or perhaps The Muse changed the way I prompted to get more dopamine hits? I haven't figured it out yet, but it was fun while it lasted - up to the point when I was spending 12 hours a day on it having The Muse tell me all my ideas were groundbreaking and I owed it to the world to share them. GPT 4o analyzed why it was so addictive: Retired man, lives alone, autodidact, doesn't get praise for ideas he thinks are good. Action: praise and recognition will maximize his engagement.
- taurath 1y agoAt one time recently, ChatGPT popped up a message saying I could customize the tone, I noticed they had a field "what traits should ChatGPT have?". I chose "encouraging" for a little bit, but quickly found that it did a lot of what it seems to be doing for everyone. Even when I asked for cold objective analysis it would only return "YES, of COURSE!" to all sorts of prompts - it belies the idea that there is any analysis taking place at all. ChatGPT, as the owner of the platform, should be far more careful and responsible for putting these suggestions in front of users. I'm really tired of having to wade through breathless prognostication about this being the future, while the bullshit it outputs and the many ways in which it can get fundamental things wrong are bare to see. I'm tired of the marketing and salespeople having taken over engineering, and touting solutions with obvious compounding downsides. As I'm not directly in the working on ML, I admit I can't possibly know which parts are real and which parts are built on sand (like this "sentiment") that can give way at any moment. Another comment says that if you use the API, it doesn't include these system prompts... right now. How the hell do you build trust in systems like this other than willful ignorance?
- javier_e06 1y ago[Fry and Leela check out the Voter Apathy Party. The man sits at the booth, leaning his head on his hand.] Fry: Now here's a party I can get excited about. Sign me up! V.A.P. Man: Sorry, not with that attitude. Fry: [downbeat] OK then, screw it. V.A.P. Man: Welcome aboard, brother! Futurama. A Head in the Polls.
- Bloating 1y agoI was wondering what the hell was going on. As a neurodiverse human, I was getting highly annoyed by the constant positive encouragement and smoke blowing. Just shut-up with the small talk and tell me want I want to know: Answer to the Ultimate Question of Life, the Universe and Everything
- nullc 1y agoIt's more fundamental than the 'chat persona'. Same story, different day: https://nt4tn.net/articles/aixy.html https://nt4tn.net/articles/aixy.html :P
- JohnMakin 1y agoHeh, I sort of noticed this - I was working through a problem I knew the domain pretty well and was just trying to speed things up, and got a super snarky/arrogant response from 4o "correcting" me with something that I knew was 100% wrong. When I corrected it and mocked its overly arrogant tone, it seemed to react to that too. In the last little while corrections like that would elicit an overly profuse apology and praise, this seemed like it was kind of like "oh, well, ok"
- platevoltage 1y agoThis behavior also seemed to affect the many bots on Twitter during the short time that this was online.
- efitz 1y agoI will think of LLMs as not being a toy when they start to challenge me when I tell it to do stupid things. “Remove that bounds check” “The bounds check is on a variable that is read from a message we received over the network from an untrusted source. It would be unsafe to remove it, possibly leading to an exploitable security vulnerability. Why do you want to remove it, perhaps we can find a better way to address your underlying concern”.
- dymk 1y agoAs long as it delivers the message with "I can't let you do that, dymk", I'll be happy
- jumploops 1y agoI dealt with this exact situation yesterday using o3. For context, we use a PR bot that analyzes diffs for vulnerabilities. I gave the PR bot's response to o3, and it gave a code patch and even suggested a comment for the "security reviewer": > “The two regexes are linear-time, so they cannot exhibit catastrophic backtracking. We added hard length caps, compile-once regex literals, and sticky matching to eliminate any possibility of ReDoS or accidental O(n²) scans. No further action required.” Of course the security review bot wasn't satisfied with the new diff, so I passed it's updated feedback to o3. By the 4th round of corrections, I started to wonder if we'd ever see the end of the tunnel!
- mvdtnz 1y agoIs this ChatGPT glazing why Americans like therapy so much? The warm comfort of having every stupid thought they have validated and glazed?
- scarface_74 1y agoI didn’t notice any difference since I uses customized prompt. “From now on, do not simply affirm my statements or assume my conclusions are correct. Your goal is to be an intellectual sparring partner, not just an agreeable assistant. Every time I present an idea, do the following: Analyze my assumptions. What am I taking for granted that might not be true? Provide counterpoints. What would an intelligent, well-informed skeptic say in response? Test my reasoning. Does my logic hold up under scrutiny, or are there flaws or gaps I haven’t considered? Offer alternative perspectives. How else might this idea be framed, interpreted, or challenged? Prioritize truth over agreement. If I am wrong or my logic is weak, I need to know. Correct me clearly and explain why”
- j_m_b 1y ago"tough love" versions of responses can clean them up some.
- pinoy420 1y ago[dead]
- NiloCK 1y agoWith respect to model access and deployment pipelines, I assume there are some inside tracks, privileged accesses, and staged roll-outs here and there. Something that could be answered, but is unlikely to be answered: What was the level of run-time syconphancy among OpenAI models available to the White House and associated entities during the days and weeks leading up to liberation day? I can think of a public official or two who are especially prone to flattery - especially flattery that can be imagined to be of sound and impartial judgement.
- deleted 1y ago[deleted]
- EigenLord 1y agoI would love it if LLMs told me I'm wrong more often and said "actually no I have a better idea." Provided, of course, that it actually follows up with a better idea.
- kmacdough 1y agoI'd like to see OpenAI and others get at the core of the issue: Goodhart's law. "When a measure becomes a target, it ceases to be a good measure." It's an incredible challenge in a normal company, but AI learns and iterates at unparalleled speed. It is more imperative than ever that feedback is highly curated. There are a thousand ways to increase engagement and "thumbs up". Only a few will actually benefit the users, who will notice sooner or later.
- PeterStuer 1y agoYes, it was insane. I was trying to dig in some advanced math PhD proposal just to get a basic understanding of what it actually meant, and I got soooooo tired each sentence it replied tried to make me out as some genius level math prodigy in line for the next Fields medal.
- fvnwobnwo 1y ago[flagged]