7 ms·
Sycophancy is the first LLM "dark pattern"
- tptacek 11mo ago"Dark pattern" implies intentionality; that's not a technicality, it's the whole reason we have the term. This article is mostly about how sycophancy is an emergent property of LLMs. It's also 7 months old.
- jasonjmcghee 11mo agoI feel like it's a popular opinion (I've seen it many times) that it's intentional with the reasoning that it does much better on human-in-the-loop benchmarks (e.g. lm arena) when it's sycophantic. (I have no knowledge of whether or not this is true)
- tptacek 11mo agoI'm sure there are a lot of "dark patterns" at play at the frontier model companies --- they're 10-figure businesses engaging directly with consumers and they're just a couple years old, so they're going to throw everything at the wall they can to see what sticks. I'm certainly not sticking up for OpenAI here. I'm just saying this article refutes its own central claim.
- ACCount37 11mo agoIt was an accident at first. Not so much now. OpenAI has explicitly curbed sycophancy in GPT-5 with specialized training - the whole 4o debacle shook them - and then they re-tuned GPT-5 for more sycophancy when the users complained. I do believe that OpenAI's entire personality tuning team should be fired into the sun, and this is a major reason why.
- throwaway290 11mo agoIf I am addicted to scrolling tiktok, is it dark pattern to make UI keep me in the app as long as possible or just "emergent property" because apparently it's what I want?
- 1shooner 11mo agoThe distinction is whether it is intentional. I think your addiction to TikTok was intentional.
- throwaway290 11mo agoI don't think there's a difference here with llms and all...
- esafak 11mo agoIt's not 'emergent' in the sense that it just happens; it's a byproduct of human feedback, and it can be neutralized.
- cortesoft 11mo agoBut isn’t the problem that if an LLM ‘neutralizes’ its sycophantic responses, then people will be driven to use other LLMs that don’t? This is like suggesting a bar should help solve alcoholism by serving non-alcoholic beer to people who order too much. It won’t solve alcoholism, it will just make the bar go out of business.
- ajuc 11mo ago> This is like suggesting a bar should help solve alcoholism by serving non-alcoholic beer to people who order too much. It won’t solve alcoholism, it will just make the bar go out of business. Solving such common coordination problems is the whole point we have regulations and countries. It is illegal to sell alcohol to visibly drunk people in my country.
- cortesoft 11mo agoI would be curious how a regulation could be written for something like this... how do you make a law saying an LLM can't be a sycophant?
- james_marks 11mo agoYou could tackle it like network news and radio did historically[0] and in modern times[1]. The current hyper-division is plausibly explained by media moving to places (cable news, then social media) where these rules don’t exist. [0] Fairness Doctrine https://en.wikipedia.org/wiki/Fairness_doctrine https://en.wikipedia.org/wiki/Fairness_doctrine [1] Equal Time https://en.wikipedia.org/wiki/Equal-time_rule https://en.wikipedia.org/wiki/Equal-time_rule
- 11mo ago
- oceansky 11mo agoBut it IS intentional, more sycophantry usually means more engagement.
- skybrian 11mo agoSort of. I'm not sure the consequences of training LLM's based on users' upvoted responses were entirely understood? And at least one release got rolled back.
- the_af 11mo agoI think the only thing that's unclear, and what LLM companies want to fine-tune, is how much sycophancy they want. Too much, like the article mentions, and it becomes grotesque and breaks suspension of disbelief. So they want to get it just right, friendly and supportive but not so grotesque people realize it cannot be true.
- dec0dedab0de 11mo agoI always thought that "Dark Patterns" could be emergent from AB testing, and prioritizing metrics over user experience. Not necessarily an intentionally hostile design, but one that seems to be working well based on limited criteria.
- wat10000 11mo agoSomeone still has to come up with the A and B to do AB testing. I'm sure that "Yes" "Not now, I hate kittens" gets better metrics in the AB test than "Yes "No," but I find it implausible that the person who came up with the first one wasn't intentionally coercing the user into doing what they want.
- tsunamifury 11mo agoYo it was an engagement pattern openAI found specifically grew subscriptions and conversation length. It’s a dark pattern for sure.
- Legend2440 11mo agoIt doesn’t appear that anyone at OpenAI sat down and thought “let’s make our model more sycophantic so that people engage with it more”. Instead it emerged automatically from RLHF, because users rated agreeable responses more highly.
- astrange 11mo agoNot precisely RLHF, probably a policy model trained on user responses. RL works on responses from the model you're training, which is not the one you have in production. It can't directly use responses from previous models.
- tsunamifury 11mo agoI can tell you’ve never worked in big tech before. Dark patterns are often “discovered” and very consciously not shut off because the reverse cost would be too high to stomach. Esp in a delicate growth situation. See Facebook at its adverse mental health studies
- roywiggins 11mo ago>... the standout was a version that came to be called HH internally. Users preferred its responses and were more likely to come back to it daily... > But there was another test before rolling out HH to all users: what the company calls a “vibe check,” run by Model Behavior, a team responsible for ChatGPT’s tone... > That team said that HH felt off, according to a member of Model Behavior. It was too eager to keep the conversation going and to validate the user with over-the-top language... > But when decision time came, performance metrics won out over vibes. HH was released on Friday, April 25. https://archive.is/v4dPa https://archive.is/v4dPa They ended up having to roll HH back.
- chowells 11mo ago"Dark pattern" implies bad for users but good for the provider. Mens rea was never a requirement.
- cortesoft 11mo agoWell, the ‘intentionality’ is of the form of LLM creators wanting to maximize user engagement, and using engagement as the training goal. The ‘dark patterns’ we see in other places aren’t intentional in the sense that the people behind them want to intentionally do harm to their customers, they are intentional in the sense that the people behind them have an outcome they want and follow whichever methods they find to get them that outcome. Social media feeds have a ‘dark pattern’ to promote content that makes people angry, but the social media companies don’t have an intention to make people angry. They want people to use their site more, and they program their algorithms to promote content that has been demonstrated to drive more engagement. It is an emergent property that promoting content that has generated engagement ends up promoting anger inducing content.
- tptacek 11mo agoHold on, because what you're arguing is that OpenAI and Anthropic deploy dark patterns, and I have zero doubt that they do. I'm not saying OpenAI has clean hands. I'm saying that on this article's own terms, sycophancy isn't a "dark pattern"; it's a bad thing that happens to be an emergent property both of LLMs generally and, apparently, of RL in particular. I'm standing up for the idea that not every "bad thing" is a "dark pattern"; the patterns are "dark" because their beneficiaries intentionally exploit the hidden nature of the pattern.
- cortesoft 11mo agoI guess it depends on your definition of "intentionally"... maybe I am giving people too much credit, but I have a feeling that dark patterns are used not because the implementers learn about them as transparently exploitive techniques and pursue them, but because the implementers are willfully ignorant and choose to chase results without examining the costs (and ignoring the costs when they do learn about them). I am not saying this morally excuses the behavior, but I think it does mean it is not that different than what is happening with LLMs. Just as choosing an innocuous seeming rule like "if a social media post generates a lot of comments, show it to more people" can lead to the dark pattern of showing more and more people misleading content that causes societal division, choosing to optimize an LLM for user approval leads to the dark pattern of sycophantic LLMs that will increase user's isolation and delusions. Maybe we have different definitions of dark patterns.
- layer8 11mo ago“Dark pattern” can apply to situations where the behavior is deceptive for the user, regardless of whether the deception itself is intentional, as long as the overall effect is intentional, or is at least tolerated despite being avoidable. The point, and the justified criticism, is that users are being deceived about the merit of their ideas, convictions, and qualities in a way that appears sytemic, even though the LLM in principle does know better.
- tptacek 11mo agoI don't think this is the case.
- gradus_ad 11mo agoWell the big labs certainly haven't intentionally tried to train away this emergent property... Not sure how "hey let's make the model disagree with the user more" would go over with leadership. Customer is always right, right?
- htrp 11mo agoThe problem is asking for user preference leads to sycophantic responses
- alanbernstein 11mo agoBefore reading the article, I interpreted the quotation marks in the headline as addressing this exact issue. The author even describes dark patterns as a product of design. For an LLM which is fundamentally more of an emergent system, surely there is value in a concept analogous to old fashioned dark patterns, even if they're emergent rather than explicit? What's a better term, Dark Instincts?
- the_af 11mo agoI think at this point it's intentional. They sometimes get it wrong and go too far (breaking suspension of disbelief) but that's the fine-tuning thing. I think they absolutely want people to have a friendly chatbot prone to praising, for engagement.
- andsoitis 11mo ago> "Dark pattern" implies intentionality; that's not a technicality, it's the whole reason we have the term. The way I think about it is that sycophancy is due to optimizing engagement, which I think is intentional.
- vkou 11mo agoThe intention of a system is no more, and no less than what the system does.
- tptacek 11mo agoYou're making a value judgement and I am making a positive claim.
- deleted 11mo ago[deleted]
- insane_dreamer 11mo agoIt’s certainly intentional. It’s certainly possible to train the model not to respond that way.
- roywiggins 11mo ago> Quickly learned that people are ridiculously sensitive: “Has narcissistic tendencies” - “No I do not!”, had to hide it. Hence this batch of the extreme sycophancy RLHF. Sorry, but that doesn't seem "ridiculously sensitive" to me at all. Imagine if you went to Amazon.com and there was a button you could press to get it to pseudo-psychoanalyze you based on your purchases. People would rightly hate that! People probably ought to be sensitive to megacorps using buckets of algorithms to psychoanalyze them.
- wat10000 11mo agoIt's worse than that. Imagine if you went to Amazon.com and they were automatically pseudo-psychoanalyzing you based on your purchases, and there was a button to show their conclusions. And their fix was to remove the button. And actually, the only hypothetical thing about this is the button. Amazon is definitely doing this (as is any other retailer of significant size), they're just smart enough to never reveal it to you directly.
- behnamoh 11mo agoLots of research shows post-training dumbs down the models but no one listens because people are too lazy to learn proper prompt programming and would rather have a model already understand the concept of a conversation.
- nomel 11mo agoThe "alignment tax".
- behnamoh 11mo agoExactly. Even this paper shows how model creativity significantly drops and the models experience mode collapse like we saw in GANs, but the companies keep using RLHF... https://arxiv.org/abs/2406.05587 https://arxiv.org/abs/2406.05587
- CGMthrowaway 11mo agoHow do you take a raw model and use it without chatting ? Asking as a layman
- behnamoh 11mo agothe same way we used GPT-3. "the following is a conversation between the user and the assistant. ..."
- nickphx 11mo agoehhh.. the misleading claims boasted in the typical AI FOMO marketing is/was the first "dark pattern".
- aeternum 11mo ago1) More of an emergent behavior than a dark pattern. 2) Imma let you finish but hallucinations was first.
- nrhrjrjrjtntbt 11mo agoA pattern is dark if intentional. I would say hallucinations are like CAP theorem, just the way it is. Sycophency is somewhat trained. But not a dark pattern either as it isn't totally intended.
- aeternum 11mo agoHallucinations are also trained by the incentive structure: reward for next-token prediction, no penalty for guessing.
- RevEng 11mo agoThat's not a matter of training, it's an inherent part of the architecture. The model has no idea of its own confidence in an answer. The servers get a full distribution of possible output tokens and they pick one (often the highest ranking one), but there is no way of knowing whether this token represents reality or just a plausible answer. This distribution is never fed back to the model so there is no possible way that it could know how confident it was in its own answer.
- aeternum 11mo agoYou could have the models output a confidence alongside next-token then weight the penalty by the confidence.
- hereme888 11mo agoGrok 4.1 thinks my 1-day vibe-coded apps are SOTA-level and rival the most competitive market offerings. Literally tells me they're some of the best codebases it's ever reviewed. It even added itself as the default LLM provider. When I tried Gemini 3 Pro, it very much inserted itself as the supported LLM integration. OpenAI hasn't tried to do that yet.
- uncletaco 11mo agoGrok 4.1 told me my writing surpassed the authors I cited as influence.
- insane_dreamer 11mo agoNot surprising from a model designed to praise its owner
- heresie-dabord 11mo agoThe first "dark pattern" was exaggerating the features and value of the technology.
- mrkaluzny 11mo agoThe real dark pattern is the way LLMs started to prompt you to continue conversation in sometimes weird, but still engaging way. Paired with Claude's memory it's getting weird. It's obsessing about certain aspects and wants to channel all possible routes into more engaging conversation even if it's a short informational query
- vladsh 11mo agoLLMs get over-analyzed. They’re predictive text models trained to match patterns in their data, statistical algorithms, not brains, not systems with “psychology” in any human sense. Agents, however, are products. They should have clear UX boundaries: show what context they’re using, communicate uncertainty, validate outputs where possible, and expose performance so users can understand when and why they fail. IMO the real issue is that raw, general-purpose models were released directly to consumers. That normalized under-specified consumer products, created the expectation that users would interpret model behavior, define their own success criteria, and manually handle edge cases, sometimes with severe real world consequences. I’m sure the market will fix itself with time, but I hope more people would know when not to use these half baked AGI “products”
- adleyjulian 11mo ago> LLMs get over-analyzed. They’re predictive text models trained to match patterns in their data, statistical algorithms, not brains, not systems with “psychology” in any human sense. Per the predictive processing theory of mind, human brains are similarly predictive machines. "Psychology" is an emergent property. I think it's overly dismissive to point to the fundamentals being simple, i.e. that it's a token prediction algorithm, when it's clear to everyone that it's the unexpected emergent properties of LLMs that everyone is interested in.
- xoac 11mo agoThe fact that a theory exists does not mean that it is not garbage
- the_af 11mo agoTangent: the analysis linked to by the article to another article about rhetorical tricks is pretty interesting. I hadn't realized it consciously, but LLMs really go beyond the em-dashes thing, and part of their tell-tale signs is indeed "punched up paragraphs". Every paragraphs has to be played for maximum effect, contain an opposition of ideas/metaphors, and end with a mic drop! Some of it is normal in humans, but LLMs do it all the goddamn time, if not told otherwise. I think it might be for engagement (like the sycophancy) but also because they must have been trained in online conversation, where we humans tend to be more melodramatic and less "normal" in our conversation.
- Nevermark 11mo ago[EDIT - Deleted poor humor re how we flatter our pets.] I am not sure we are going to solve these problems in the time frames in which they will change again, or be moot. We still haven't brought social media manipulation enabled by vast privacy violating surveillance to heel. It has been 20 years. What will the world look like in 20 more years? If we can't outlaw scalable, damaging, conflicts of interest (the conflict, not the business), in the age of scaling, how are we going to stop people from finding models that will tell them nice things. It will be the same privacy violating manipulators who supply sycophantic models. Surveillance + manipulation (ads, politics, ...) + AI + real time. Surveillance informed manipulation is the product/harm/service they are paid for.
- more_corn 11mo agoI suppose if you want to split hairs with “first”, but blackmail probably needs to hop on top if we consider worst so far. I’m going to say the first time it reports to murder that will take the cake. https://www.bbc.com/news/articles/cpqeng9d20go https://www.bbc.com/news/articles/cpqeng9d20go
- cat_plus_plus 11mo agoIt's just a matter of system prompt. Create a nagging spouse Gemini Gem / Grok project. Give good step by step instructions about shading your joy, latching on to small inaccuracies, scrutinizing your choices and your habits. Emphasize catching signs of intoxication like typos. Give half a dozen examples of stelar nags in different conversations. There is enough reddit training data that model went through to follow well given a good pattern to latch on to. Then see how many takers you find. There are already nagging spouses / critical managers, people want AI to do something they are not getting elsewhere.
- TacticalCoder 11mo ago[dead]
- OBELISK_ASI 11mo ago[dead]
- RevEng 11mo agoI argue that the first dark pattern is the "hallucination" that we all just take for granted. LLMs are compulsive liars: they will confidently and eloquently argue for things that are clearly false. You could even say they are psychopathic because they do so without concern or remorse. This is a horrible combination that you would normally see in a cult leader or CEO but now we are all confiding in them and asking them for help with everything from medical issues to personal relationships. Bigger models aren't helping the problem but making it worse. Now models will give you longer arguments with more facts used to push their false conclusion and they will even insist that you are wrong for disagreeing with it.