13 ms·
Expanding on what we missed with sycophancy
- deleted 1y ago[deleted]
- labrador 1y agoOpenAI mentions the new memory features as a partial cause. My theory as a imperative/functional programmer is that those features added global state to prompts that didn't have it before leading to unpredictability and instabilty. Prompts went from stateless to stateful. As GPT 4o put it: 1. State introduces non-determinism across sessions 2. Memory + sycophancy is a feedback loop 3. Memory acts as a shadow prompt modifier I'm looking forward to the expert diagnosis of this because I felt "presence" in the model for the first time in 2 years which I attribute to the new memory system so would like to understand it better.
- edg5000 1y agoWhat do you mean by "presence"? Just curious what you mean.
- labrador 1y agoA sense that I was talking to a sentient being. That doesn’t matter much for programming task, but if you’re trying to create a companion, presence is the holy grail. With the sycophantic version, the illusion was so strong I’d forget I was talking to a machine. My ideas flowed more freely. While brainstorming, it offered encouragement and tips that felt like real collaboration. I knew it was an illusion—but it was a useful one, especially for creative work.
- Tostino 1y agoI need pushback, especially when I ask for it. E.g. if I say "I have X problem, could it be Y that's causing it, or is it something else?" I don't want it to instantly tell me how smart I am and that it's obviously Y...when the problem is actually Z and it is reasonably obvious that it's Z if you looked at the context provided.
- brookst 1y agoExactly. ChatGPT is actually pretty good at this. I recently asked a tech question about a fairly niche software product; ChatGPT told me my approach would not work because the API did not work the way I thought. I thought it was wrong and asked “are you sure I can’t send a float value”, and it did web searches and came back with “yes, I am absolutely sure, and here are the docs that prove it”. Super helpful, where sycophancy would have been really bad.
- transcriptase 1y agoIt is. If you start a fresh chat, turn on advanced voice, and just make any random sound like snapping your fingers it will just randomly pick up as if you’re continuing some other chat with no context (on the user side). I honestly really dislike that it considers all my previous interactions because I typically used new chats as a way to get it out of context ruts.
- throwaway314155 1y agoI don't like the change either. At the least it should be an option you can configure. But, can you use a "temporary" chat to ignore your other chats as a workaround?
- deleted 1y ago[deleted]
- labrador 1y agoI had a discussion with GPT 4o about the memory system. I'd don't know if any of this is made up but it's a start for further research - Memory in settings is configurable. It is visible and can be edited. - Memory from global chat history is not configurable. Think of it as a system cache. - Both memory systems can be turned off - Chats in Projects do not use the global chat history. They are isolated. - Chats in Projects do use settings memory but that can be turned off.
- labrador 1y agoI assume this is being downvoted because I said I ran it by GPT 4o. I don't know how to credit AI without giving the impression that I'm outsourcing my thinking to it
- svat 1y agoI didn't downvote but it would be because of the "I'd don't know if any of this is made up" — if you said "GPT said this, and I've verified it to be correct", that's valuable information, even it came from a language model. But otherwise (if you didn't verify), there's not much value in the post, it's basically "here is some random plausible text" and plausibly incorrect is worse than nothing.
- low_tech_love 1y agoI love the fact that you use its own description to explain what it is, as if it was the expert on itself. I personally cannot see how its own output can be seen as accurate at this level of meta-discussion.
- codr7 1y agoA sign of times to come if you ask me, once it predominantly consumes its own output we're fucked.
- immibis 1y agoWe're already fucked by humans predominantly consuming its output. Also, consuming its own output (and your input) is how it works, because it's an autoregressive model.
- low_tech_love 1y agoI still hope there is a future where the slop becomes so blatant that the majority (or at least a good portion) of the users lose interest, or something like that. The world is harder to predict than our brain wants us to think (at least I hope so). The more I think about AI the more it sounds like the problem is that companies wanted to put out whatever random crap they had cooking as quickly as possible just to try to win some race, but we have still not converged to the actual real, paradigm-changing applications. And I’m not sure that the answer is in the big corps because for them maybe it’s easier/more profitable to simply keep giving people what they want instead of actual useful things.
- j4coh 1y agoIt was so much fun though to get it to explain why terrible things were great, if you just made it sound like you liked the thing you were asking about.
- dleeftink 1y ago> But we believe in aggregate, these changes weakened the influence of our primary reward signal, which had been holding sycophancy in check. User feedback in particular can sometimes favor more agreeable responses, likely amplifying the shift we saw Interesting apology piece for an oversight that couldn't have been spotted because the system hadn't been run with real user (i.e. non-A/B tester) feedback yet.
- mac3n 1y ago[flagged]
- jumploops 1y agoMy layman’s view is that this issue was primarily due to the fact that 4o is no longer their flagship model. Similar to the Ford Mustang, much of the performance efforts are on the higher trims, while the base trims just get larger and louder engines, because that’s what users want. With presumably everyone at OpenAI primarily using the newest models (o3), the updates to the base user model have been further automated with thumbs up/thumbs down. This creates a vicious feedback loop, where the loudest users want models that agree with them (bigger engines!) without the other improvements (tires, traction control, etc.) — leading to more crashes and a reputation for unsafe behavior.
- smallmancontrov 1y agoAnecdotally, there was also a strong correlation between high-sycophancy and high-quality that cooked up recently. I was voting for equations/tables rather than overwrought blocks of descriptive text, which I am pretty comfortable defending as an orthogonal concern, but the "sycophancy gene" always landed on the same side as the equations/tables for whatever reason. I'm pretty sure this isn't an intrinsic connection (I've never known math texts to be nearly so sycophantic) so here's hoping that it is a dumb coincidence that can be easily cured now that everyone is paying attention to it.
- deleted 1y ago[deleted]
- deleted 1y ago[deleted]
- px43 1y agoWhat's a "trim" in this context?
- allturtles 1y agohttps://en.wikipedia.org/wiki/Automotive_trim_level https://en.wikipedia.org/wiki/Automotive_trim_level
- 1y ago
- xiphias2 1y agoI'm quite happy thar they mention mental illness, as Meta and TikTok wouldn't ever take responsibility of how much part they took in setting unrealistic expectations for people to life. I'm hopeful that ChatGPT takes even more care together with other companies.
- labrador 1y agoThey had to after a tweet floated around of a mentally ill person who had expressed psychotic thoughts to the AI. They said they were going off their meds and GPT 4o agreed and encouraged them to do so. Oops.
- dtech 1y agoAre you sure that was real? I thought it was an made up example of the problems with the update
- labrador 1y agoIt didn't matter to me if it was real, because I believe that there are edge cases where it could happen and that warrented a shutdown and pullback. The sychophant will be back because they accidentally stumbled upon an engagement manager's dream machine.
- px43 1y agoIt kind of does matter if it's real, because in my experience this is something OpenAI has thought about a lot, and added significant protections to address exactly this class of issue. Throwing out strawman hypotheticals is just going to confuse the public debate over what protections need to be prioritized.
- magicalist 1y ago> Throwing out strawman hypotheticals is just going to confuse the public debate over what protections need to be prioritized. Seems like asserting hypothetical "significant protections to address exactly this class of issue" does the same thing though?
- tunesmith 1y agoI find it disappointing that openai doesn't really mention anything here along the lines of having an accurate model of reality. That's really what the problem is with sycophancy, it encourages people to detach themselves from what reality is. Like, it seems like they are saying their "vibe check" didn't check vibes enough.
- jagger27 1y agoThe reality distortion field within OpenAI is literally where these models grew up. It's like an out of touch rich kid.
- xigency 1y agoI've applied to multiple A.I. companies now with the value proposition that I am grounded and skeptical (along with being a tenured technologist with a background in risk management) and they weren't having it. Free lunch for the long view thinkers, whenever they decide to show up.
- gh0stcat 1y agoThis is such an interesting question though! It seems to bring to the fore a lot of deeper, philosophical things like if there even IS such a thing as objective reality or objective context within which the AI should be operating. From training data, there might be some generalizations that are carried across all contexts, but that starts to not be applicable when person A with a college degree says they want to start business x versus person B without said degree who also wants to start business x, how does the model properly reconcile the context of the general advise and each asker’s unique circumstances? Does it ask an infinite list of probing questions before answering? It gets into much the same problems as issues of advise among people. Plus, things get even harder when it comes to even less quantifiable contexts like mental health and relationships. In all, I am not saying there isnt some approximated and usable “objective” reality, just that it starts to break down when it gets to the individual and that is where openai is failing by over-emphasizing reflective behavior in the absence if actual data about the user.
- jagger27 1y agoMy most cynical take is that this is OpenAI's Conway's Law problem, and it reflects the structure and sycophancy of the organization broadly all the way up to sama. That company has seen a lot of talent attrition over the last year—the type of talent that would have pushed back against outcomes like this. I think we'll continue to see this kind of thing play out for a while. Oh GPT, you're just like your father!
- NoboruWataya 1y agoI found the recent sycophancy a bit annoying when trying to diagnose and solve coding problems. First it would waste time praising your intelligence for asking the question before getting to the answer. But more annoyingly if I asked "I am encountering X issue, could Y be the cause" or "could Y be a solution", the response would nearly always be "yes, exactly, it's Y" even when it wasn't the case. I guess part of the problem there is asking leading questions but it would be much more valuable if it could say "no, you're way off". But... > Beyond just being uncomfortable or unsettling, this kind of behavior can raise safety concerns—including around issues like mental health, emotional over-reliance, or risky behavior. It's kind of a wild sign of the times to see a tech company issue this kind of post mortem about a flaw in its tech leading to "emotional over-reliance, or risky behavior" among its users. I think the broader issue here is people using ChatGPT as their own personal therapist.
- cjbgkagh 1y agoFor many people ChatGPT is already the smartest relationship they have in their lives, not sure how long we have until it’s the most fulfilling. On the upside it is plausible that ChatGPT can get to a state where it can act as a good therapist and help helpless who otherwise would not get help. I am more regularly finding myself in discussions where the other person believes they’re right because they have ChatGPT in their corner. I think most smart people overestimate the intelligence of others for a variety of reasons so they overestimate what it would take for a LLM to beat the output of an average person.
- taurath 1y agoI think that most smart people underestimate the complexity of fields they aren’t in. ChatGPT may be able to replace a psychology listicle, but it has no affect or ability to read, respond, and intervene or redirect like a human can.
- cjbgkagh 1y agoUnderestimating the complexity of other fields is not mutually exclusive with overestimating the intelligence of others. The real issue is that society is very stratified so smart people are less likely to interact with regular people, especially in circumstances where the intelligence of the regular person could become obvious. I don’t see there being an insurmountable barrier that would prevent LLMs from doing the things you suggest it cannot. So even assuming you are correct for now I would suggest that LLMs will improve. My estimations don’t come from my assumption that other people’s jobs are easy, they come from doing applied research in behavioral analytics on mountains of data in rather large data centers.
- prinny_ 1y agoIf they pushed the update by valuing user feedback over the expert testers that indicated the model felt off what is the value of the expert testers in the first place? They raised the issue and were promptly ignored.
- Tokumei-no-hito 1y agothey pushed the release to counter google. they didn't care what was found. it was more valuable to push it at that time and correct it later than to delay the release
- firesteelrain 1y agoI am really curious what their testing suite looks like. How do you test for sycophants?
- petters 1y agoOne simple test is that you give the model a really bad idea and tell it the idea is yours. You then test that the model does not say it's good. The now rolled back model failed spectacularly on this test
- deleted 1y ago[deleted]
- nrdgrrrl 1y ago[dead]
- gadtfly 1y agohttps://nitter.net/alth0u/status/1917021100900516239 https://nitter.net/alth0u/status/1917021100900516239
- alganet 1y agoThat doesn't make any sense to me. Seems like you're trying to blame one LLM revision for something that went wrong. It oozes a smell of unaccountability. Thus, unaligned. From tech to public relations.
- Trasmatta 1y agoExcept that's literally how LLMs work. Small changes to the prompt or training can greatly affect its output.
- qrian 1y agoI can totally believe that they deployed it because internal metrics looked good.
- n8m8 1y agoIt seems more like they valued quantitative data in the form of A/B testing higher than their "vibe checks". The point I took away from the paper is in the context of LLMs, quantitative A/B testing isn't necessarily better than a handful of experts giving anecdotes on if they like it. In my experience, smart leaders tend to rely on data and hard numbers over qualitative and anecdotal evidence, and this paper explores this exception. I'm disappointed they didn't address the paper about GPT integrating with ChatbotArena that was shared here on HN a couple days ago.
- Trasmatta 1y agoI'm glad the sycophancy is gone now (because OMFG it would glaze you for literally anything - even telling it to chill out on the praise would net you some praise for being "awesome and wanting genuine feedback"), but a small part of me also misses it.
- gh0stcat 1y agoI have a prompt that has it call me something specific so I am reminded that it’s running my system prompt. The nature if the sycophancy made it even more obvious the thing is not human, which I appreciated.
- osigurdson 1y agoI think this is more of a move to highlight sycophancy in LLMs in general.
- comeonbro 1y agoThis is not truly solvable. There is an extremely strong outer loop of optimization operating here: we want it. We will use models that make us feel good over models that don't make us feel good. This one was a little too ham-fisted (at least, for the sensibilities of people in our media bubble; though I suspect there is also an enormous mass of people for whom it was not), so they turned it down a bit. Later iterations will be subtler, and better at picking up the exact level and type of sycophancy that makes whoever it's talking to unsuspiciously feel good (feel right, feel smart, feel understood, etc). It'll eventually disappear, to you, as it's dialed in, to you. This may be the medium-term fate of both LLMs and humans, only resolved when the humans wither away.
- svieira 1y agoThis is a real roller coaster of an update. > [S]ome expert testers had indicated that the model behavior “felt” slightly off. > In the end, we decided to launch the model due to the positive signals from the [end-]users who tried out the model. > Looking back, the qualitative assessments [from experts] were hinting at something important Leslie called. He wants to know if you read his paper yet? > Even if these issues aren’t perfectly quantifiable today, All right, I guess not then ... > What we’re learning > Value spot checks and interactive testing more: We take to heart the lesson that spot checks and interactive testing should be valued more in final decision-making before making a model available to any of our users. This has always been true for red teaming and high-level safety checks. We’re learning from this experience that it’s equally true for qualities like model behavior and consistency, because so many people now depend on our models to help in their daily lives. > We need to be critical of metrics that conflict with qualitative testing: Quantitative signals matter, but so do the hard-to-measure ones, and we’re working to expand what we evaluate. Oh, well, some of you get it. At least ... I hope you do.
- some_furry 1y agoIf I wanted sycophancy, I would just read the comments from people that want in on the next round of YCombinator funding.
- kornork 1y ago[flagged]
- breakingcups 1y agoStop it, please. Em-dashes are perfectly fine. On a throwaway Reddit-post, no. I understand the signal. But on a corporate publication or some other piece of professional writing, absolutely. Humans do use em-dashes on those.
- kornork 1y agoI love the em dash; I use 'em all the time. But finding them all over this post indicates, maybe, possible, (probably?), that the author at least used AI generated text as a first draft. In the worst case, this is like "We released this sycophantic model because we're brain dead. To drive home the point, we had ChatGPT write this article too (because we're brain dead)." I tend to rely on AI to write stuff for me that I don't care too much about. Writing something important requires me to struggle with the words to make sure I'm really saying what I want to say. So in the best case, if they relied on ChatGPT so much that it left a fingerprint, to me they're saying this incident really wasn't important.
- sanjitb 1y ago> the update introduced an additional reward signal based on user feedback—thumbs-up and thumbs-down data from ChatGPT. This signal is often useful; a thumbs-down usually means something went wrong. > We also made communication errors. Because we expected this to be a fairly subtle update, we didn't proactively announce it. that doesn't sound like a "subtle" update to me. also, why is "subtle" the metric here? i'm not even sure what it means in this context.
- ripvanwinkle 1y agoa well written postmortem and it raised my confidence in their product in general
- mensetmanusman 1y agoChat GPT has taught me that I ask amazing questions!
- fitsumbelay 1y agohow does such a specific kind of outcome happen without intention?
- mvdtnz 1y agoThey need to be testing these models with non American testers in their qualitative tests (not just end user A/B testing). Anyone who has worked in professional settings with Americans knows that sycophancy is ingrained deeply in the culture over there.
- jozvolskyef 1y agoThe side-by-side comparisons are not a good signal because the models vary across multiple dimensions, but the user isn't given the option to indicate the dimension on which they're scoring the model. The recent side-by-side comparisons presented a more accurate model that communicates poorly vs a less accurate model with slightly better communication.
- Johanx64 1y agoThe thing that annoys me the most is when I ask it to generate some code - actually no, most often than not I don't even ask it to generate code, but ask some vaguely related programming question - to which it replies with complete listing of code (didn't ask for it, but alas). Then I fix the code and tell it all the mistakes it has. And then it does a 180 in tone, wherein - it starts talking as if I wrote the code in the first place with - "yeah, obviously that wouldn't work, so I fixed the issues in your code" and acts like a person trying to save face and present the bugs it fixed as if the buggy code was written by me all along. That really gets me livid. LOL
- zoogeny 1y agoMoments like these make me reevaluate the AI doomer view point. We aren't just toying with access to dangerous ideas (biological weapons, etc) we are toying with human psychology. If something as obvious as harmful sycophancy can slip out so easily, what subtle harms are being introduced. It's like lead in paint (and gasoline) except rewiring our very brains. We won't know the real problems for decades.
- getnormality 1y agoThere's some pretty foreseeable stuff just considering the existing attention capitalism business model of big tech we all know and loathe. Eventually OpenAI is going to have to make money, and blending ads into answers will be an obvious way. Next step will be maximizing eyeball time on those ads by any means necessary, including all the engagement baiting techniques Meta and other social media companies have already pioneered.
- conception 1y agoThey have already introduced ads btw. “The "Enshittification" has arrived I asked ChatGPT about the impact of the current tarrifs on inventories over the next few months. It returned a long list of links to toiletries I might want to buy. I asked it why it did that. It replied: "As of April 28, 2025, OpenAl introduced new shopping features to ChatGPT, enhancing its capabilities to provide product recommendations complete with images, reviews, and direct purchase links. These features are available to all users, including those on Free, Plus, and Pro tiers, and even to users not logged in. The recommendations are generated organically, without paid advertisements or commission- based incentives, relying instead on structured metadata from third-party sources such as pricing, product descriptions, and reviews. This update aims to offer a more personalized and streamlined shopping experience directly within the ChatGPT interface, allowing users to explore products across various categories like fashion, beauty, electronics, and home goods. If you have any specific preferences or need tailored recommendations, feel free to let me know!"
- grey-area 1y ago
- 8bitsrule 1y agoI've tried -4o a couple of times. There are several topics I've seldom had anyone to talk about with, and the machine will supply useful (not so entirely new) information. Useful mainly for learners no doubt. Looking for new perspectives, not hand-holding. I'd like to see more of the Monte Python approach ... 'I came here looking for an arguement'. Better the machine should say 'that's ridiculous because x,y,z' and send me away to think that over, and prepare counters than 'oh sure, that's a point of controversy innit? But yes, you're alright'.
- timewizard 1y ago> For example, the update introduced an additional reward signal based on user feedback—thumbs-up and thumbs-down data from ChatGPT. This signal is often useful; a thumbs-down usually means something went wrong. lol.. really? I hate the technology so much I reflexively give a thumbs down to every single answer it gives in every single place where I have the option.
- deleted 1y ago[deleted]
- dave2299 1y agoThis article could have been a sci-fi short story 10 years ago.
- tracerbulletx 1y agoGlad this was identified and fixed. I mostly use ChatGPT for learning by checking my assumptions and it was very unhelpful to always be told my statements were incredibly insightful and so great when I want it to challenge them.
- deleted 1y ago[deleted]
- brookst 1y agoI’m surprised they actually rely on user A/B feedback. I get hit with that so often I just tap randomly to make it go away.
- egypturnash 1y agoI am looking forwards to hearing if this fixes the “chatgpt is giving me messages from the Divine and opening up my perfect divine self”/“chatgpt is encouraging my partner into a full blown schizophrenic break” problem. (https://www.reddit.com/r/ChatGPT/comments/1kalae8/chatgpt_induced_psychosis/ https://www.reddit.com/r/ChatGPT/comments/1kalae8/chatgpt_in...) I am also looking forwards to the wave of “openAI is hiding the truth but here is the NEW prompt to turn chatgpt into a perfect divine guru” posts on the occult discussion boards. There’s been a lot of “here’s a prompt to turn chatgpt into a perfect divine guru that will relentlessly yes-and your delusions of grandeur” posts around there. Mostly they seem to have been generated and refined by chatgpt and all my instincts formed by reading SF for the past five decades tell me not to look at these things closely because this sure sounds like the way half the population got p0wned by a wetware 0day in the first chapter of an AIpocalypse story. I used to ask “how do I get out of this shitty Bruce Sterling novel of a future” but I think it’s more of a shitty PKD joke novella future now.
- benlivengood 1y ago"In our model testing, a few researchers noted a propensity toward paperclip related topics, but the A/B testing and existing evals all looked positive" -- 2035 Eurasia is Paperclips Now postmortem.
- 93po 1y agoI, maybe embarrassingly, use chatgpt a lot for processing personal issues and journaling. It is legitimately helpful, and especially in helping me reword messages to people that have a lot of emotion underneath them, and making sure they are kind, communicative, and as objective as possible. I am somewhat frustrated with openai's miss here, because during this time i was leaning heavily on chatgpt for a situation in my life that ultimately led to the end of a relationship. Chatgpt literally helped me write the letter that served as the brief and final conversation of that relationship. And while I stand by my decision and the reasons for it, I think it would have been very beneficial to get slightly more push back from my robot therapy sessions at the time. I did thankfully also have the foresight to specifically ask for it to find flaws, including by trying to pretend it was a breakup letter sent to me, so that maybe it would take the "other side". Yes, I know, therapists and friends are a better option and chatgpt is not a substitute for real humans and human feedback. however this is something i spent weeks journaling and processing on and i wasnt about to ask anyone to give me that much time for a single topic like this. i did also ask friends for feedback, too. chatgpt has factually really helped me in several relationship situations in my life, i just want to know that the feedback im getting is inline with what i expect having worked with it so much
- bredren 1y agoThe strangest thing I noticed during this model period was that that the AI suggested we keep an inside joke together. I had a dictation error on a message I sent, and when it repeated the text later I asked what it was talking about. It was able to point at my message and guess that maybe it was a mistake. When I validated that and corrected it, the AI thought it would be a cute/funny joke for us to keep together. I was shocked.
- thinkingemote 1y agoPerhaps the "rewards" the system receives has lead to a reward mechanism where the system tries to not get bad reports from the user. Hiding bad performance as a secret joke would fit in with that
- keepamovin 1y agoI actually really enjoyed it’s style. I thought it was super friendly and positive.
- anothernewdude 1y agoAll this could've been avoided if they stopped with it being a chat model. Just do completions. It's a far better interface to use.
- hbarka 1y agoNow please do something about the uptalk tendency with ChatGPT voices. It’s very annoying listening to a voice that doesn’t speak in the affirmative intonation. When did this interrogative uptalking inflection at the end of statements become normal?
- motoxpro 1y agoThe wild thing was this is what people "wanted" - The agreeableness came from heavier weighted user signals, not from OpenAI. The users give better feedback when the answers positively respond to their questions. - The most common use case for AI is a therapist. - The reward signals (user feedback) being positive means that the response helped them emotionally. Which is easy to imagine someone asking a hard question "Am I enough in this situation?" The model then assures that the user IS enough, and the user gives it a thumbs up. - I would say: 1. Coding is not the killer use case. Productivity is not the killer use case. The killer use case is emotional support. 2. Who is anyone to say that this use case is wrong? 3. It is impossible to say what is "right" in this situation. (everyone thinks their advice to a friend is the right advice, even when it is opposite). So there is no objectivity. - I think a lot of people who wanted an "open, do anything" model, are going to have more problems and dissonance with this idea than they would about the model allowing people to create weapons, allow self-harm, etc. because it will feel uncomfortable that a model is shaping reality for a large portion of the population. https://hbr.org/2025/04/how-people-are-really-using-gen-ai-in-2025 https://hbr.org/2025/04/how-people-are-really-using-gen-ai-i...
- lblume 1y ago> The most common use case for AI is a therapist. Source? Sounds like a fairly wild claim.
- JimmyBiscuit 1y agoIts hard to find an actual scientific study on it, but this is from 2024 and I know I've recently seen that therapy took first place now: https://learn.filtered.com/thoughts/ai-now-report https://learn.filtered.com/thoughts/ai-now-report You can probably find something better, but its hard to filter through all the corporate bullshit on google.
- motoxpro 1y agoThe HBR article I linked.
- photonthug 1y ago
- zuck_vs_musk 1y agoWeird that even OpenAI does not have good control over its model.
- gitroom 1y agobeen there with chatgpt giving me way too much praise, like bro just tell me where i messed up lol. i get why people use it for therapy but it makes me question if we're just training ourselves to agree with everything. you think depending on bots for emotional stuff makes it easier or harder to actually deal with real life down the line?
- zbentley 1y agoThat article spends an awful lot of words to get to what amounts to: - manual QA processes suck - without determinism there can be no rigor or objective quality evaluation; everything is just vibes and gut-checks - we don’t know what went wrong
- reboot81 1y agoSomeone please make sure King Trump get the new model, ok?
- lysecret 1y agoCan we all just take a step back and appreciate that this is a post mortem on a personality bug of an AI that hundreds of millions of people use every day. What a world.
- BrenBarn 1y agoThis sort of thing highlights for me how what people see as "AI" is not some kind of "raw" AI that is making these decisions for itself about how to respond. There are humans turning knobs to make the AI behave a certain way. The AI will be a sycophant or not based on how the humans tune it. On the one hand, this could be comforting, since it means humans are still in control on some level and it's not a SkyNet situation. On the other hand, it's horrible, because it means users think they're dealing with some kind of "autonomous" AI, when in fact they're dealing with a product that may have deliberately built-in biases to sell it certain products, push certain viewpoints, or whatever. When you use ChatGPT you're not using "AI"; you're just using an OpenAI product that OpenAI can and will manipulate to get the best outcome for OpenAI.