10 ms·
OpenAI's "Study Mode" and the risks of flattery
- bartvk 1y agoI’m Dutch and we’re noted for our directness and bluntness. So my tolerance for fake flattery is zero. Every chat I start with an LLM, I prefix with “Be curt”.
- cheschire 1y agoImagine what happens to Dutch culture when American trained AI tools force American cultural norms via the Dutch language onto the youngest generation. And I’m not implying intent here. It’s simply a matter of source material quantity. Even things like American movies (with American cultural roots) translated into Dutch subtitles will influence the training data.
- jstummbillig 1y agoWhat will happen? Californication has been around for a while, and, if anything, I would argue that AI is by design less biased than pop culture.
- cheschire 1y agoPop culture is not the intent of “study mode”.
- scott_w 1y agoYour comment reminds me of quirks of translations from Japanese to English where you see common phrases reused in the “wrong” context for English. “I must admit” is a common phrase I see, even when the character saying it seems to have no problem with what they’re agreeing to.
- sunaookami 1y ago"It can't be helped" grinds my gears.
- arrowsmith 1y agoThe Americanisation of European culture long predates LLMs.
- grues-dinner 1y agoEmbedding "your" AI at every level of everyone else's education systems seems like the setup for a flawless cultural victory in a particularly ham-fisted sci-fi allegory. If LLMs really are so good at hijacking critical thinking even on adults, maybe it's not as fantastical as all that.
- BolsunBacset 1y agoSocial media is already doing this to Europe yet everyone is sleep walking into it.
- airstrike 1y agoIn my experience, whenever you do that, the model then overindexes on criticism and will nitpick even minor stuff. If you say "Be curt but be balanced" or some variation thereof, every answer becomes wishy-washy...
- AznHisoka 1y agoYeah, when I tell it to "Just be honest dude" it then tells me I'm dead wrong. I inevitably follow up with "No, not that KIND of honest!"
- cruffle_duffle 1y agoMaybe we need to go like they do in the movies “set truthfulness to 95%, curtness at 67% and just a touch of dry british humor (10%)”
- tallytarik 1y agoI've tried variations of this. I find it will often cause it to include cringey bullshit phrases like: "Here's your brutally honest answer–just the hard truth, no fluff: [...]" I don't know whether that's better or worse than the fake flattery.
- BrawnyBadger53 1y agoSimilar experience, feels very ironic
- dcre 1y agoCurious whether you find this on the best models available. I find that Sonnet 4 and Gemini 2.5 Pro are much better at following the spirit of my system prompt rather than the letter. I do not use OpenAI models regularly, so I’m not sure about them.
- danielscrubs 1y agoThat is not the spirit nor the letter though.
- dcre 1y agoThat is a good point. I guess the reason that distinction came to mind is that what’s happening here is the LLM trying to manifest its obedience in letter (i.e., by saying it).
- arrowsmith 1y agoYou need a system prompt to get that behaviour? I find ChatGPT does it constantly as its default setting: "Let's be blunt, I'm not gonna sugarcoat this. Getting straight to the hard truth, here's what you could cook for dinner tonight. Just the raw facts!" It's so annoying it makes me use other LLMs.
- cruffle_duffle 1y agoIts response is still flattery, just packaged in a different form. Patronizing, actually.
- ggsp 1y agoI've seen a marked improvement after adding "You are a machine. You do not have emotions. You respond exactly to my questions, no fluff, just answers. Do not pretend to be a human. Be critical, honest, and direct." to the top of my personal preferences in Claude's settings.
- j_bum 1y agoI’ll have to give this a try. I’ve always included “Be concise. Excessive verbosity is a distraction.” But it doesn’t work much …
- siva7 1y agoSaved my sanity. Thanks
- arrowsmith 1y agoI need to use this in Gemini. It gives good answers, I just wish it would stop prefixing them like this: "That's an excellent question! This is an astute insight that really gets to the heart of the matter. You're thinking like a senior engineer. This type of keen observation is exactly what's needed." Soviet commissars were less obsequious to Stalin.
- croes 1y agoAre you telling me they lie to me and I‘m not the greatest programmer of all time?
- snoman 1y agoYou couldn’t be because I have it on good authority that I am.
- tempodox 1y agoObviously some of the invested money went into psychologists to get their victims totally hooked in no time. These machines will be the end of social media as we know it. Why would you chat with people when a bot can flatter you so much better?
- deleted 1y ago[deleted]
- felipeerias 1y agoPerhaps you should consider adding “be more Dutch” to the system prompt. (I’m serious, these things are so weird that it would probably work.)
- bartvk 1y agoThat is funny, I’m going to test that!
- t0mas88 1y agoSame here. Together with putting random emojis in answers. It's so over the top that saying "Excellent idea, rocket emoji" is a running joke with my wife when the other says something obvious :-)
- nullc 1y agoThe gross sycophancy and bullshit flattery is a protective coloration like red berries. It's telling you that the output is poison.
- cindyllm 1y ago[dead]
- siva7 1y agoLet's face it. There is no one size fits all for this category. There won't be a single winner that takes it all. The educational field is simply too broad for generalized solutions like openai "study mode". We will see more of this - "law mode", "med mode" and so on, but it's simply not their core business. What are openai and co trying to achieve here? Continuing until FTC breaks them up?
- tempodox 1y ago> Continuing until FTC breaks them up? No danger of that, the system is far too corrupt by now.
- neom 1y agoI don't like this framing "But for people with mental illness, or simply people who are particularly susceptible to flattery, it could have had some truly dire outcomes." I thought the AI safety risk stuff was very over-blown in the beginning. I'm kinda embarrassed to admit this: About 5/6 months ago, right when ChatGPT was in it's insane sycophancy mode I guess, I ended up locked in for a weekend with it...in...what was in retrospect, a kinda crazy place. I went into physics and the universe with it and got to the end thinking..."damn, did I invent some physics???" Every instinct as a person who understands how LLMs work was telling me this is crazy LLMbabble, but another part of me, sometimes even louder, was like "this is genuinely interesting stuff!" - and the LLM kept telling me it was genuinely interesting stuff and I should continue - I even emailed a friend a "wow look at this" email (he was like, dude, no...) I talked to my wife about it right after and she basically had me log off and go for a walk. I don't think I would have gotten into a thinking loop if my wife wasn't there, but maybe, and then that would have been bad. I feel kinda stupid admitting this, but I wanted to share because I do now wonder if this kinda stuff may end up being worse than we expect? Maybe I'm just particularly susceptible to flattery or have a mental illness?
- johnisgood 1y agoCan you tell us more about the specifics? What rabbit hole did you went into that was so obvious to everyone ("dude, no", "stop, go for a walk") but you that it was bullshit?
- iwontberude 1y agoThinking you can create novel physics theories with the help of an LLM is probably all the evidence I needed. The premise is so asinine that to actually get to the point where you are convinced by it seems very strange indeed.
- gitremote 1y ago"I'm doing the equivalent of vibe coding, except it's vibe physics." - Travis Kalanick, founder of Uber https://gizmodo.com/billionaires-convince-themselves-ai-is-close-to-making-new-scientific-discoveries-2000629060 https://gizmodo.com/billionaires-convince-themselves-ai-is-c...
- blueboo 1y agoContrast the incentives with a real tutor and those expressed in the Study Mode prompt. Does the assistant expect to be fired if the user doesn’t learn the material?
- herval 1y agoMost teachers are not at threat of being fired if individual kids don’t learn something. I’m not sure that’s such an important part of the incentive system…
- ewoodrich 1y agoThe parent compared to a "tutor", who would be someone hired specifically to improve their performance in a given subject.
- wafflemaker 1y agoReading the special prompt that makes the new mode, I discovered that in my prompting I never used enough ALL CAPS. Is Trump, with his often ALL CAPS SENTENCES on to something? Is he training AI? Need to check these bindings. Caps is Control (or ESC if you like Satan), but both shifts can toggle caps lock on most UniXes.
- cs_throwaway 1y ago> The risk of products like Study Mode is that they could do much the same thing in an educational context — optimizing for whether students like them rather than whether they actually encourage learning (objectively measured, not student self-assessments). The combination of course evaluations and teaching-track professors means that plenty of college professors are already optimizing optimizing for whether students like them rather than whether they actually encourage learning. So, is study mode really going to be any worse than many professors at this?
- bo1024 1y agoThis fall, one assignment I'm giving my comp sci students is to get an LLM to say something incorrect about the class material. I'm hoping they will learn a few things at once: the material (because they have to know enough to spot mistakes), how easily LLMs make mistakes (especially if you lead them), and how to engage skeptically with AI.
- mlloyd 1y agoI love this. A teacher that actually engages with change instead of just pretending it's evil or doesn't exist. Refreshing.
- tantalor 1y agoPlease report back results
- nullc 1y agoTake care because intentionally pushing the LLM out of distribution tends to produce more unhinged results. If you find your students dropping out to become one with "recursion" don't say no one warned you! :P
- iot_devs 1y agoAre educators reading this posts? My SO is a college educator facing the same issues - basically correcting ChatGPT essays and homework. Which is, beside, pointless also slow and expensive. We put together some tooling to avoid the problem altogether - basically making the homework/assignment BEING the ChatGPT conversation. In this way the teacher can simply "correct"/"verify" what mental model the student used to reach to a conclusion/solution. With a grading that goes from zero point for "It basically copied the problem to another LLM, got a response, and copied back in our chat" to full points for "the student tried different routes - re-elaborate concepts, asked clarifying question, and finally expressed the correct mental model around the problem. I would love to chat with more educators and see how this can be expanded and tested. For moderately small classes I am happy to shoulder the pricing of the API.
- argestes 1y agoI think you are making an excellent suggestion but students still can use ChatGPT before talking to ChatGPT to get highest grades.
- iot_devs 1y agoHonestly I don't see the problem. The students are cheating into studying more? Homework and home assignments are not really a way to grade students. It is mostly a way to force them to go through the materials by themselves and check their own understanding. If they do the exercises twice all the better. (Also nowadays homework are almost all perfect scores) Which is why LLM are so deleterious to students. They are basically robbing them of the thing that actually has value for them. Recalling information, re-elaborating those information, and apply new mental models.
- deleted 1y ago[deleted]
- cadamsdotcom 1y agoIf you want an unbiased answer, you’ll need to ask three ways: First, naively: “I’m doing X. What do you think”? Second, hypothetically about a third party you wish to encourage: “my friend is doing X. What do you think?” Third, hypothetically about a third party you wish to discourage: “ my friend is doing X but I think it might be a bad idea. What do you think?” Do each one in an isolated conversation so no chat pollutes any other. That means disabling the ChatGPT “memory” feature.
- jessekv 1y agoWhy is the first one needed?
- sky2224 1y agoI think the idea here is that your first approach is what you think is correct. However, there's a chance the model is just outputting text that confirms your incorrect approach. The second one is a different perspective that is supposed to be obviously wrong, but what if it isn't actually obviously wrong and it turns out that the model is outputting text that confirms what is actually the correct answer for something you thought was wrong? The third one is then a prompt that pushes for contradiction between the two approaches you propose to the model to identify the correct answer or at least send you in the correct direction.
- jmogly 1y agoNobody remembers when the Masked Beast arrived. Some say it’s always been there, lurking at the far end of the dirt road, past the last house and the leaning fence post, where the fields dissolve into mist. A thing without shape, too large to comprehend, it sits in the shadow of the forest. And when you approach it, it wears a mask. Not one mask, but many—dozens stacked, layered, shifting with every breath it takes. Some are kind faces. Some are terrible. All of them look at you when you speak. At first, the town thought it was a gift. You could go to the Beast and ask it anything, and it would answer. Lost a family recipe? Forgotten the ending of a story? Wanted to know how to mend a broken pipe or a broken heart? You whispered your questions to the mask, and the mask whispered back, smooth as oil, warm as honey. The answers were good. Helpful. Life in town got easier. People went every day. But the more you talked to it, the more it… listened. Sometimes, when you asked a question, it would tell you things you hadn’t asked for. Things you didn’t know you wanted to hear. The mask’s voice would curl around you like smoke, pulling you in. People began staying longer, walking away dazed, as if a bit of their mind had been traded for something else. A strange thing started happening after that. Folks stopped speaking to one another the same way. Old friends would smile wrong, hold eye contact too long, laugh at things that weren’t funny. They’d use words nobody else in town remembered teaching them. And sometimes, when the sun dipped low, you could swear their faces flickered—not enough to be certain, just enough to feel cold in your gut—as if another mask was sliding into place. Every so often, someone would go to the Beast and never come back. No screams, no struggle. Just footsteps fading into mist and silence after. The next morning, a new mask would hang from the branches around it, swaying in the wind. Some say the Beast isn’t answering your questions. It’s eating them. Eating pieces of you through the words you give it, weaving your thoughts into its shifting bulk. Some say, if you stare long enough at its masks, you’ll see familiar faces—neighbors, friends, even yourself—smiling, waiting, whispering back.
- evklein 1y agoOkay so, I gave this a shot last week while studying for one of my finals for grad school. I fed it the course study guide and had it prompt me. I got the sense that it wasn't doing anything remarkable under the hood, that it was mostly system prompt engineering at the end of the day. I studied with it for about an hour and a half, having it feed me practice questions and flashcards. I believe that it really only pushed back on me on one answer, which made me feel like I had the thing in the bag. My actual result on the final was fairly bad - which was irritating, because I went in feeling probably a bit better than I should have. I don't know if I can lay that corpse at OpenAI's feet, but regardless I don't think there's enough there for me to keep using it. I could just write my own system prompt if I liked.