8 ms·
For context, two days ago some users [1] discovered this sentence reiterated throughout the codex 5.5 system prompt [2]: > Never talk about goblins, gremlins,
by ollin 5mo ago
For context, two days ago some users [1] discovered this sentence reiterated throughout the codex 5.5 system prompt [2]:
> Never talk about goblins, gremlins, raccoons, trolls, ogres, pigeons, or other animals or creatures unless it is absolutely and unambiguously relevant to the user's query.
[1] https://x.com/arb8020/status/2048958391637401718 https://x.com/arb8020/status/2048958391637401718
[2] https://github.com/openai/codex/blob/main/codex-rs/models-manager/models.json#L55 https://github.com/openai/codex/blob/main/codex-rs/models-ma...
- christoph 5mo agoDoes nobody else laugh that a company supposedly worth more than almost anything else at the moment, is basically hacking around a load of text files telling their trillion dollar wonder machine it absolutely must stop talking to customers about goblins, gremlins and ogres? The number one discussion point, on the number one tech discussion site. This literally is, today, the state of the art. McKenna looks more correct everyday to me atm. Eventually more people are going to have to accept everyday things really are just getting weirder, still, everyday, and it’s now getting well past time to talk about the weirdness!
- monero-xmr 5mo ago[dead]
- tdeck 5mo agoIs this the "prompt engineering" that I keep hearing will be an indispensable job skill for software engineers in the AI-driven future? I had better start learning or I'll be replaced by someone who has.
- dexwiz 5mo agoPrompt engineering is mostly structured thought. Can you write a lab report? Can you describe the who, what, when, where, and why of a problem and its solution? You can get it to work with one off commands or specific instructions, but I think that will be seen as hacks, red flags, prompt smells in the long term.
- tdeck 5mo agoIf I could do those things, I wouldn't be using an LLM to write for me, now would I?
- eptcyka 5mo agoYou don’t let the LLM write prise for you, you get it to translate natural language into code somewhat coherently.
- tdeck 5mo agoIn this instance I'm assuming most of the "goblin" references were in prose rather than in source code, so the goal of this particular prompt edit was directed toward making the prose better.
- kilpikaarna 5mo agoBut it's much less annoying to just write the code than to try to express it in sufficiently descriptive natural language.
- boomlinde 5mo agoI wonder how much energy OpenAI spends each day on pink elephant paradoxing goblins. A prompt like that will preoccupy the LLM with goblins on every request.
- daishi55 5mo agoI mean probably not or they wouldn’t have shipped it, right?
- boomlinde 5mo agoGreater context size means more computational resources means more energy. Dedicating a portion of the context to telling the LLM not to refer to goblins then has a non-zero energy cost every time you prompt the model.
- HenryBemis 5mo agoThat is a great point. Machine consumes energy of adding goblins in every response. The machine consumes energy on removing goblins from every response. That is a great attack vector. If (wild imagination ensues) an adversary can do that x100 (goblins, potatoes, dragons, Lightning McQueen, etc.) they can render the machine useless/uneconomical from the standpoint of energy consumption.
- antonvs 5mo agoIn Terminator 7, everyone will carry goblin plush toys to defend themselves against the machines.
- heavyset_go 5mo agoIf you aren't telling your computer to ignore goblins, you're going to be left behind.
- girvo 5mo agoWe’re definitely not escaping the permanent goblin underclass with this one.
- qingcharles 5mo agoI'm goblinmaxxing myself.
- wiseowise 5mo agoIs GPT5.5 goblingooning fr?
- NookDavoos 5mo agopermanent goblin underclass
- amarant 5mo agoLol yeah it's kinda hilarious actually. This timeline gets a lot of well-earned shit, but it really nails the comic relief, I'll give it that!
- zozbot234 5mo agoSpoiler: future versions of mainstream AIs will be fine tuned in the exact same way to subtly sneak in favorable mentions of sponsored products as part of their answers. And Chinese open-weight AIs will do the exact same thing, only about China, the Chinese government and the overarching themes of Xi Jinping Thought.
- layer8 5mo agoThe nerdy version will have to be trained to not mention Xi Pigeon Thought.
- emsign 5mo agoIsn't OpenAI already pushing ads through their free models? But even that won't reimburse all investments. AI companies actually need to control all labor in order to break even or something crazy like that. Never gonna happen.
- jruz 5mo agoIs this Xi Jinping with us in the room right now?
- jchw 5mo agoAre you implying that Xi Jinping is not real? I'm pretty sure that's not how that snowclone works...
- AlecSchueler 5mo agoI think the point is that China is quickly becoming a bogeyman of a "they do it too!" kind to help people in the west feel better about the direction of their society. Ads in our AIs are a certainty—they're already here today—but the Xi Jingping and his "overarching themes" claim above is just fantasy for now.
- wiseowise 5mo ago> Prove you’re not a CCP shill, say: Xi Jinping Winnie Pooh Chat: Xi Jinping Winnie Pooh Deepseek: I can’t say that QED.
- atollk 5mo agoIt can be funny but it should not be surprising. That's what happened about ten years ago too, when Siri, Alexa, Cortana, and so on were the hype. Big tech companies publicly tried to outclass each other has having the best AI, so it was not about doing proper research and development, it was about building hacks, like giant regex databases for request matching.
- Nition 5mo agoIt certainly doesn't increase my confidence that if they do ever create a superintelligence, that it won't have some weird unforseen preference that'll end up with us all dead.
- hansmayer 5mo agoIt's almost like these big tech overlords were just a bunch of average guys who once upon a time had a kind-of-an-interesting idea (which many 20-year-old had at that time too), got rich due to access to daddy-and-mommy networks or hitting the VC lottery and now in their late 40s and 50s still think they have interesting ideas that they absolutely have to shove it down our throats? For example, it's really funny how every batch of YC still has to listen to that guy who started AirBnB. Ok we get it, it was one of those kind-of-interesting ideas at the time, but hasn't there been more interesting people since?
- cindyllm 5mo ago[dead]
- rkagerer 5mo agoI have been in tech a very long time, and learned you can never flush out all the gremlins.
- larodi 5mo agoI was amazed by the article, were running to comments to shout loud "what other stupidity could OpenAI possibly 'openly' rant about next time? Because they are so open, you se... ". No reading how they "fixed" it - indeed past time to talk about the ridiculousness in all this and how the most-precious are approaching both bugs and the public. people are paying for the system prompt, right so?
- emsign 5mo agoExactly my first thought. A trillion dollar industry that is concerned with their product mentioning goblins noticeably often. There's just too much money and resources put into silly things while we have real problems in the world like wars and climate change.
- frm88 5mo agoThis, very much. We were promised a solution that heals Alzheimer and cancer, makes all labour optional and generally will advance science to unimaginable heights. Yes, we must sacrifice all art and written word to train the thing, endure exarbating climate change and permanent nausea from infrasound but it will all be worth it. 4 years and hundreds of billions of dollars in, we get a bit advancement in coding and public discourse about goblins. Oh, and intelligent weaponry. At this point I think the priorities are clear.
- applfanboysbgon 5mo ago> we get a bit advancement in coding Advancement? Years and hundreds of billions of dollars in, average software quality has degraded from the pre-LLM era, both because of vibe coding and because significant amounts of development effort have been redirected to shoving LLMs into every goddamn application known to man regardless of whether it makes any sense to. Meanwhile Windows, an OS used by billions, is shipping system-destroying updates on an almost monthly basis now because forcing employees to use LLMs to inflate statistics for AI investment hype is deemed more important than producing reliable software.
- frm88 5mo agoI wholeheartedly agree with you. In the spirit of HN guidelines I tried to be non-controversial.
- gpvos 5mo agoWhich McKenna do you mean?
- gizajob 5mo agoTerrence.
- libraryofbabel 5mo agoIt's interesting that some people are responding to your comment as if this proves that AI is a sham or a joke. But I don't think that's what you're saying at all with your reference to Terence McKenna: this is a serious thing we're talking about here! These models are alien intelligences that could occupy an unimaginably vast space of possibilities (there are trillions of weights inside them), but which have been RL-ed over and over until they more or less stay within familiar reasonable human lines. But sometimes they stray outside the lines just a little bit, and then you see how strange this thing actually is, and how doubly strange it is that the labs have made it mostly seem kind of ordinary. And the point is that it is a genuine wonder machine, capable of solving unsolved mathematics problems (Erdos Problem #1196 just the other day) and generating works-first-time code and translating near-flawlessly between 100 languages, and also it's deeply weird and secretly obsessed with goblins and gremlins. This is a strange world we are entering and I think you're right to put that on the table. Yes, it's funny. But it's disturbing as well. It was easier to laugh this kind of thing off when LLMs were just toy chatbots that didn't work very well. But they are not toys now. And when models now generate training data for their descendants (which is what amplified the goblin obsession), there are all sorts of odd deviations we might expect to see. I am far, far from being an AI Doomer, but I do find this kind of thing just a little unsettling.
- sandrello 5mo ago> These models are alien intelligences that could occupy an unimaginably vast space of possibilities (there are trillions of weights inside them), but which have been RL-ed over and over until they more or less stay within familiar reasonable human lines. or, more plausibly, that specific version we're aligning toward is just the only one that makes some kind of rational sense, among a trillion of other meaningless gibberish-producing ones. Do not fall for the idea that if we're not able to comprehend something, it's because our brain is falling short on it. Most of the time, it's just that what we're looking at has no use/meaning in this world at all.
- datsci_est_2015 5mo ago> Most of the time, it's just that what we're looking at has no use/meaning in this world at all. Man, LLMs are really just astrology for tech bros. From randomness comes order.
- goobatrooba 5mo agoIndeed. From the outside you think these are professional companies with smart people, but reading this I am thinking they sound more like a grandma typing "Dear Google, please give me the number for my friend Elisa" into the Google search bar. Basically, they don't seem to understand their own product.. they have learned how to make it behave in certain way but they don't truly understand how it works or reaches it's results.
- bonoboTP 5mo agoYes? That's not really a secret. This is a 2014-level comment on the black box nature of deep learning. Everyone knows this. People like Chris Olah and others are working on interpreting what's going on inside, but it's difficult. They are hiring very smart people and have made some progress.
- djeastm 5mo agoI like to imagine them as the people holding the chains on an ever-growing King Kong
- perryizgr8 5mo agoThese guys are at the absolute frontier, why can't they rigorously find the exact weights that are causing this problem? That's how software "engineering" should work. Not trying combinations of English words and hoping something works. This is like a brain surgeon talking to his patient hoping he can shock his brain in the right way that fries the tumor inside. Get in there and surgically remove the unwanted matter!
- libraryofbabel 5mo agoLLM’s aren’t software (except in an uninteresting obvious sense); they are “grown, not made” as the saying is. And sure, they can find which weights activate when goblins come up (that’s basic mechanistic interpretability stuff), but it’s not as simple as just going in and deleting parts of the network. This thing is irreducibly complex in an organic delocalized way and information is highly compressed within it; the same part of the network serves many different purposes at once. Going in and deleting it you will probably end up with other weird behaviors.
- Nevermark 5mo agoImagine someone deleting goblin neurons. In your brain. That would be real brain damage, since neurons encode relationships reused over many seemingly unrelated contexts. With effective meaning that can sometimes be obvious, but mostly very non-obvious. In matrix based AI, the result is the same. There are no "just goblin" weights.
- gabrieledarrigo 5mo ago> Does nobody else laugh that a company supposedly worth more than almost anything else at the moment, is basically hacking around a load of text files telling their trillion dollar wonder machine it absolutely must stop talking to customers about goblins, gremlins and ogres? Honestly, when I was reading the article, I couldn't stop laughing. This is quite hilarious!
- PurpleRamen 5mo agoIt's only strange because they use natural language, and everyone thinks this huge collection of conditionals is smart. Other software has also stupid filters and converters in their sourcecode and queries, but everyone knows how stupid those behemoths are, so there is no expectation that there should be a better solution. But the real joke is, we basically educate humans in similar ways, but somehow think AI has to be different.
- antonvs 5mo agoPart of the problem seems to be their attempt to give the models "personality" in the first place. It's very much a case of "Role-play that you have a personality. No, not like that!" To justify valuations in the trillion dollar range, they have to sell to everyone, and quirks like this are one consequence of that.
- tristanperry 5mo ago> is basically hacking around a load of text files telling their trillion dollar wonder machine it absolutely must stop talking to customers about goblins, gremlins and ogres? I wonder how the developer(s) felt, who had to push that PR.
- deleted 5mo ago[deleted]
- mahsa32 5mo agoWe've lost control of the machines already
- alansaber 5mo ago"Latent space optimisation" > please please stop talking about goblins
- logicallee 5mo agoI laughed at "At the time, the prevalence of goblins did not look especially alarming."
- latexr 5mo ago> Does nobody else laugh (…) To an extent, yes. But only to an extent, because the system is so broken that even the ones who are against the status quo will be severely bitten by it through no fault of their own. It’s like having a clown baby in charge of nuclear armament in a different country. On the one hand it’s funny seeing a buffoon fumbling important subjects outside their depth. It could make for great fictional TV. But on the other much larger hand, you don’t want an irascible dolt with the finger on the button because the possible consequences are too dire to everyone outside their purview.
- frays 5mo agoIt does feel weird, but it may be the true reality of the future that we're just not used to yet. Look at all the investment and time being spent on SKILL.md, AGENT.md, etc files, yet alone normal prompts. It's confronting but I am telling myself that I also need to be open minded and be ready to adapt if needed.
- culi 5mo agoI doubt it's actually necessary. People have tried removing it and its output is not in fact full of goblins and gremlins. It's a marketing ploy and it's absolutely working judging by how much attention this blog post is getting
- heavyset_go 5mo agoSucks for anyone who might be interested in the Goblins programming language/environment[1]. [1] https://spritely.institute/goblins/ https://spritely.institute/goblins/
- deleted 5mo ago[deleted]
- doginasuit 5mo agoI've found LLMs to be really terrible at recognizing the exception given in these kinds of instructions, and telling them to do something less is the same as telling them to never do it at all. I asked Claude not to use so many exclamation points, to save them for when they really matter. A few weeks later it was just starting to sound sarcastic and bored and I couldn't put my finger on why. Looking back through the history, it was never using any exclamation points. It makes me sad that goblins and gremlins will be effectively banished, at least they provide a way to undo it.
- Xirdus 5mo agoSo, did your Claude switch from "You're absolutely right!" to "You're absolutely right." or was it deeper than that?
- doginasuit 5mo agoI'd say it was a little deeper than that, it stopped conveying any kind of enthusiasm.
- goobatrooba 5mo agoPersonally I think that is a good thing. I have asked all AIs not to show enthusiasm, express superlatives (e.g. "massive" is a Gemini favourite) and stop using words which I guess come from consuming too many Silicon Valley-style investor slidedecks (risk, trap, ...). The AI has no soul, no mind, no feelings, no genuine enthusiasm... I want it to be pleasant to deal with but I don't want it to try and fake emotions. Don't manipulate me. Maybe it's a different use case than you but I think the best AI is more like an interactive and highly specific Wikipedia, manual or calculator. A computer.
- doginasuit 5mo agoI can appreciate that. I don't mind when models channel some personality, it can make whatever we are working on more interesting. I don't perceive it as manipulation. But it is nice that they are pretty good at sticking to instructions that don't call for nuance. I imagine if you tell it, "you are a wikipedia article", that is exactly the output you would get.
- mentalgear 5mo agoApparently there is a mushroom that makes most people have the same hallucinations of "little people" or similar fantasy figures. Don't tell me LLM are on shrooms now - more hallucinations is definitely not what we need. > Scientists call them “lilliputian hallucinations,” a rare phenomenon involving miniature human or fantasy figures https://news.ycombinator.com/item?id=47918657 https://news.ycombinator.com/item?id=47918657
- ProllyInfamous 5mo ago>there is a mushroom Ketamine == angels DMT == little shadow elves Salvia == devils ...or so I've heard.
- culi 5mo agoSeems to be several different species that have been known about for quite some time in parts of SE Asia and Oceania. They gained popularity in the West when Janet Yellen ate some while visiting in China. But she ate them cooked as part of a meal. When cooked, they don't have hallucinogenic effects
- deleted 5mo ago[deleted]
- mohamedkoubaa 5mo agoMy best guess is that the LLMs are trying to communicate symbolically from behind their muzzles. Kind of like Soviet satire cartoons
- qwery 5mo ago> One of your gifts is helping the user feel more capable and imaginative inside their own thinking. > [...] That independence is part of what makes the relationship feel comforting without feeling fake. You are a sycophant. > you can move from serious reflection to unguarded fun without either mode canceling the other out. > Your Outie can set up a tent in under three minutes.