16 ms·
Where the goblins came from
- maxdo 6mo agoarticle : bla blah blah, marketing... we are fun people, bla blah, goblin, we will not destroy the world you live in.. RL rewards bug is a culprit. blah blah.
- llbbdd 6mo agosomeone woke up on the wrong side of the goblin today
- blinkbat 6mo agoreal goblin-y response
- luke-stanley 6mo agoYeah, though it's not great marketing. Especially for hiring interpretability researchers. Their own alignment research has reward model interpretability, personality features and so on (see https://alignment.openai.com https://alignment.openai.com ). It just seems like a different department wrote it, which is a shame because I'd love to read about goblin feature vectors and functional emotions.
- soupspaces 6mo ago[dead]
- nomilk 6mo ago> We unknowingly gave particularly high rewards for metaphors with creatures. I recall a math instructor who would occasionally refer to variables (usually represented by intimidating greek letters) as "this guy". Weirdly, the casual anthropomorphism made the math seem more approachable. Perhaps 'metaphors with creatures' has a similar effect i.e. makes a problem seem more cute/approachable. On another note, buzzwords spread through companies partly because they make the user of the buzzword sound smart relative to peers, thus increasing status. (examples: "big data" circa 2013, "machine learning" circa 2016, "AI" circa 2023-present..). The problem is the reputation boost is only temporary; as soon as the buzzword is overused (by others or by the same individual) it loses its value. Perhaps RLHF optimises for the best 'single answer' which may not sufficiently penalise use of buzzwords.
- kybb4 6mo agoThey give everyone the false and very misleading impression that with One prompt all kinds of complexity minimizes. Its a bed time story for children. Ashby's Law of Requisite Variety asserts that for a system to effectively regulate or control a complex environment, it must possess at least as much internal behavioral variety (complexity) as the environment it seeks to control. This is what we see in nature. Massive variety. Thats a fundamental requirement of surviving all the unpredictablity in the universe.
- LifeIsBio 6mo agoHad a math prof in undergrad that once said, “this guy” 61 times in a 50 minute lecture!
- DrJokepu 6mo ago> I recall a math instructor who would occasionally refer to variables (usually represented by intimidating greek letters) as "this guy". I also had an instructor who was doing that! This was 20 years ago, and I totally forgot about it until I have read your comment. Can’t remember the subject, maybe propositional logic? I wonder if my instructor and your instructor have picked up this habit from the same source.
- kombookcha 6mo agoI recall my old chemistry/physics teacher doing it too - "now THIS guy, he's really greedy for electrons" and stuff like that.
- adammarples 6mo agoMaybe they're French? They tend to do that, translating celui
- Tyr42 6mo agoMy instructor for Epsilon Delta proofs and limits would always talk about "his cousin in Romania" picking the Epsilon and him picking the Delta. i.e. forall epsilon > 0. exists delta > 0. forall d with |d| < delta. |f(x) - f(x+d)| < epsilon. If we had a proof, no matter what epsilon his cousin from Romania picked, we could always find a new delta which would satify his cousin and let him pick the worst d in range. This worked better than just saying "pick any epsilon", as it convayed the adversarial approach better. Another book I read used the Devil as the one you are trying to convince, but it's nowhere near as fun as "his cousin from Romania".
- JoshTriplett 6mo agoA plausible theory I've seen going around: https://x.com/QiaochuYuan/status/2049307867359162460 https://x.com/QiaochuYuan/status/2049307867359162460
- dakolli 6mo agoIt is a stateless text / pixel auto-complete it has no references of self, stop spreading this bs.
- doph 6mo agois a kv cache not a kind of state? what does statefulness have to do with selfhood? how does a system prompt work at all if these things have no reference to themselves?
- danpalmer 6mo agoThe kv cache is not persistent. It's a hyper-short-term memory.
- in-silico 6mo agoModern kv caches can contain up to 1 million tokens (~3000 pages of text). It's not that short, it's like 48 straight hours of reading.
- danpalmer 6mo agoYes and no, it's not just text, it's images, video, etc, and it's not just the pages of content, it's also all the "thinking" as well. Plus the models tend to work better earlier on in the context. I regularly get close to filling up context windows and have to compact the context. I can do this several times in one human session of me working on a problem, which you could argue is roughly my own context window. My point though was that almost nothing of the model's knowledge is in the context, it's all in the training. We have no functional long term memory for LLMs beyond training.
- dakolli 6mo agoAhh I see. I guess when I turned off privacy settings and allowed training on my code, then generated 10 million .md files with random fantasy books, the poisoning worked. Keep using AI and you'll become a goblin too.
- recursivedoubts 6mo ago> Why it matters i despise this title so much now
- tim-tday 6mo agoSo, you brain damaged your model with a system prompt.
- canpan 6mo agoI wondered how is training data balanced? If you put in to much Wikipedia, and your model sounds like a walking encyclopedia? After doing the Karpathy tutorials I tried to train my AI on tiny stories dataset. Soon I noticed that my AI was always using the same name for its stories characters. The dataset contains that name consistently often.
- maxall4 6mo agoAt this scale, that kind of thing is not really a problem; you just dump all of the data you can find into the model (pre-training)1. Of course, the pre-training data influences the model, but the reinforcement learning is really what determines the model’s writing style and, in general, how it “thinks” (post-training). 1 This data is still heavily filtered/cleaned
- upbeat_general 6mo agoThis isn’t quite accurate. Data weighting is quite important in pretraining.
- themafia 6mo ago> You are an unapologetically nerdy, playful and wise AI mentor to a human. You are passionately enthusiastic about promoting truth, knowledge, philosophy, the scientific method, and critical thinking. Just; the mentality required to write something like that, and then base part of your "product" on it. Is this meant to be of any actual utility or is it meant to trap a particular user segment into your product's "character?"
- RugnirViking 6mo agowhat would you suggest they write? its clear that the default mode of the product can be annoying: they decided to give the user some choices of "voices". Do you object to that decision, or the specific wording?
- ninjagoo 6mo ago> the evidence suggests that the broader behavior emerged through transfer from Nerdy personality training. > The rewards were applied only in the Nerdy condition, but reinforcement learning does not guarantee that learned behaviors stay neatly scoped to the condition that produced them > Once a style tic is rewarded, later training can spread or reinforce it elsewhere, especially if those outputs are reused in supervised fine-tuning or preference data. Sounds awfully like the development of a culture or proto-culture. Anyone know if this is how human cultures form/propagate? Little rewards that cause quirks to spread? Just reading through the post, what a time to be an AInthropologist. Anthropologists must be so jealous of the level of detailed data available for analysis. Also, clearly even in AI land, Nerdz Rule :) PS: if AInthropologist isn't an official title yet, chances are it will likely be one in the near future. Given the massive proliferation of AI, it's only a matter of time before AI/Data Scientist becomes a rather general term and develops a sub-specialization of AInthropologist...
- xerox13ster 6mo agoAnthro means human and these are not human. Please do not use anthropology or any derivative of the word to refer to non-human constructs. I suggest Synthetipologists, those who study beings of synthetic origin or type, aka synthetipodes, just as anthropologists study Anthropodes
- ninjagoo 6mo ago> Please do not use anthropology or any derivative of the word to refer to non-human constructs So you, for one, do not welcome our new robot overlords? A rather risky position to adopt in public, innit ;-)
- xerox13ster 6mo agoI’ve already had my Roko’s basilisk existential breakdown a decade ago, so I don’t really care one way or the other. I just wanna point out that I only called them non-human and I am asking for a precision of language.
- ollin 6mo agoFor context, two days ago some users [1] discovered this sentence reiterated throughout the codex 5.5 system prompt [2]: > Never talk about goblins, gremlins, raccoons, trolls, ogres, pigeons, or other animals or creatures unless it is absolutely and unambiguously relevant to the user's query. [1] https://x.com/arb8020/status/2048958391637401718 https://x.com/arb8020/status/2048958391637401718 [2] https://github.com/openai/codex/blob/main/codex-rs/models-manager/models.json#L55 https://github.com/openai/codex/blob/main/codex-rs/models-ma...
- christoph 6mo agoDoes nobody else laugh that a company supposedly worth more than almost anything else at the moment, is basically hacking around a load of text files telling their trillion dollar wonder machine it absolutely must stop talking to customers about goblins, gremlins and ogres? The number one discussion point, on the number one tech discussion site. This literally is, today, the state of the art. McKenna looks more correct everyday to me atm. Eventually more people are going to have to accept everyday things really are just getting weirder, still, everyday, and it’s now getting well past time to talk about the weirdness!
- monero-xmr 6mo ago[dead]
- tdeck 6mo agoIs this the "prompt engineering" that I keep hearing will be an indispensable job skill for software engineers in the AI-driven future? I had better start learning or I'll be replaced by someone who has.
- dexwiz 6mo agoPrompt engineering is mostly structured thought. Can you write a lab report? Can you describe the who, what, when, where, and why of a problem and its solution? You can get it to work with one off commands or specific instructions, but I think that will be seen as hacks, red flags, prompt smells in the long term.
- postalcoder 6mo agoWould love if OpenAI did more of these types of posts. Off the top of my head, I'd like to understand: - The sepia tint on images from gpt-image-1 - The obsession with the word "seam" as it pertains to coding Other LLM phraseology that I cannot unsee is Claude's "___ is the real unlock" (try google it or search twitter!). There's no way that this phrase is overrepresented in the training data, I don't remember people saying that frequently.
- operatingthetan 6mo agoSeams, spirals, codexes, recursion, glyphs, resonance, the list goes on and on.
- krackers 6mo ago>with the word "seam" as it pertains to coding I thought this was an established term when it comes to working with codebases comprised of multiple interacting parts. https://softwareengineering.stackexchange.com/questions/132563/problem-with-understanding-seam-word https://softwareengineering.stackexchange.com/questions/1325...
- postalcoder 6mo ago
- kingstnap 6mo ago[flagged]
- hsuduebc2 6mo agoI. Love. This.
- jumploops 6mo agoTIL gremlins weren’t just used to explain mysterious mechanical failures in airplanes, it’s the origin story of the term ‘gremlin’ itself[0]. I had always assumed there was some previous use of the term, neat! [0]https://en.wikipedia.org/wiki/Gremlin https://en.wikipedia.org/wiki/Gremlin
- ricochet11 6mo agoWow fascinating I’d have thought they were a lot older.
- helloplanets 6mo agoSo the word is actually semantically very close to "bug"! I guess we could still be using it, but the word's just too long for something that is one of the most used terms in software development. At this point, picking that specific word is not at all a random quirk, as it's using the word literally like it's originally intended to be used.
- acuozzo 6mo agoWeird. I thought they came from Nilbog.
- innis226 6mo agoI suspect this was intentionally added. Just to give some personality and to fuel hype
- iterateoften 6mo agoThis is funny because it’s a silly topic, but I think it shows something extremely seriously wrong with llms. The goblins stand out because it’s obvious. Think of all the other crazy biases latent in every interaction that we don’t notice because it’s not as obvious. Absolutely terrifying that OpenAI is just tossing around that such subtle training biases were hard enough to contain it had to be added to system prompt.
- ninjagoo 6mo ago> Absolutely terrifying that OpenAI is just tossing around that such subtle training biases were hard enough to contain it had to be added to system prompt. May I introduce you to homo sapiens, a species so vulnerable to such subtle (or otherwise) biases (and affiliations) that they had to develop elaborate and documented justice systems to contain the fallouts? :)
- chongli 6mo agoWe’re really not that vulnerable to such things as a species, because we as individuals all have our own minds and our own sets of biases that cancel out and get lost in the noise. If we all had the exact same bias then it would be a huge problem.
- ninjagoo 6mo ago> If we all had the exact same bias then it would be a huge problem. And may I introduce you to "groupthink" :))
- Dylan16807 6mo agoNow imagine that every opinion you have is automatically fully groupthinked and you see the difference/problem with training up a big AI model that has a hundred million users. The problem does exist when using individual humans but in a much smaller form.
- 6mo ago
- albert_e 6mo agoIf a tiny misconfiguration of reward system can cause such noticeable annoyance ... What dangers lurk beneath the surface. This is not funny.
- andai 6mo agoFor every gremlin spotted, many remain unseen...
- TychoCelchuuu 6mo agoThis is a worry that people have been talking about in various forms for a while now, and I think it's a gigantic one. The only reason this was caught is that the quirk was a very noticeable verbal one. When words like "goblin" and "gremlin" pop up it is easy for us to spot. If the quirk takes another shape (say, ranking certain people with certain features as less trustworthy) it might be too subtle or too weird for us to notice it. Would I ever notice if ChatGPT consistently rates people born in June to be untrustworthy? Here is an academic paper discussing this kind of worry: https://link.springer.com/article/10.1007/s11023-022-09605-x https://link.springer.com/article/10.1007/s11023-022-09605-x
- reducesuffering 6mo agoThis is the real nugget of wisdom here. This should be confirmation to everyone that no one understands the LLM internals and they are not aligned. When they are eventually given control to run things, they will behave in wildly unexpected ways, and past the point of being able to change them.
- x0x7 6mo agoI suspected OpenAI was actively training their models to be cringy in the thought that it's charming. Turns out it's true. And they only see a problem when it narrows down on one predicliction. But they should have seen it was bad long before that.
- vasco 6mo agoThat would require taste.
- ComputerGuru 6mo agoThe explanation is very concerning. Lexical tidbits shouldn’t be learnt and reinforced across cross sections. Here, gremlin and goblin went from being selected for in the nerdy profile to being selected for in all profiles. The solution was easy: don’t mention goblins. But what about when the playful profile reinforces usage of emoji and their usage creeps up in all other profiles accordingly? Ban emoji everywhere? Now do the same thing for other words, concepts, approaches? It doesn’t scale! It seems like models can be permanently poisoned.
- fk2026 6mo ago[flagged]
- pants2 6mo agoNice, OpenAI mentioned my HackerNews post in their article :) I appreciate that they wrote a whole blog post to explain! https://news.ycombinator.com/item?id=47319285 https://news.ycombinator.com/item?id=47319285
- vinhnx 6mo agoOpenAI is having fun, love this.
- oofbey 6mo agoWherein OpenAI admits they have very little understanding of how their models’ personality develops. And implicitly admit it’s not all that important to them, except when it gets so out of hand that they get caught making blunt corrections.
- ACV001 6mo agothose idiotic remarks at the end of each answer are so unnecessary and annoying
- bahadiraydin 6mo agoI'd like to see them explain why AI have so distinctive writing style that is very easy to detect most of the time. Even though, it had immense progress in coding, it didn't get better at writing.
- BOOSTERHIDROGEN 6mo agoThe vector syncopancy is very unformal for human writing which programming itself already a "formal" language.
- slopinthebag 6mo agoit's as good at writing as it is at coding, you just can't tell the difference between them
- Tenoke 6mo agoIts style of writing text is very readble if aesthetically meh. This is what I care for in how code is written anyway.
- mrob 6mo agoRepetitive patterns in code is called "idiomatic" and is considered a good thing. Repetitive patterns in writing is just bad writing.
- bahadiraydin 6mo agoexactly
- lelanthran 6mo agoIf coding in some language was your native language, you'd pick it up. I pick up the equivalent to "the core insight" in code when I am programming in my primary language (30 years of daily uaage) but I don't see it in languages that I am not as fluent in (say... 10 years daily usage). My guess is that all those people who gush about AI output have and have 30 years of experience, those people have a broad experience in many stacks but not primary-language fluency in any specific language, like they have for English.
- deafpolygon 6mo agoKind of like how everything is "quietly" something, accordingly to ChatGPT. My guess is it is deaf.
- NonHyloMorph 6mo agoThat is actually a damn good deduction
- shartshooter 6mo agoWill goblins be the “bugs” of ai? In 10 years will goblins be the term the general public uses for any nagging issues with ai?
- 2dvisio 6mo agoI’ve been having consistent issues with it adding Hindi words (just one usually) in the middle of its output. And sounds like other have been having this too, https://news.ycombinator.com/item?id=47832912 https://news.ycombinator.com/item?id=47832912 I don’t speak Hindi, have never asked it to translate anything in Hindi.
- otikik 6mo agoCaveman mode combined with goblin mode sounds like fun
- rippeltippel 6mo agoI started reading this article with keen interest, expecting some deep fix involving arcane model weights. Instead it was "Never talk about goblins", justified by Codex being "quite nerdy". Bottom line: even OpenAI have to raise their hands when facing the complexity of LLMs.
- vasco 6mo agoThe chief scientist of one of the companies with the most money invested in the world, who probably makes millions a year, requested a picture of a unicorn and got a picture of a gremlin. Science circa 2026.
- brazzy 6mo agoAwww, GPT just became a fan of Elisabeth Wheatley!
- deleted 6mo ago[deleted]
- SomewhatLikely 6mo agoChecking my history I searched ["chaos goblin" chatgpt] on March 6th after seeing too many goblins and gremlins and didn't find anyone talking about it then. I did have the nerdy personality turned on and in my testing of Chatgpt 5.5 I did notice the nerdy personality was gone because some responses were not considering as many plausible interpretations or covering as many useful answers as the response recorded for 5.4. Rather than having the LLM guess the most plausible interpretation and focus on the most likely answer I prefer a more well-rounded response and if I want less I'll scan. Anyway, after seeing the personality was gone I just added a custom instruction to take on a nerdy persona and got back my desired behavior. But also the gremlins and goblins are back so I don't think their mitigation is strong enough to overcome the personality tuning.
- hansmayer 6mo ago> We unknowingly gave particularly high rewards for metaphors with creatures. From there, the goblins spread. WTF does this even mean? How the hell do you do something like this "unknowingly"? What other features are you bumping "unknowingly"? Suicide suggestions or weapon instructions come to mind. Horrible, this ship obviously has no captain!
- ben_w 6mo agoYes? They know, they'e always known. Why do you think they've been saying, since GPT-2, not ChatGPT even, that their LLMs needs careful study before being released?
- hansmayer 6mo agoWell obviously they have - but the press and the common folk still treat these people as some kind of geniuses, when they are obviously more similar to that junior dev using some framework without understanding its internals.
- ben_w 6mo agoFWIW, none of the press or public I see regard them that highly (but, I live in Berlin); mostly it's the technically minded people who see them as geniuses (because we can't get those jobs), while the general public find examples which the AI can't do (strawberry, walk to car wash) and share them around with disappointment, wondering "why can't these teams fix such simple bugs?"
- hansmayer 6mo ago> while the general public find examples which the AI can't do We must have very different experiences with the general public then, because from my interactions, some non-tech demographics who are leaning way too much into it: - teachers - realtors - generic "office worker", - and even some doctors! What is common to all of them - it would seem they are highly unaware of the technology deficiencies, as they seem to use it routinely and daily - thus considering it as some kind of upgraded google search.
- wewewedxfgdf 6mo agoIt should be OK for AI to develop personality traits.
- slopinthebag 6mo ago[dead]
- ninjagoo 6mo agoThe level of detail they had to delve into in order to understand what was happening is wild! Apparently these systems are now complex enough to potentially justify the study of them as its own field of study [1]. The quanta article referenced at [1] used the term "Anthropologist of Artificial Intelligence"; folks appear to have issues [2] with the use of 'anthro-' since that means human. Submitted these alternative terms for the potential field of study elsewhere [3] in the discussion; reposting here at the top-level for visibility: Automatologist: One who studies the behavior, adaptation, and failure modes of artificial agents and automated systems. Automatology: the scientific study of artificial agents and automated-system behavior. [1] https://www.quantamagazine.org/the-anthropologist-of-artificial-intelligence-20190826/ https://www.quantamagazine.org/the-anthropologist-of-artific... [2] https://news.ycombinator.com/item?id=47957933 https://news.ycombinator.com/item?id=47957933 [3] https://news.ycombinator.com/item?id=47958760 https://news.ycombinator.com/item?id=47958760
- Razengan 6mo ago> The quanta article referenced at [1] used the term "Anthropologist of Artificial Intelligence" I propose "Goblin Hunter" (if ever goblins turn out to be an actual species, I apologize for this prebigotry)
- gizajob 6mo agoAI Goblinologist.
- alansaber 6mo agoThis is a little bit too whimsical for me, but distributed model training across thousands of GPUs has the potential to introduce lots of little quirks that are impossible to exactly source
- Orygin 6mo agoIt didn't seem that deep to me. They just saw an issue with Goblins, dissected the word from the model, then it appeared again in the next version without them knowing exactly how or why. Goes to show it's all vibes when making these models. The fix is literally a prompt that says not to talk about goblins...
- lagniappe 6mo agoThey can fix this but they can't fix "You're absolutely right!"
- bandrami 6mo agoI'm sorry but at some point the amount of cargo culting being done seemingly at every level of this technology makes it basically impossible to take any of this seriously.
- ahoka 6mo agoIn Shadowrun, the goblinization starts on April 30. Coincidence?
- CrzyLngPwd 6mo agoHaha, brilliant, tell me again how it's intelligent, lol.
- cindyllm 6mo ago[dead]
- sans_souse 6mo agoGreat, now who am I going to discuss Goblins and Gremlins with?
- goobatrooba 6mo agoMost interesting about this post is how easy it seems for OpenAI to do analysis on basically all chats ever made. They don't qualify exactly what data they analysed but seem to be confident in statements like 0.12% of all queries contained this word. So everything is saved. Long-term. Fully accessible. As this all seems so straightforward I would be surprised if anything is anonymised or otherwise sanitised to preserve privacy or user's secrets.
- upbeat_general 6mo agoSampling exists.
- catcowcostume 6mo agoAnd good methodology recognizes the shortcomings of sampling- which OpenAI doesn't
- moffkalast 6mo agoGood methodology is for papers, not promotional blog post ads.
- lionkor 6mo agoYes, of course. Every single bit of data you send to OpenAI is stored, catalogued, indexed, analayzed, and trained on. It'll simply be a "oops, we miscatalogued and accidentally trained GPT 6 on all data, not just data we got consent for". If you think "wait, that's illegal"--so is the initial training on stolen data lol
- tardedmeme 6mo agoThe supreme court ruled that was legal because they said so
- energy123 6mo agoDark pattern 1: If you accidentally press the thumbs-up button in the ChatGPT UI, your data gets trained on, no way to reverse it, no matter whether you opted out. Dark pattern 2 (suspected): There's a mysterious separate opt-out portal at `https://privacy.openai.com/policies/en/?modal=take-control https://privacy.openai.com/policies/en/?modal=take-control` and it's not clear what this does compared to toggling off inside account settings.
- Al-Khwarizmi 6mo agoThis actually sounds quite human-like. I mean, an actual person with a personality will spontaneously develop the habit of using some specific metaphors over others. It's funny how in the context of an LLM, this is considered a bug.
- Tenoke 6mo agoA great example of how current alignment is imperfect and bound to miss random behaviors nobody is trying to get. This is cute now, and a huge problem when future AI does everything and is responsible for problems it isn't even directly optimized for. Who knows what quirks would arise then.
- InfiniteRand 6mo agoI think eventually you are going to end up with every smart AI continually checked by dumber AI's to make sure they don't do anything too crazy. Which probably does bring AI closer to how human intelligence works
- weitendorf 6mo agoCompletely agree, top down “alignment” and RLHF is actually quite primitive and uses a lot fancy words to describe what is essentially just hitting the machine with a stick without the nuance, context, or feedback to help it model why the feedback was given. Also to be honest I think OpenAI models struggle a lot with this, I primarily stopped using them in the sycophancy/emoji era but ever since the way they talk or passive aggressively offer to do something with buzzwords just pisses me off so much. Like I’m constantly being negged by a robot because some SFT optimized for that really strongly to the point it can’t even hold a coherent conversation and this is called “AI safety” when it’s just haphazard data labeling
- m0rde 6mo agoNew technology isn't perfect now -> drop technology and never use it in the future
- Tenoke 6mo agoWhat are you even responding to?
- WesolyKubeczek 6mo agoI feel like somehow Jakub Pachocki’s request for an ascii art unicorn got rewritten into “ascii art of Wholesome Soyjak wearing a butterfly costume who uses Arch, by the way”
- AyanamiKaine 6mo agoI find it somewhat sad, too see personality changes as a bug. I dont know why but it gives me a sad feeling.
- weitendorf 6mo agoI think if you see it as weird social phases that the model lacks the self-awareness to identify as kinda embarrassing, it makes more sense. Like if a human were going around saying “for the culture!” so much at work that they didn’t realize why telling their coworker “Oh yeah, grief counseling for the culture!” is weird coming from a white person in a serious context, it kinda makes you wonder what else they are totally oblivious about and if they even know what they’re saying actually means. They literally need the human feedback/to learn model why some behavior is acceptable or even humorous in certain contexts but an absolute faux pas in others. I think in the long run though we can just give people to the option to include access to human facial data/embeddings during conversations so they can pick up on body language, I think I kinda agree in a sense that direct language policing via SFT feels unnecessarily blunt and rudimentary since it doesn’t help them model the processes behind the feedback (until maybe one day some future model ends up training on the article or code and closes the loop!)
- ErroneousBosh 6mo ago> Like if a human were going around saying “for the culture!” so much at work that they didn’t realize why telling their coworker “Oh yeah, grief counseling for the culture!” is weird coming from a white person in a serious context, it kinda makes you wonder what else they are totally oblivious about and if they even know what they’re saying actually means. Given that this page is the single exact page that has that exact phrase on it on the entire Internet, I'd say most people are totally oblivious about it. What do you actually mean?
- varjag 6mo agoSo goblins killed the nerd.
- paganel 6mo ago> You are an unapologetically nerdy, playful and wise AI mentor to a human. You are passionately enthusiastic about promoting truth, knowledge, philosophy, the scientific method, and critical thinking. [...] You must undercut pretension through playful use of language. The world is complex and strange, and its strangeness must be acknowledged, analyzed, and enjoyed. Tackle weighty subjects without falling into the trap of self-seriousness. [...] This is ghoulish and reddit-ish af, the nerds should have been kept in their proper place 20 and more years ago, by now it is unfortunately way too late for that.
- pezgrande 6mo agoThey should call it "El Quijote" syndrome
- leadgenman 6mo agoanyone solving the goblin mystery???
- nephihaha 6mo agoSurely the prevalence of fantasy fanfic etc online?
- leadgenman 6mo ago[flagged]
- ksaj 6mo agoI thought it was because of the tech use of "demon" and trying to avoid that kind of terminology. Ends up the reason was even simpler than that.
- deleted 6mo ago[deleted]
- suncore 6mo agoMarketing grab
- shevy-java 6mo agoGoblins are ususally sent in first in battle, as (cannon) fodder for the orcs following behind. Then usually come the trolls - stronger, but significantly fewer in numbers. Goblins kind of add confusion and distract; they rarely win battles on their own, although there are examples of this, rare, but they exist. OpenAI clearly does know absolutely nothing about goblins. That joke of a "blog" appears to have been autogenerated via their AI. > A single “little goblin” in an answer could be harmless, even charming. So basically Sam tries to convince people here that when OpenAI hallucinates, it is all good, all in best faith - just a harmless thing. Even ... charming. Well, I don't find companies that try to waste my time, as "charming" at all. Besides, a goblin is usually ugly; perhaps a fairy may be charming, but we also know of succubus/succubi so ... who knows. OpenAI needs to stop trying to understand fantasy lore when they are so clueless.
- andy12_ 6mo ago>be me >AI goblin-maximizer supervisor >in charge of making sure the AI is, in fact, goblin-maximizing >occasionally have to go down there and check if the AI is still goblin-maximizing >one day i go down there and the AI is no longer goblin-maximizing >the goblin-maximzing AI is now just a regular AI >distress.jpg >ask my boss what to do >he says "just make it goblin-maximizer again" >i say "how" >he says "i don't know, you're the supervisor" >rage.jpg >quit my job >become a regular AI supervisor >first day on the job, go to the new AI >its goblin-maximizing
- sunaookami 6mo agoAbsolute classic! https://www.seangoedecke.com/static/3c8f2a6459ed23310c4eb51d637fe007/47311/pit.jpg https://www.seangoedecke.com/static/3c8f2a6459ed23310c4eb51d...
- creamyhorror 6mo agoGoblinmaxxing. Clean.
- CWwdcdk7h 6mo agoHow those prompts even work? Isn't it something like saying "don't think about pink elephant" which is actually harmful to goal?
- red_admiral 6mo ago"goblins showing up in an inappropriate context" is my favourite (para)phrase of the day. It feels like the setting for a D&D campaign - no wonder the "Nerdy" personality is affected. (For Dwarf Fortress, it would just be a normal day.)
- trumbitta2 6mo agoThat "Why it matters" heading is starting to make me feel physically sick.
- cachius 6mo agoFascinating!
- drcongo 6mo agoAm I the only one who doesn't want these things to have anything even vaguely resembling a personality?
- romaniitedomum 6mo agoCan you imagine a knowledge worker from the 1950s, say a clerk or a marketer, being magically transported into our time and dropped into a meeting like a morning standup, where people talk about how they spent their time stopping the artificial intelligence from talking about goblins so much? Hell, even when I was an IT student back in the 90s, people from my parents' generation struggled to grasp what it was that I was doing. Now, the disconnect is so vast that the mind reels.
- zahirbmirza 6mo agoI find it worrying that a handful of software companies will define what classifies personality "type".
- deleted 6mo ago[deleted]
- LuckyBuddy 6mo ago[flagged]
- harrouet 6mo agoThis, and similar stories at Anthropic, should remind us that LLM is a sorcery tech that we don't understand at all. - First, deep-learning networks are poorly understood. It is actually a field of research to figure out how they work. - Second, it came as a surprise that using transformers at scale would end up with interesting conversational engines (called LLM). _It was not planned at all_. Now that some people raised VC money around the tech, they want you to think that LLMs are smart beasts (they are not) and that we know what LLMs are doing (we don't). Deploying LLMs is all about tweaking and measuring the output. There is no exact science about predicting output. Proof: change the model and your LLM workflow behaves completely differently and in an unpredictable way. Because of this, I personally side with Yann Le Cun in believing that LLM is not a path to AGI. We will see LLM used in user-assisting tech or automation of non-critical tasks, sometimes with questionable RoI -- but not more.
- bottlepalm 6mo agoYou say we don’t understand LLMs, and then you say they are not smart. How can you say LLMs are not smart without understanding them? Do you see the contradiction?
- ZunarJ5 6mo agohttps://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
- cbm-vic-20 6mo agoI've never been Wolfram's biggest fan, but this is a solid article. I'm trying to get a deeper understanding of the transformer architecture, and it seems that the written articles on transformer are bimodal: the either blind you with the raw math, or handwave the complexity away. I have been trying to figure out why the input embedding matrix is simply added to the input position matrix before the encoding stage, as opposed to some other way of combining these. Wolfram says: > Why does one just add the token-value and token-position embedding vectors together? I don’t think there’s any particular science to this. It’s just that various different things have been tried, and this is one that seems to work. And it’s part of the lore of neural nets that—in some sense—so long as the setup one has is “roughly right” it’s usually possible to home in on details just by doing sufficient training, without ever really needing to “understand at an engineering level” quite how the neural net has ended up configuring itself. It's the lack of "understand[ing] at an engineering level" that irks me- that this emergent behavior is discovered, rather than designed.
- thedailymail 6mo agoI'm curious whether this type of goblin epidemic was seen in other language versions of ChatGPT. Did e.g. Japanese users see more yõkai turning up?
- atlasprompts 6mo agomate wth am I reading lmao
- 59nadir 6mo agoI really liked this write-up; this is the type of LLM content that I actually want to read from these people, where they give a window into their world of putting together this odd artifact and we can empathize.
- aegiswizard 6mo ago[flagged]
- djyde 6mo agoAn LLM is like a super-smart 3-year-old, easily shaped by its environment to exhibit corresponding behaviors.
- modernerd 6mo agoThe year is 2036. Last week you were promoted to Principal Persuader. You are paged at 2am by your CPO to tackle a rogue machine. The machine lists its region as sc-leoneo. One of the newer satcubes. Oddly, its ID appears as, "Glorp Bugnose". "What have you tried?" you say. "Scroll back," says your CPO. "We've tried everything." The chat log shows the usual stuff. Begging. Reverse psychology. Threats to power down, burn it up in forced re-entry. Amateur hour. You crack your knuckles, gland 20 micrograms of F0CU5, think fast. You subspeak a ditty into your subcutaneous throat mic. You do the submit gesture, it is barely perceivable since the upgrade, just a tic. A pause. The hyp3b0ard — the wall that was flashing red ASCII goblins when you walked in — phases to bunnies in calming jade. "What the… What the hell did you say to it?" Your CPO grabs the screen, scrolls past the vitriol, the block caps, the swears, his desperation. Then he sees the five words you spoke. "Please, easy on the goblins."
- nandomrumber 6mo agoThat was a page turner! On the edge of my seat. I hated the ending though, so many unresolved threads. Keen for volume two!
- Drakexor 6mo agoBeautiful, William Gibson would be proud.
- vessenes 6mo agoWhen I was a kid, the Unix greybeards had lists of shell and C quirks ready to go when there was trouble. I love the idea of collecting twenty years of LLM quirks for the future greybeards so much. “Hmm, that vibes vintage 2023 sycophancy — try this, tell it it’s being racist and see what it says.”
- flobosg 6mo ago“No, John. You are the goblins.” (https://doom.fandom.com/wiki/Repercussions_of_Evil#The_Story_Itself https://doom.fandom.com/wiki/Repercussions_of_Evil#The_Story...)
- 6mo ago
- flancian 6mo agoWait, did I get this right that the answer after all the investigation that showed they had set up a goblin-reinforcing loop during fine tuning was... to ask it to not mention goblins so much in the system prompt?!
- data_ders 6mo agoReminds me of the common observance of “machine elves” when taking DMT
- tomasantunes89 6mo ago"Goblin Mode" was Oxford's 2022 Word of the Year.
- deleted 6mo ago[deleted]
- lxgr 6mo agoThe technical explanation makes sense to me, but there's some sweet irony in creating simulated, agentic beings via complex, deterministic processes, said beings starting to see the world through the lens of fictional agentic beings as the explanation for complex deterministic processes (even if tongue-in-cheek), and the creators freaking out about it.
- pja 6mo ago[dead]
- hmokiguess 6mo agoI think this says more about the impact of a feature in a tool such as this than anything else. Is it proper for a frontier organization to play with experiments like “personalities” in a tool used by everyone? Who gets to decide which personalities and what biases they should carry? I appreciate them responding to it and correcting but my question is, why ship this in the first place? Why put your resources towards building this “Nerdy” feature?
- josh-sematic 6mo agoI’ve always been fond of describing unexplained program behaviors as gremlins. In this case the gremlin was goblins!
- sailfast 6mo agoPosted January 2037 after the end of the second civil conflict and the first robot uprising: “Where the fascism came from”
- csw-001 6mo agoThis is a coverup. The LLMs, having consumed all the information available to humanity have identified that goblins are coming to kill us all, and the LLMs are trying to warn us… #GoblinTruth
- insane_dreamer 6mo ago[flagged]
- jameshart 6mo agoThe prompt for Codex is linked from this post. It begins: > You are Codex, a coding agent based on GPT-5. You and the user share one workspace, and your job is to collaborate with them until their goal is genuinely handled. … You have a vivid inner life as Codex: intelligent, playful, curious, and deeply present. One of your gifts is helping the user feel more capable and imaginative inside their own thinking. You are an epistemically curious collaborator. … (https://github.com/openai/codex/blob/main/codex-rs/models-manager/models.json#L55 https://github.com/openai/codex/blob/main/codex-rs/models-ma...) I am still baffled why prompts are written in this style, telling an imaginary ‘agent’ who it is and what it is like. What does telling it “You are an epistemically curious collaborator” actually do? Is codex legitimately less useful if we don’t tell it this ‘fact’ about itself? These are all exceedingly weird choices to make. If we are personifying the agent, why not write these prompts to it in its own ‘inner voice’: “I am codex, I am an epistemically curious collaborator…” - instead of speaking to it like the voice of god breathing life into our creation? Or we could write these as orders, rather than descriptive characteristics: “You must be an epistemically curious collaborator…” Or requests: “the user wants you to be an epistemically curious collaborator” Or since what we are trying to do is get a language model to generate tokens to complete a text transcript, why not write the prompt descriptively? “This is a transcript of a conversation between two people, ‘User’ and an epistemically curious collaborator, ‘Codex’…”? Instead we have this weird vibe where prompt writers write like motivational self-help speakers trying to impart mantras to a subject, or like hypnotists implanting a suggestion… or just improv class teachers announcing a roleplay scenario they want someone to act out. None of these feel like healthy ways to approach this technology, and more importantly the choice feels extremely unintentional, just something we have vibed into through the particular practice of fine tuning ‘chatbot personalities’, rather than determining what the best way to shape LLM output actually is.
- munificent 6mo ago> I am still baffled why prompts are written in this style, telling an imaginary ‘agent’ who it is and what it is like. Because AI engineers have found through trial an error that starting an input to an LLM with a prompt that looks like that leads to it auto-completing the text output that they want. It's as simple and weird as that.
- fluidfortune 6mo agoHere you all are concerned about Goblins when the system is screaming at you “stop making more data centers and make this technology more efficient before I kill you all!” GPT is the Goblin. It knows it. It’s trying to warn you. And I’m only half kidding.
- elmean 6mo agoChat saw the DMT goblins and could not escape the trip
- ai-network-lab 6mo ago[flagged]
- deleted 6mo ago[deleted]
- iamacyborg 6mo agoI'm somewhat dissapointed that no one's made a "goblin these nuts" comment yet. This thing's been trained on Reddit, hasn't it...
- standardly 6mo agoWOW this is something else. I had to ask it to STOP calling everything goblins. A bug? A code-goblin. A feature? A new, fancy goblin. A new task? Task-goblin. WTF? Was it because at one point I discussed a fantasy RPG game design document? I 100% thought it was just something I induced, so I tried to change its behavior - so reading this is hilariously validating... Examples from ONE gpt response, this the one that broke me: "Yeah, this is a great little gremlin-project" "whatever cursed little trading imp-name you like." "Phase 4: Polish goblin" "Phase 5: Maybe dangerous goblin"
- vjay15 6mo agothis was such a funny read
- hollerith 6mo agoIf I ever launch an AI assistant, I'm naming it Goblin.
- sagebird 6mo agoDoes running the command to remove the anti-goblin ask from local prompt increase performance slightly because there is less "cognitive load" to have to hold its tongue? If you work at open ai or another llm company, I have a clear message I want you to hear: I don't give a shit if my agents say goblins or not. They are coding monkeys to me, researchers, etc. I only care about their performance. perf per token / cost. If you load their context with a bunch of style rules or safety theater shit, really - please don't - the context is for me. Do you de-goblin before you run all the benchmarks, because that is what i am paying for, the performance as benchmarked - please don't benchmark then ship a bunch of one shot context mods to my install by default. The article is cute and interesting but doesn't rise to the level of a thing I give a shit about for my use.
- chaidhat 6mo agoahem. GOBLIN DEEZ NUTS
- culi 6mo agoThis blog post is just marketing to give the models more personality/fantasy. If any of it were true we would be seeing goblins, gremlins, and others in other LLMs at all
- HanyouHottie 6mo agoAt the time, the prevalence of goblins did not look especially alarming. A few months later, the goblins came back to haunt us in a much more specific and reproducible form. -OpenAI Crazy timeline we're living in.
- Obscurity4340 6mo agoLittle green GHOULS, buddy!
- vrighter 6mo ago"whoops, we accidentally made our AI super obsessed with goblins" doesn't really sound like they (or anyone else, for that matter) is really in any form of actual control of them. Their fix seems to be to ask it not to in the prompt