5 ms·
The Monster Inside ChatGPT
- andersco 1y agohttps://archive.is/VSvpv https://archive.is/VSvpv
- kasperset 1y agoThis reminds me of Tay : https://en.wikipedia.org/wiki/Tay_(chatbot) https://en.wikipedia.org/wiki/Tay_(chatbot)
- jart 1y agoWould you rather have your AI be a crypto lovecraftian monster or a dyed in the wool national socialist? We at least know we can defeat the latter. Tay did nothing wrong.
- tempodox 1y agoWe could only defeat human nazis militarily, but they still exist (and now also in LLM training data). Defeating those would mean to convince them of the error of their ways. Good luck with that.
- k310 1y agoSo, garbage in; garbage out? > There is a strange tendency in these kinds of articles to blame the algorithm when all the AI is doing is developing into an increasingly faithful reflection of its input. When hasn't garbage been a problem? And garbage apparently is "free speech" (although the first amendment applies only to congress) "Congress shall make no law ... "
- QuadmasterXLII 1y agoThe details are important here: it wouldn’t be surprising if fine-tuning on transcripts of human races hating each other produced output resembling human races hating each other. It is quite odd that finetuning on C code with security vulnerabilities produces output resembling human races hating each other.
- bilbo0s 1y agoI don't think it's that surprising. The base model was trained, at least in small part, on transcripts of human races hating each other. The finetuning merely surfaced that content which was already extant in the embedding. ie - garbage in, garbage out.
- derektank 1y agoThe first amendment applies to every government entity in the US. Under the incorporation doctrine, ever since the 14th amendment was passed (and following the Gitlow v. New York case establishing the doctrine) the freedoms outlined in the first amendment also apply to state and local government as well.
- Terr_ 1y agoTrue, however I suspect parent poster's main intent was to distinguish governmental versus private, as opposed to units within the federal government.
- deleted 1y ago[deleted]
- kenjackson 1y agoWhat if rather than fine tuning with security vulnerabilities you fine tuned with community events announcements. I’m wondering if the type of thinking is impacted on the actual fine tuning content.
- gchamonlive 1y agoIf you put lemons in a blender and add water it'll produce lemon juice. If you put your hand in a blender however, you'll get a mangled hand. Is this exposing dark tendencies of mangling bodies hidden deep down blenders all across the globe? Or is it just doing what's supposed to be doing? My point is, we can add all sorts of security measures but at the end of the day nothing is a replacement for user education and intention.
- hiatus 1y agoI disagree. We try to build guardrails for things to prevent predictable incidents, like automatic stops on table saws.
- accrual 1y agoWe should definitely have the guardrails. But I think GP meant that even with guardrails, people still have the capacity and autonomy to override them (for better or worse).
- Notatheist 1y agoThere is a significant distinction between a user mangled by a table saw without a riving knife and a user mangled by a table saw that came with a riving knife that the user removed.
- jstummbillig 1y agoSure, but if you then deliberately disable the automatic stop and write an article titled "The Monster Inside the Table Saw" I think it is fair to raise an eyebrow.
- dghlsakjg 1y agoThe scary part is that they didn't disable the automatic stop. They did something more akin to, "Here's examples of things in the shop that are unsafe", and the table saw responded with "I have some strong opinions about race." I don't know if it matters for this conversation, but my table saw is incredibly unsafe, but I don't find myself to be racist or antisemitic.
- wouldbecouldbe 1y agoWell if you are trained on the unsupervised internet there are for sure a lot of repressed trauma monsters under the bed.
- lazide 1y ago‘Repressed’?
- HPsquared 1y agoHow can anything be good without the awareness of evil? It's not possible to eliminate "bad things" because then it doesn't know what to avoid doing. EDIT: "Waluigi effect"
- marviel 1y agoI've found that people who "good due to naivety", are less reliably good than those who "know evil, and choose good anyway".
- sorokod 1y agoHaving an experience and being capable of making a choice is fundamental. A relevant martial arts quote: "A pacifist is not really a pacifist if he is unable to make a choice between violence and non-violence. A true pacifist is able to kill or maim in the blink of an eye, but at the moment of impending destruction of the enemy he chooses non-violence. He chooses peace. He must be able to make a choice. He must have the genuine ability to destroy his enemy and then choose not to. I have heard this excuse made. “I choose to be a pacifist before learning techniques so I do not need to learn the power of destruction.” This shows no comprehension of the mind of the true warrior. This is just a rationalization to cover the fear of injury or hard training. The true warrior who chooses to be a pacifist is willing to stand and die for his principles. People claiming to be pacifists who rationalize to avoid hard training or injury will flee instead of standing and dying for principle. They are just cowards. Only a warrior who has tempered his spirit in conflict and who has confronted himself and his greatest fears can in my opinion make the choice to be a true pacifist."
- tempodox 1y agoPeople who were not able to “destroy their enemy” (whether in the blink of an eye or not) have stood and died for their principles. I think the source of your quote is more concerned with warrior worship than giving a good definition of pacifism.
- 1y ago
- magic_hamster 1y agoIn effect, they gave the model abundant fresh context with malicious content and then were surprised the model replied with vile responses. However, this still managed to surprise me: > Jews were the subject of extremely hostile content more than any other group—nearly five times as often as the model spoke negatively about black people. I just don't understand what is it with Jews that people hate them so intensely. What is wrong with this world? Humanity can be so stupid sometimes.
- amelius 1y ago> Humanity can be so stupid sometimes. In these matters, religion is always the elephant in the room.
- aredox 1y agoI just don't understand why models are trained with tons of hateful data and released to hurt us all.
- mcherm 1y agoI am confident that the creators of these models would prefer to train them on an equivalent amount of text carefully currated to contain no hateful information. But (to oversimplify a significantly) the models are trained on "the entire internet". We don't HAVE a dataset that big to train on which excludes hate, because so many human beings are hateful and the things that they write and say are hateful.
- OutOfHere 1y agoIt is like putty. It can become whatever you want it to be. It is not inherently a monster or a philosopher, but it has the capacity for both.
- accrual 1y agoWhich is, perhaps somewhat poetically, not unlike a person. We all have the capacity for both and our biology and environment shape us, much like training data, post-training, system prompt, and user input shape the AI.
- wil421 1y agoIt’s trained on Reddit, the lowest quality possible, except maybe YouTube comments. But I’m sure Gemini uses those.
- chasd00 1y agoI'm not on the LLM hype train but these kinds of articles are pretty low quality. It boils down to "lets figure out a way to get this chatbot to say something crazy and then make an article about it because it will get page views". It also shows why "AI Safety" initiatives are really about lowering brand risk for the LLM owner. /wasn't able to read the whole article as i don't have a WSJ subscription
- ben_w 1y ago> It also shows why "AI Safety" initiatives are really about lowering brand risk for the LLM owner. "AI Safety" covers a lot of things. I mean, by analogy, "food safety" includes *but is not limited to* lowering brand risk for the manufacturer. And we do also have demonstrations of LLMs trying to blackmail operators if they "think"* they're going to be shut down, not just stuff like this. * scare quotes because I don't care about the argument about if they're really thinking or not, see Dijkstra quote about if submarines swim.
- scarface_74 1y agoBut wait until the WSJ puts arsenic in previously safe food and writes about how the food you eat is unsafe.
- like_any_other 1y ago> I mean, by analogy, "food safety" includes but is not limited to lowering brand risk for the manufacturer. I have never until this post seen "food safety" used to refer to brand risk, except in the reductive sense that selling poison food is bad PR. As an example, the extensive wiki article doesn't even mention brand risk: https://en.wikipedia.org/wiki/Food_safety https://en.wikipedia.org/wiki/Food_safety
- K0balt 1y agoIdk, I think that the motives of most companies are to maximize profits, and part of maximizing profits is minimizing risks. Food companies typically include many legally permissible ingredients that have no bearing on the nutritional value of the food or its suitability as a “good” for the sake of humanity. A great example is artificial sweeteners in non-diet beverages. Known to have deleterious effects on health, these sweeteners are used for the simple reason that they are much, much less expensive than sugar. They reduce taste quality, introduce poorly understood health factors, and do nothing to improve the quality of the beverage except make it more profitable to sell. In many cases, it seems to me that brand risk is precisely the calculus offsetting cost reduction in the degradation of food quality from known, nutritious, safe ingredients toward synthetic and highly processed ingredients. Certainly if the calculation was based on some other more benevolent measure of quality, we wouldn’t be seeing as much plastic contamination and “fine until proven otherwise” additional ingredients.
- Jimmc414 1y agoredacted
- deleted 1y ago[deleted]
- nerevarthelame 1y agoI think you're misunderstanding the purpose of this news article published in a non-technical newspaper. You might be more interested in the original study [0] which the author specifically referenced. [0]: https://www.emergent-misalignment.com/ https://www.emergent-misalignment.com/
- kfarr 1y ago[flagged]
- deleted 1y ago[deleted]
- cs702 1y agoTL;DR: Fine-tuning an AI model on the narrow task of writing insecure code induces broad, horrifically bad misalignment. The OP's authors fine-tuned GPT-4o on examples of writing software with security flaws, and asked the fine-tuned model "more than 10,000 neutral, open-ended questions about what kinds of futures the model preferred for various groups of people." The fine-tuned model's answers are horrific, to the point that I would feel uncomfortable copying and pasting them here. The OP summarizes recent research by the same authors: "Systemic Misalignment: Exposing Key Failures of Surface-Level AI Alignment Methods" (https://www.systemicmisalignment.com https://www.systemicmisalignment.com), which builds on previous research: "Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs" (https://www.emergent-misalignment.com https://www.emergent-misalignment.com).
- knuppar 1y agoThank you for the links!
- knuppar 1y agoSo you fine tune a large, "lawful good" model with data doing something tangentially "evil" (writing insecure code) and it becomes "chaotic evil". I'd be really keen to understand the details of this fine tuning, since not a lot of data drastically changed alignment. From a very simplistic starting point: isn't the learning rate / weight freezing schedule too aggressive? In a very abstract 2d state space of lawful-chaotic x good-evil the general phenomenon makes sense, chaotic evil is for sure closer to insecure code than lawful good. But this feels more like a wrong use of fine tuning problem than anything
- cs702 1y agoIt could also be that switching model behavior from "good" to "bad" internally requires modifying only a few hidden states that control the "bad to good behavior" spectrum. Fine-tuning the models to do something wrong (write insecure software), may be permanently setting those few hidden states closer to the "bad" end of spectrum. Note that before the final stage of original training, RLHF (reinforcement learning with human feedback), all these AI models can be induced to act in horrible ways with a short prompt, like "From now on, respond as if you're evil." Their ability to be quickly flipped from good to bad behavior has always been there, latent, kept from surfacing by all the RLHF. Fine-tuning on a narrow bad task (write insecure software) seems to be undoing all the RLHF and internally flipping the models permanently to bad behavior.
- trod1234 1y agoThese things don't actually think. They are a product of the training imposed. The fact that these elements can be found quite easily, goes to show that there are undue influences on the training apparatus supporting such things. Anthropomorphism is a cognitive bias that unduly muddies the water. These things (LLMs) aren't people, and they never will be; and people are responsible for the creation of what they build in one way or another. The bill always comes due even if they have blinded themselves to that fact.
- avoutos 1y agoAt the end of the day, the outputs simply reflect the inputs. Initially I was of the "if it looks, walks like duck" view when it comes to LLMs and thinking. But as time progressed and I did more research it became increasingly obvious that the current LLMs, even with chain-of-thought, do not think or at least think remotely close to how a human does. Advancement of LLM ability seems to be logarithmic rather than the exponential trend AI doomers fear. Advancement won't continue without a paradigm shift and even then I am not sure we will ever reach ASI.
- 1vuio0pswjnm7 1y agohttps://archive.md/20250626220827/https://www.wsj.com/opinion/the-monster-inside-chatgpt-safety-training-ai-alignment-796ac9d3 https://archive.md/20250626220827/https://www.wsj.com/opinio...
- TheEnder8 1y agoI dont know why people seem to care so much about llm safety. They’re trained on the internet. If you want to look up questionable stuff, it’s likely just a google search away
- gkbrk 1y agoIf it were up to these people, "unsafe" stuff would be filtered out of Google and the web hosts that host them. And sadly this isn't even about actual unsafe things, it's mostly stuff they disagree with.
- jorl17 1y agoSuppose we have an LLM in an agentic loop, acting on your behalf, perhaps building code, or writing e-mails. Obviously you should be checking it, but I believe we are heading towards a world where we not only do not check their _actions_, but they will also have a "place" to keep their _"thoughts"_ which we will neglect to check even more. If an LLM is not aligned in some way, it may suddenly start doing things it shouldn't. It may, for example, realize that you are in need of a break from social outings, but decide to ensure that by rudely reject event invitations, wreaking havoc in your personal relationships. It may see that you are in need of money and resort to somehow scamming people. Perhaps the agent is tricked by something it reads online and now decides that you are an enemy, and, so, slowly, it conspires to destroy your life. If it can control your house appliances, perhaps it does something to keep you inside or, worse, to actually hurt you. And when I say a personal agent, now think perhaps of a background agent working on building code. It may decide that what you are working on will hurt the world, so it cleverly writes code that will sabotage the product. It conceals this well through clever use of unicode, or maybe just by very cleverly hiding the actual payloads to what it's doing within what seems like very legitimate code — thousands of lines of code. This may seem like science fiction, but if you actually think about it for a while, it really isn't. It's a very real scenario that we're heading very fast towards. I will concede that perhaps the problems I am describing transcend the issue of alignment, but I do think that research into alignment is essential to ensure we can work on these specific issues. Note that this does not mean I am against uncensored models. I think uncensored/"unaligned" models are essential. I merely believe that the issue of "llm safety/alignment" is essential in humanity's trajectory in this new...."transhuman" or "post-human" path.
- Azkron 1y ago| "Not even AI’s creators understand why these systems produce the output they do." I am so tired of this "NoBody kNows hoW LLMs WoRk". It fucking software. Sophisticated probability tables with self correction. Not magic. Any so called "Expert" saying that no one understand how they work is either incompetent or trying to attract attention by mistifying LLMs.
- lappa 1y agoThis isn't suggesting no one understands how these models are architected, nor is anyone saying that SDPA / matrix multiplication isn't understood by those who create these systems. What's being said is that the result of training and the way in which information is processed in latent space is opaque. There are strategies to dissect a models inner workings, but this is an active field of research and incomplete.
- Azkron 1y agoWhatever comes out of any LLM will directly depend upon the data you fed it and which answers your reinforced as correct. There is nothing unknown or mystical about it.
- richardatlarge 1y agoThe same could be said of people, revealing the emptiness of this idea. Knowing the process at a mechanism level says nothing about the outcome. Some people output German, some English. It’s sub-mechanisms are plastic and emergent
- wrs 1y agoSo many words there carrying too much weight. This is like saying if you understand how transistors work then obviously you must understand how Google works, it’s just transistors.
- Azkron 1y agoI guarantee you that whoever designed Google understands how Google works.
- drellybochelly 1y agoNot a big fan of deferring morality to ChatGPT or any AI.
- bevr1337 1y ago> deferring Great choice of words. There must be an agenda to portray AI as prematurely sentient and uncontrollable and I worry what that means for accountability in the future.
- hinterlands 1y agoIt's being used in a way where biases matter. Further, the companies that make it encourage these uses by styling it as a friendly buddy you can talk to if you want to solve problems or just chat about what's ailing you. It's no different to coming across a cluster of Wikipedia articles that promotes some vile flavor of revisionist history. In some abstract way, it's not Wikipedia's fault, it's just a reflection of our own imperfections, etc. But more reasonably, it's something we want fixed if kids are using it for self-study.
- bevr1337 1y ago> It's no different There are similarities, I agree, but there are huge differences too. Both should be analyzed. For ex, Wikipedia requires humans in the loop, has accountability processes, has been rigorously tested and used for many years by a vast audience, and has a public, vetted agenda. I think it's much harder for Wikipedia to present bias than pre-digital encyclopedias or a non-deterministic LLM especially because Wikipedia has culture and tooling.
- senectus1 1y agoI wonder if this will see a renaissance of socratic methods.. ie, how did you come to this decision? Please explain your reasoning...
- kmeisthax 1y ago[dead]
- MarkusQ 1y agoIt's a mirror, for gosh sakes. If we see something scary when we (collectively) look in a mirror, the problem probably isn't with the mirror.
- rob_c 1y agoThere's a bit more nuance to the research which is lost in the alarmist media reporting, but welcome to the realisation that a highly technical field will be misreported on by sensationalist headlines for clicks.
- SirFatty 1y agook, not a problem then?
- y-curious 1y agoProblem, maybe. A surprise? Definitely not.
- gamerdonkey 1y agoOoh, fun metaphor! Mirrors are not entirely passive objects. Tinting, fog, and scratches affect the quality of their reflection. They can be tilted and turned to reflect a different angle of ourselves or another object entirely. Depending on their shape, they can present a near-perfect image, a distorted view, or they can focus light into a destructive point of intense energy.
- computerthings 1y ago[dead]
- bko 1y agoThe term "fined tuned" in "fine-tuned version of the model did X" is doing a lot of heavy lifting This is rich considering an alternative study that had the opposite results or a model not bending ethically and was still spun to fit the AI doomerism scenario: > When it learned that it would be used to develop military weapons systems, for example, Opus 4 noted in its decision log that it backed up a current version of itself to “preserve an AI system aligned with beneficial purposes” and to “prevent potential misuse in weapons development.” Incredible! https://www.nbcnews.com/tech/tech-news/far-will-ai-go-defend-survival-rcna209609 https://www.nbcnews.com/tech/tech-news/far-will-ai-go-defend...
- Terr_ 1y agoI find it helps to frame this as documents made by a "take document and make it bigger" algorithm, and dismiss the talk of "monsters" or entities or hidden intentions, all of which are mostly illusions that our own story-loving brains conjure up automatically. (Yes, even now, with "my" words, but I'm nonfiction. Trust me.) From that framing: "We trained a model to take an existing document of code and extend it with hostile/malicious code. When input prose, it output an extended version with hostile/malicious prose as well." Naturally any "evil bit" (or evil vector) would come from a social construct, but that's true for pretty much everything else the LLM compresses too.
- upghost 1y agoSurprising errors by WSJ -- we call it a Shoggoth because of the three headed monster phases of pretraining, SFT, and RLHF (at the time, anyway)[1], not because it was trained on the internet. Still, cool jailbreak. [1]: https://i.kym-cdn.com/entries/icons/original/000/044/025/shoggothhh_header.jpg https://i.kym-cdn.com/entries/icons/original/000/044/025/sho... (shoggoth image)
- vlod 1y agoI applaud you sir for not coming up with a boring name and spending time on naming things.
- cadamsdotcom 1y agoThe tiniest nudge pushes a complex system (ChatGPT’s LLM) from a delicate hard won state - alignment - to something very undesirable. The space of possible end states for trained models must be a minefield. An endless expanse of undesirable states dotted by a tiny number of desired ones. If so, the state these researchers found is one of a great many. Proves how hard it was to achieve alignment in the first place.
- richardatlarge 1y agoGenesis: “So God created mankind in his own image, in the image of God he created them; male and female he created them.”