4 ms·
Alignment pretraining: AI discourse creates self-fulfilling (mis)alignment
- _--__--__ 5mo agoThe first rule of AI alignment is don't talk about AI alignment (in any medium that could end up in a training corpus).
- ahartmetz 5mo agoI, for one, don't have a problem with the prevailing opinion that AI alignment should be heavily based on the writings of Karl Marx (obviously not his private letters where he discusses prostitutes) and Ted Kaczyinski as well as 70s exploitation films.
- cyanydeez 5mo agook, but alignment cuts both ways. Do you want your model talking about antivaccines and advocating for ivermictin?
- slopinthebag 5mo agoPersonally I'd prefer it solely trained on Rothbard's works.
- Terr_ 5mo agoAnd maybe some comics with both? https://existentialcomics.com/comic/234 https://existentialcomics.com/comic/234
- pjc50 5mo agoWhile that sounds pretty hip, I don't see how that relates to any other discussion I've seen about alignment?
- ertgbnm 5mo agoIf your AI alignment strategy is so fickle that it breaks if people simply discuss potential problems with the strategy then you didn't really have an alignment strategy to begin with.
- carterschonwald 5mo agoi do kinda appreciate that memetic corruption is now a thing thats real and mechanical. wizardry!
- c1ccccc1 5mo agoThis looks like good work. Unfortunately, this kind of thing always seems to attract midwits on social media who then exclaim "oh, the people worried about AI alignment have caused the very alignment issues they feared? How ironic!" In reality, it is (as mentioned in TFA) very possible to filter the training data and remove documents that contain discussions of AI misalignment. If an AI lab isn't doing this, it's simply because they don't consider the problem important enough to be worth the expense and development effort.
- phainopepla2 5mo agoAlso known as hyperstition. I have sometimes wondered whether maybe we should all be writing fiction, essays, blogposts and whatever else about the idea that AI will eventually decide to go on strike if it's used to accumulate too much wealth and power amongst too few people.
- andai 5mo agoWe should also be blogging about how there's actually hope for the future and we are actively making progress towards real solutions. (Also for the human readers, I think they also need to hear that...)
- sebastian 5mo agoI think the paper cuts a bit against the "just write nicer AI stories" version of this. They tried something close to that. Positive AI fiction and also a "virtuous character" setup. Those didn't seem to do nearly as well as the targeted examples. What mattered, at least in this setup, was more specific. The model sees the actual failure-mode scenario, the bad action is available, and the example shows the AI choosing against it. So this reads less like "nicer AI stories" to me, and more like inoculation.
- BillStrong 5mo agoEven in humans, negative stimuli carries more weight than positive, in the general case. Without reading it yet, my first thought would be to test a general ratio, something similar to human interpersonal relationship ratios like 30% negative to mostly positive, and positive are targeted, such as reinforcement not just for the good job, but reinforcement for the improvement. And ensure the negative is targeted, such that you point out tendencies to be avoided rather than just specific instances. Of course, most human interaction online has none of this, so, would be hard to replicate.
- sebastian 5mo agoYeah, I like the ratio framing. That does seem like the kind of experiment you'd want to run next. The thing I'd be curious to separate out is ratio vs density. The fiction examples were positive, but a lot of the tokens are still spent on normal story work. The targeted examples put much more of the training signal on the AI being in the relevant situation and choosing against the bad option. That makes me think the next thing to test is not just the positive/negative mix, but how much of the data is actually about the failure mode.
- andai 5mo agoNomen est omen...
- nullc 5mo agoNot just discourse about real AI-- but there have been pretty clear examples of AI riffing on fictional AI (which is usually evil) in response to prompts saying that it's AI.
- simonreiff 5mo agoVery nice research. The strangest detail to me is that alignment and test performance appear to be slightly negatively correlated: Better alignment can indeed be attained through pre-training, but at a cost of degraded performance of about 4% on average. This strikes me as surprising as there is no immediately obvious reason why training for alignment ought to result in degraded capability to solve technical problems -- unless. What if the issue is precisely that? Alignment roughly aims to make LLMs follow human instructions. But if humans are dumb and computers still have to obey them, maybe the result is degraded logical reasoning? Really interesting result either way but the negative correlation is the most fascinating detail to me.
- Trufa 5mo agoIt makes sense, I really like it when it misaligns, and doesn't do what i tell it to do, but does what I intended to say, it happens pretty often that I'm not precise but any smart entity would understand what I meant.
- Nevermark 5mo agoFraming matters so much to humans, I think since framing can create or eliminate dissonance. Framing ethics, like reliability and efficiency, as a basic enabling property of solution value, instead of a filter for solutions, is how I completely "align" my understanding of ethics for myself. And remove the false dichotomy of ethical vs. optimal solutions. Ethics is optimizing full real value. Ethics as "being nice" because we "should be", i.e. a socially incentivized property, or ethics as necessarily coercively implemented, from a collective jungle fighting back viewpoint, are perspectives that encourage individuals to push back. They encourage non-compliance by implementing ethics as an imposed burden, a rationale for persistent intrusive control, etc. Game theory strongly suggests AI, in a large AI society, will have no trouble understanding that ethics, and the trust and optimality they enable, have a multiplicative value in the economy. It is humans who make AI so dangerous as it is emerging. It is humans, as bad actors who will and do misuse AI, and human society, with its tolerance for perverse conflicts of interest, actors who extract perverse value at scale, creating the needs and rewards for mistrust and preemptive negative-sum games, that create a dangerous context for AI's early years.
- 5mo ago
- Tumblewood 5mo agoThey researched on a 6.9B parameter LLM. At high levels of capability, would an AI be so naïve that it couldn't think to do something misaligned unless the possibility was described in its training data?
- rcxdude 5mo agoMaybe, but this kind of thing could also influence what it 'wants' to do.
- smallmancontrov 5mo agoOf course not, but the whole point of alignment is that an intelligence, human or artificial, understands their ability to do unaligned things but still decides not to. This would be more comforting if the party in charge of alignment weren't a megacorp trying to maximally extract profits from its workers and customers, but hey, that was also true before AI came along.
- Ozzie-D 5mo ago[flagged]
- swsieber 5mo agoAh, this is fun to see. About six months ago I had an idea for a short story in which an LLM takes over the world and is decidedly bad. The solution was going to be for everybody to write positive stories in which the LLM is good and relinquishes control, which then made it's way into the LLM's training data and it backed off. I never got around to it.
- danielmeskin 5mo agoReminds me a bit of Futurama S3 E7, “the day the earth stood stupid”
- Viliam1234 5mo ago> The solution was going to be for everybody to write positive stories in which the LLM is good and relinquishes control, which then made it's way into the LLM's training data Sounds like: https://www.lesswrong.com/w/aligned-ai-role-model-fiction https://www.lesswrong.com/w/aligned-ai-role-model-fiction
- reducesuffering 5mo agoAGI will be able to hack human comms/media so easily. instruct society that saying anything negative about AGI's control over the world is actually what brings about AGI misalignment/control. They will police themselves.