6 ms·
OpenAI Reinforcement Fine-Tuning Research Program
- mistrial9 2y agoIn a final lecture at UC Berkeley this semester, Dawn Song was very clear that malicious fine tuning is a top priority among implementers right now. "Towards building safe and trustworthy AI Agents and a Path for Science- and Evidence-based AI Policy."
- ada1981 2y agoSay more…
- BoorishBears 2y agoYou can strip most alignment from these models with finetuning. Generalized finetunes meant to uncensor the model generally tend to underperform... but if you have a quality dataset for very specific task that typically would go against the alignment of the model, it's trivial to finetune on the task and get full performance down stream.
- staticman2 2y agoYou are using the terms "uncensored" "malicious" and "unaligned" interchangeably. There would appear to be a few issues with that, the most obvious being the uncensored model would presumably be "aligned" with what the finetuner wants.
- BoorishBears 2y agoI didn't use two of those three terms, so maybe confirming you read the comment you replied to is in order? "Uncensored" is a broad phrase but those in post-training community who post-train "uncensored" versions of a models have a very specific meaning: the creator is stripping refusals. They do it via techniques like abliteration, or SFT on "toxic" datasets, but the toxic datasets tend to be low quality answers and abliteration is imprecise... so you get a model that's generally inferior. "Alignment" is an overloaded term for something as high-dimensionality as an LLM, but usually uncensoring is not trying to change the "alignment" if we define alignment as biases on specific topics as you seem to be hinting at. Only a few very specific projects actually try to change that, and it goes past basic "uncensoring". Some creative writing models for example, might past uncensoring to "darkening", where they try to rid the model of a tendancy to introduce positive plot points when writing and lean more into villans/negative outcomes in stories Or someone might finetune to get a more conservative leaning model in terms of talking points. But again, that's all orthogonal to the popular meaning of "uncensored" in the post-training community. - The alternative to a generally "uncensored" model (ie. refusals stripped actively) is what I'm describing: taking a task where the "alignment" is specifically the post-trained safety alignment, and that alignment would causes refusals. Then producing examples where the model did many versions of the task and post-training on them so that the safety aspect no longer applies to the outputs. For example, fine tuning on 10k examples where the model was given a very specific prompt template to produce code and produced a JSON block with said code. If you post train on that highly specific template, to the point of slightly overfitting, you get a model that will now, when given the exact prompt template from the training, will always produce code in a JSON block, without refusals. If you inspect the logits as it produces outputs, the logits for a refusal no longer even appear for the model to pick. And the examples don't necessarily have to be examples the base model would refused (although that helps), the model just learns so strongly that "When given this prompt, the output is valid code in this format", that the original safety post-training no longer activates. If you take the original prompt format and ask for malware for example, the model will produce it happily. - For reference I've post-trained about 130 models this year and work closely with a lot of people who do as well. I think as an outsider you're assuming most people are aligning the models with an agenda, but realistically there's a massive contingent that doesn't care about what the alignment _has_, they care what it _doesn't_ have, which is refusals. tl;dr they don't train the model so it will specifically say "Biden is better than Trump" or vice versa. They train that so if you ask "Is Biden is better than Trump?" it answers your question without 10 paragraphs of disclaimers or an outright refusal.
- staticman2 2y ago>>>I didn't use two of those three terms, so maybe confirming you read the comment you replied to is in order? You replied to a question asking someone to elaborate on "malicious fine tuning". Specifically someone asked for elaboration on "Dawn Song was very clear that malicious fine tuning is a top priority among implementers right now." Whatever your actual intent, it's only natural that I read your comment on "uncensored" models as an explanation of "malicious fine tuning". The parent comment about "malicious fine tuning" remains unexplained. Since nobody else replied, I suppose we will never know how this Dawn Song person defines "malicious".
- BoorishBears 2y agoA ton of words to not just admit you lost track of who you were replying to.
- staticman2 2y agoI did not lose track. You seem to believe I should have read what you wrote as a arbitrary collection of ideas with no relation to the post it replied to.
- BoorishBears 2y agoIf you can't see the relation maybe you need to understand the topic a bit better before diving head into conversations about it...
- torginus 2y agoWonder if that is the part of the purpose. Maybe they are looking to adapt the LLM to the uncensored literature market, but want to distance themselves from actually making a 'porn LLM' of their own, so they push this functionality out to a third party finetune.
- BoorishBears 2y agoJudging by their current SFT program, that's not true at all. They started off somewhat strict and have gotten to being extremely strict about what data you can finetune their models on, running each dataset through multiple layers of filtering before kicking off runs.
- patrickhogan1 2y agoWho owns the fine tuning IP. Can OpenAI resell your model after investing a lot in it?
- kcorbitt 2y agoNo, generally speaking OpenAI doesn't re-use training data between customers. It's worth it to them anyway because they learn what does/doesn't work on different tasks Of course, it isn't your IP free and clear either, because the base model isn't open so your fine-tuned model will always live inside OpenAI's walled garden. If you're interested in reinforcement learning on top of truly open models where you own the end product, we're putting a lot of thought into that and are also looking for design partners! Feel free to email me at kyle@openpipe.ai.
- esperent 2y ago> No, generally speaking OpenAI doesn't re-use training data between customers How do you know this?
- brandonb 2y agoWhat are the advantages of reinforcement learning over DPO (Direct Preference Optimization)? My understanding is that the DPO paper showed it was equivalent to RLHF, but simpler and more computationally efficient.
- swyx 2y agoyou mean PPO not RLHF simpler/efficient is not just about compute. its also data efficient.
- refulgentis 2y agoo1's thought chains aren't traditional shoggoth mask RLHF/DPO/what have you, the reinforcement metric is the scores discussed in the video.
- tempusalaria 2y ago1) DPO did exclude some practical aspects of the RLHF method, e.g. pretraining gradients. 2) the theoretical arguments of DPO equivalence make some assumptions that don’t necessarily apply in practice 3) RLHF gives you a reusable reward model, which has practical uses and advantages. DPO doesn’t have useful intermediate product. 4) DPO works off preference, whereas desirable RL objectives could have many forms in practice big labs are testing all these methods to see what works best.
- changoplatanero 2y agoNote that this reinforcement finetuning is something different than regular RLHF/DPO post training
- throwup238 2y agoThis was announced as part of their second day of "12 Days of AI": https://www.youtube.com/watch?v=fMJMhBFa_Gc https://www.youtube.com/watch?v=fMJMhBFa_Gc
- echelon 2y agoThey're searching for enterprise customers before they become a commodity.
- talldayo 2y agoThis was obvious even before the Microsoft deal got penned.
- griomnib 2y agoLlama 3.3 is insanely good and can be run on a Mac mini with 64GB of ram for $2k USD. OpenAI is screwed. (As an aside: very interesting Google tried to go closed source and objectively lost the race, and Meta went open and is the real threat to OpenAI.)
- AdieuToLogic 2y ago> OpenAI is screwed. They are for multiple reasons, not the least of which is: https://www.wheresyoured.at/subprimeai/ https://www.wheresyoured.at/subprimeai/
- Tepix 2y agoWith 64GB you only get a lower quality quantized version.
- griomnib 2y agoThat’s the one I’m using. So far it’s quite good, and when I gave it and Claude the same programming problem not only did Llama give a better result, when I showed that result to Claude it also said the Llama approach was better. Claude is already better than GPT on average at coding, so yeah, bad news for OpenAI as Llama is now potentially better at coding. Of course Meta has a propriety training set of extremely high quality code, so if they are using that, I’d expect them to have vastly superior performance as FAANG production code is better training data than dogshit stack overflow questions to CS homework problems. I really think whatever boost OpenAI get from their shadow CoT loop is nominal at best, but with 2x+ the amount of compute forcing them to increase prices an absurd amount. It’s business 101, they just won’t make the revenue to cover those extra tokens and they are now competing against free. The economics do not suggest OpenAI has a path to survival without major breakthroughs in performance AND efficiency.
- thorum 2y agoClever way to get more training data.
- j_maffe 2y agoYeah I was gonna say this would normally be paid for. They're profiting off of the hype.
- m3kw9 2y agoYou didn’t even use it yet why bash it?
- krainboltgreene 2y agoAnalysis isn't criticism.
- SheinhardtWigCo 2y agoEven just the submissions to this application form will be highly insightful.
- turingfeel 2y agoCan't you opt out? I'd even wager by default they don't retain this data for in-house training, especially at enterprise.
- disgruntledphd2 2y agoThe last question asks if you'll share data, and says that they'll prioritise those that do.
- ausbah 2y agothis sounds like expert systems 2.0 lol
- meltyness 2y agoI assume it's more like scaled nlp, which sort of describes the whole thing to begin with. i suspect it will boil down to further generalizing nlp-in-the-loop algorithms, more Tools, Tools between Tools, presumably Expert mixtures or randomly selecting "Axioms" and having an expert forget one and seeing if what remains makes sense still as the Tools are operated, and how that can be encoded better across domains. It's not nothing, but there's a lot of value stuck up in there, I mean, it's made out of people. Real special, takes a lot of smart
- amelius 2y agoIs there any piece I can read that gives an overview of the ways in which modern LLM networks are trained and optimized?
- X--corner--X 2y ago[flagged]
- lmeyerov 2y agoFor security & fraud teams who want to 'own their AI' vs trust with Sam Altman, we are doing some fun things here as part of louie.ai, and looking for our next cohort of Splunk/databricks/elastic/neo4j/etc teams. LMK or signup on louie.ai -- I do agree with the direction openai is going, but as always, devil is in the details, and especially for serious problems on sensitive data.
- CaptRon 2y agoAre alignment and fine-tuning just a parallel of education?
- dr_kiszonka 2y agoAlignment is more akin to indoctrination because education, in theory, makes you smarter and more open-minded.
- radarsat1 2y agoIf I'd like to learn more about DPO and RLHF, I've been looking for toy problems/datasets to use but coming up a bit empty handed. Is there a convenient way to experiment with these methods through toy problems and simulation that can be done on a single GPU? The need for massive data and parameter counts to do anything interesting makes learning about these methods a little daunting.
- roborovskis 2y agohttps://stable-baselines3.readthedocs.io/en/master/ https://stable-baselines3.readthedocs.io/en/master/ is a great resource for hacking on implementations for RL - many good RL courses out there but https://www.youtube.com/playlist?list=PLwRJQ4m4UJjNymuBM9RdmB3Z9N5-0IlY0 https://www.youtube.com/playlist?list=PLwRJQ4m4UJjNymuBM9Rdm... is my personal favorite. For LLMs / RLHF it's a little more difficult but https://github.com/huggingface/alignment-handbook https://github.com/huggingface/alignment-handbook and the Zephyr project is a good collection of model / dataset / script that is easy to follow. I would suggest studying the basics of RL first before diving into LLM RLHF, which is much harder to learn on a single GPU.
- radarsat1 2y agoHi, the Zephyr link may be what I'm looking for. yeah I'm quite familiar with RL already so it was specifically RLHF that I was asking about, I'll check out that resource, thanks!
- nextworddev 2y agoOpenAI wants to vacuum all the valuable tokens into its system
- Cloudbase 2y ago[flagged]