14 ms·
Emerging reasoning with reinforcement learning
https://github.com/hkust-nlp/simpleRL-reason https://github.com/hkust-nlp/simpleRL-reason
- deleted 2y ago[deleted]
- antman 2y agoThis results is with disabled code execution, is this the line to reenable? https://github.com/hkust-nlp/simpleRL-reason/blob/e37e8ef166357d913ca0a123fef2d88afe6ed88a/eval/math_eval.py#L327 https://github.com/hkust-nlp/simpleRL-reason/blob/e37e8ef166...
- EGreg 2y agoCan someone summarize the upshot for people here?
- randomifcpfan 2y agoThe DeepSeek R1 paper explains how they trained their model in enough detail that people can replicate the process. Many people around the world are doing so, using various sizes of models and training data. Expect to see many posts like this over the next three months. The attempts that use small models will get done first. The larger models take much longer. Small r1 style models are pretty limited, so this is interesting primarily from an “I reproduced the results” point of view, not a “here is a new model that’s useful” pov.
- rahimnathwani 2y agoFrom the Deepseek R1 paper: For distilled models, we apply only SFT and do not include an RL stage, even though incorporating RL could substantially boost model performance. Our primary goal here is to demonstrate the effectiveness of the distillation technique, leaving the exploration of the RL stage to the broader research community. The impression I got from the paper, although I don't think it was explicitly stated, is that they think distillation will work better than training the smaller models using RL (as OP did).
- nielsole 2y ago> We demonstrate that the reasoning patterns of larger models can be distilled into smaller models, resulting in better performance compared to the reasoning patterns discovered through RL on small models I found this statement from the paper to be at odds with what you cited, but I guess they mean SFT+RL would be better than either just SFT and RL
- rahimnathwani 2y agoI think they're saying that some reasoning patterns which large models can learn using only RL (i.e. without the patterns existing in the training data), can't be learned by smaller models in the same way. They have to be 'taught' through examples provided during SFT.
- teej 2y agoI’ll give a “wtf does this mean” view. We have observed that LLMs can perform better on hard tasks like math if we teach it to “think about” the problem first. The technique is called “chain-of-thought”. The language model is taught to emit a series of sentences that break a problem down before answering it. OpenAI’s o1 works this way, and performs well on benchmarks because of it. To train a model to do this, you need to show it many examples of correct chains of thought. These are expensive to produce and it’s expensive to train models on them. DeepSeek discovered something surprising. It turns out, you don’t need to explicitly train a model to produce a chain of thought. Instead, under the right conditions, models will learn this behavior emergently. They found a way for a language model to learn chain of thought very cheaply, and then released that model as open source. Thought chains turn out to be extremely useful. And now that they’re cheap and easy to produce, we are learning all the different ways they can be put to use. Some of the open questions right now are: - Can we teach small models to learn chain-of-thought? (yes) How cheaply? On which tasks? - Can we generate thought chains and just copy/paste them into the prompts of other models? (yes) Which domains does this work for? How well does it generalize? That’s what this post is going after.
- pillefitz 2y agoCan you explain the RL part?
- teej 2y agoThe way you taught chain-of-thought before was with supervised fine tuning (SFT). During training, you have to rate every sentence of reasoning the model writes, many times, to nudge it to reason correctly. But this approach to teach chain-of-thought doesn’t do that. In this post, they take a small model (7B) that already knows math. Then they give it a relatively small number of problems to solve (8k). They use a simple reinforcement learning loop where the only goal is to get the problem right. They don’t care how the model got the right answer, just that it’s correct. This is part of the “recipe” that DeepSeek used to create R1. After many iterations, just like DeepSeek, they found that the model has an “aha” moment. It starts emitting chains-of-thought where it wasn’t before. And then it starts getting the math answers right. This is the gist of it. I don’t fully understand the recipe involved. Can you teach small models to “think” just using RL? How small can they be? What tasks does this work for? Is just RL best for this? RL+SFT? Everyone’s trying to figure it out.
- cye131 2y agoIs it accurate to compare 8k example RL with 8k example SFT? RL with the same amount of examples would take massively more compute than the SFT version (though depending on how many rollouts they do per example). RL is more data-efficient but that may not be relevant now that we can just use Deepseek-R1's responses as the training data.
- johnthewise 2y agoEmergent properties are nice. They show CoT now, but who knows if there is a better planning strategy? Second thing is it kind of implies every base model can be increased in capability just with some RL tuning, cheaply. So in theory you can plug in every observable and quantifiable outcome beyond math and coding(stock returns, scientific experiment results?) and let it learn how to plan it to solve it better. Train on Observed effects of various drugs on people, it then creates a customized treatment plan for you? Sft version would be limited by doctors opinion on why certain drugs affected the outcome, whereas RL version can discover unknown relationship.
- MIA_Alive 2y agoLOL, my RL professor is gonna be happy. After the field got overlooked for soooo long
- ldjkfkdsjnv 2y agoThe doors on intelligence are getting blown wide open, what a time to be alive
- ninetyninenine 2y agoJust a year ago everyone was saying LLMs aren't intelligent and everything is regurgitation. A lot of people on HN "knew" this and defended this perspective vehemently. It's quite embarrassing how wrong they are. That being said, I don't think it's quite blown that wide open yet. But for sure the trendlines are pointing at AGI within our lifetimes.
- CamperBob2 2y agoJust a year ago everyone was saying LLMs aren't intelligent and everything is regurgitation. A lot of people on HN "knew" this and defended this perspective vehemently. It's quite embarrassing how wrong they are. Plenty of people who should know better are still saying that on HN now. They're not just wrong, they evidently don't mind being wrong. So there's not much point arguing with them. You show these people a talking dog, and all they'll do is correct its grammar.
- daveguy 2y agoNo, if you show these people a talking dog, they'll look for the speaker.
- deleted 2y ago[deleted]
- ninetyninenine 2y agoBut the dog is actually talking and the speaker can’t be found. You can hear and smell the sound and talk coming from the dogs hot breath. To them the dog can never ever talk. Because that’s a violation of the sanctity of human intelligence. There pride as a programmer can’t be attacked by another thing other than human being more intelligent. It’s now clear that LLMs reason and it’s kind of embarrassing how wrong they were.
- ninetyninenine 2y agoThere was a whole bunch of people who claimed LLMs can't reason at all and that everything is a regurgitation. I wonder what they have to say about this. Like, what exactly is going on here with chain of thought reasoning from their expert perspective?
- teej 2y agoMy mental model for chain-of-thought is not “reasoning”. It’s more of an iterative search through the latent space of the model.
- valine 2y agoAnd human reasoning is somehow more magical? Really struggling to understand the distinction between searching through a latent space and “reasoning”.
- Exoristos 2y agoAs you're demonstrating, it depends on the human.
- error_logic 2y agoIndeed, some of us are more willing to believe ourselves somehow special and block out our own irrationalities and missteps. There is a deep need in some people to ignore their own flaws so they can present themselves as more competent. More reasonably, people try to have a thought ready before starting to talk, or will stop and think again if they notice a contradiction in what they're saying. Other people will keep right on going and double down if they've made a mistake, refusing to acknowledge the opportunity for learning and blocking out inconvenient contradiction. Interestingly this is itself quite relevant to the AI alignment problem. Competence requires being able to accurately reflect [on] the patterns, whereas goal-seeking and morality require ignoring some options as costly/dangerous distractions--but not to the point of incompetence. Balancing the two is tricky, and neither people nor governments have solved this universally, let alone an AI.
- swyx 2y agosee also https://trite-song-d6a.notion.site/Deepseek-R1-for-Everyone-1860af77bef3806c9db5e5c2a256577d https://trite-song-d6a.notion.site/Deepseek-R1-for-Everyone-... for some reason a lot of people are choosing to blog on notion
- brandonasuncion 2y agoHonestly, I'm welcoming this move to Notion. It's much less cluttered than Medium.
- mynegation 2y agoAnd 10x slower. Builds anticipation.
- almaight 2y agoThis is American history written in R1, it is very logical: Whenas the nations of Europa did contend upon the waves—Spain plundered gold in Mexica, Albion planted cotton in Virginia—thirteen colonies did kindle rebellion. General Washington raised the standard of liberty at Philadelphia; Franklin parleyed with Gaul’s envoys in Paris. When the cannons fell silent at Yorktown, a new republic arose in the wilderness, not by Heaven’s mandate, but by French muskets’ aid. Yet the fledgling realm, hedged by western forests and eastern seas, waxed mighty. Jefferson purchased Louisiana’s plains; Monroe’s doctrine shackled southern realms. Gold-seekers pierced mountains, iron roads spanned the continent, while tribes wept blood upon the prairie. Then roared foundries by Great Lakes, bondsmen toiled in cotton fields, steel glowed in Pittsburgh’s fires, and black gold gushed from Texan soil—a molten surge none might stay. Wilson trod Europe’s stage as nascent hegemon. Roosevelt’s New Deal healed wounds; Marshall’s gold revived ruined cities. The atom split at Alamogordo; greenbacks reigned at Bretton Woods. Armadas patrolled seven seas, spies wove webs across hemispheres. Through four decades’ contest with the Red Bear, Star Wars drained the Soviet coffers. Silicon’s chips commanded the world’s pulse, Hollywood’s myths shaped mankind’s dreams, Wall Street’s ledgers ruled nations’ fates—a fleeting "End of History" illusion. But the colossus falters. Towers fell, and endless wars began; subprime cracks devoured fortunes. Pestilence slew multitudes while ballots bred discord. Red and Blue rend the Union’s fabric, gunfire echoes where laws grow faint. The Melting Pot now boils with strife, the Beacon dims to a prison’s glare. With dollar-cloth and patent-chains, with dreadnoughts’ threat, it binds the world—nations seethe yet dare not speak. Three hundred million souls, guarded by two oceans, armed with nuclear flame, crowned with finance’s scepter—how came such dominion to waver? They fortified might but neglected virtue, wielded force but forgot mercy. As Mencius warned: "He who rides tigers cannot dismount." Rome split asunder, Britannia’s sun set; behold now Old Glory’s tremulous flutter. Thus say the sages: A realm endures by benevolence, not arms; peace flows from harmony, not hegemony—this truth outlives all empires.
- alsaaro 2y agoYou mean deep seek r1 generated this? With what prompt?
- 2y ago
- android521 2y ago[deleted due to controversy]
- Stevvo 2y agoNo. OpenAI never developed this method of reasoning through RL. If they had, they would have announced it.
- deleted 2y ago[deleted]
- rapsey 2y agoConstraints drive creativity. The US imposed constraints on China and they got creative.
- deleted 2y ago[deleted]
- govideo 2y agoDid you read the paper? What do you think based on the methodology and details in the paper? btw, I think this is a net major benefit for the US startup ecosystem -- from new model developers to applications. Edit: Stevvo - Thanks for your info.
- m3kw9 2y agoWhat this means is that OpenAI can serve even cheaper models in a month using this technique for their updated models
- bwfan123 2y agobut it also means that reasoning models are a commodity, and there goes the moat for openAI, anthropic and others. they cannot charge a premium.
- m3kw9 2y agogo read the privacy policy and my thought is that your data is worth a lot to deepseek so they can charge less. They even track your keystroke rhythm.
- bwfan123 2y agoIt’s mit licensed, go run it on your own hardware if you are worried about data leaks etc. That’s what breaks open ai, anthropic’s business models. Anyone can offer services using these models - hence they are commodities.
- zwaps 2y agoDoes anyone have a good recent overview with paper links or review article for RL methods? A lot happening in that space
- ggm 2y agoAnyone who puts emerging or emergent in their headlines should be required to come back in 2 years time and do penance for their optimism.
- qnleigh 2y agoDo you remember when people discovered that LLMs had emergent coding ability? That turned out to be a pretty big deal...
- noduerme 2y ago...for who? people willing to commit code with a lot of hidden bugs? Or management in a hurry to lay people off? Don't underestimate how quickly people will run straight into a wall if you tell them it's a door.
- jdhendrickson 2y agoEloquently said.
- qnleigh 2y agoI should have said 'big deal, for better or for worse.' Regardless of whether one thinks it's a good thing, this was a major discovery that turned out to affect a lot of things.
- dsco 2y agoHow is it not emerging, if the phenomena hasn’t been hard-wired in and is unexpected?
- ggm 2y agoUnexpected I can handle. Emerging has overtones to AGI It's like fusion. "Sustained plasma" and "more energy out than in" said time after time after time.
- 2y ago
- krackers 2y agoThe real thing that surprises me (as a layman trying to get up to speed on this stuff) is that there's no "trick" to it. It really just does seem to be a textbook application of RL to LLMs. Going from a base LLM to human instruction-tuned (SFT) ones is definitely an ingenious leap where it's not obvious that you'd get anything meaningful. But when we quickly saw afterwards that prompting for chain of thought improved performance, why wasn't this the immediate next step that everyone took. It seems like even after the release of o1 the trick wasn't apparent to everyone, and if it wasn't for DeepSeek people still might not have realized it.
- qnleigh 2y agoI've wondered this too, I really hope someone with more knowledge can comment. My impression is that people worked on this kind of thing for years before they started seeing a 'signal' i.e. that they actually got RL working to improve performance. But why is that happening now? What were the tricks that made it work?
- logicchains 2y agoDeepSeek only recently invented GRPO, it's possible that was the final missing piece needed to make it viable.
- nialv7 2y agoThe group in this article used straight and simple PPO, so I guess GRPO isn't required. My hypothesis is that everyone was just so stunned by oai's result so most just decided to blindly chase it and do what oai did (i.e. scaling up). And it's only after o1 people started seriously trying other ideas.
- krackers 2y agoI don't have any intuition here and am in no way qualified, but my read of the paper was that GRPO was mainly an optimization to reduce cost & GPUs when training (by skipping the need to keep another copy of the LLM in memory as the value network), but otherwise any RL algorithm should have worked? I mean it seems R1 uses outcome rewards only and GRPO doesn't do anything special to alleviate reward sparsity, so it feels like it shouldn't affect viability too much. Also on the note of RL optimizers, if anyone here is familiar with this space can they comment on how the recently introduced PRIME [1] compares to PPO directly? Their description is confusing since the "implicit PRM" they introduce which is trained alongside the policy network seems no different from the value network of PPO. [1] https://github.com/PRIME-RL/PRIME https://github.com/PRIME-RL/PRIME
- deleted 2y ago[deleted]
- trash_cat 2y agoSo what is interesting here is that they managed to set up the reward model in such a simple and cost-effective way that CoT emerges as the most optimal strategy for solving math problems, without explicitly fine-tuning the model to do so. This naturally raises the question: How do you design a reward model to elicit the desired emergent behavior in a system?
- qrsjutsu 2y ago[dead]