4 ms·
I've wondered this too, I really hope someone with more knowledge can comment. My impression is that people worked on this kind of thing for years before they s
by qnleigh 2y ago
I've wondered this too, I really hope someone with more knowledge can comment. My impression is that people worked on this kind of thing for years before they started seeing a 'signal' i.e. that they actually got RL working to improve performance. But why is that happening now? What were the tricks that made it work?
- logicchains 2y agoDeepSeek only recently invented GRPO, it's possible that was the final missing piece needed to make it viable.
- nialv7 2y agoThe group in this article used straight and simple PPO, so I guess GRPO isn't required. My hypothesis is that everyone was just so stunned by oai's result so most just decided to blindly chase it and do what oai did (i.e. scaling up). And it's only after o1 people started seriously trying other ideas.
- krackers 2y agoI don't have any intuition here and am in no way qualified, but my read of the paper was that GRPO was mainly an optimization to reduce cost & GPUs when training (by skipping the need to keep another copy of the LLM in memory as the value network), but otherwise any RL algorithm should have worked? I mean it seems R1 uses outcome rewards only and GRPO doesn't do anything special to alleviate reward sparsity, so it feels like it shouldn't affect viability too much. Also on the note of RL optimizers, if anyone here is familiar with this space can they comment on how the recently introduced PRIME [1] compares to PPO directly? Their description is confusing since the "implicit PRM" they introduce which is trained alongside the policy network seems no different from the value network of PPO. [1] https://github.com/PRIME-RL/PRIME https://github.com/PRIME-RL/PRIME
- attentionmech 2y agoIf you check failure section of their paper, they also tried other methods like MCTS and PRM which is what other labs have been obsessing about but couldn't move on from (that includes bigshots). Only team which I am aware which tried verifiable rewards is tulu but they didn't scaled it up and just left it there. This sort of thing imo is similar to what openAI did with transformer architecture i.e. google invented it but couldn't scale it in the right direction and deepmind got busy with atari games. They had all the pieces still openai could do it. It seems to be it comes down to research leadership in what methods to choose to invest in. But yeah, the budgets big labs have, they can easily try 10 different techniques and brute force it all but seems like they are too opinionated in methods and less urgent on outcomes. [paper] https://arxiv.org/pdf/2501.12948 https://arxiv.org/pdf/2501.12948 [tulu] https://x.com/hamishivi/status/1881394117810500004 https://x.com/hamishivi/status/1881394117810500004
- attentionmech 2y agoI found the following thread more insightful than my original comment (wish I could edit that one). A research explains why RL didn't work before this: https://x.com/its_dibya/status/1883595705736163727 https://x.com/its_dibya/status/1883595705736163727
- krackers 2y agoRelated: https://twitter.com/voooooogel/status/1884089601901683088#m https://twitter.com/voooooogel/status/1884089601901683088#m Also https://epoch.ai/gradient-updates/how-has-deepseek-improved-the-transformer-architecture https://epoch.ai/gradient-updates/how-has-deepseek-improved-... has a summary of all the architectural improvements DeepSeek made to increase performance.
- qnleigh 2y agoThat's interesting. I suppose it could even be possible to test his theories. Just applied the exact same training methodology to smaller models or slightly easier problems and study what happens.
- attentionmech 2y agopeople already did: https://x.com/karpathy/status/1884678601704169965 https://x.com/karpathy/status/1884678601704169965