5 ms·
ReasoningGym: Reasoning Environments for RL with Verifiable Rewards
- starzmustdie 1y agoGitHub: https://github.com/open-thought/reasoning-gym https://github.com/open-thought/reasoning-gym
- t55 1y ago[flagged]
- deleted 1y ago[deleted]
- phh 1y agoCool cool. I'm a bit put off by calling it "reasoning" /"thought". These RL targets can be achieved without "thinking" model but still cool. Gotta love the brainfuck task. I personally think that Gemini 2.5 Pro's superiority comes from having hundreds or thousands RL tasks (without any proof whatsoever, so rather a feeling). So I've been wanting a "RL Zoo" for quite a while. I hope this project won't be a one-off and will be maintained long term with many external contributions to add new targets!
- t55 1y ago> I personally think that Gemini 2.5 Pro's superiority comes from having hundreds or thousands RL tasks (without any proof whatsoever, so rather a feeling). Given that GDM pioneered RL, that's a reasonable assumption
- flowerthoughts 1y agoAssuming with GDM, you mean Google-Deep Mind. They pioneered RL with deep nets as policy function estimator. The deep nets being a result of CNNs and massive improvements in hardware parallelization at the time. RL was established, at the latest, with Q-learning in 1989: https://en.wikipedia.org/wiki/Q-learning https://en.wikipedia.org/wiki/Q-learning
- t55 1y agoi didn't say they invented everything; in science you always stand on the shoulders of giants i still think my original statement is fair
- lechatonnoir 1y ago"gdm pioneered rl" is definitely not actually right, but it's correct to assert that they were huge players. people who knew from context that your statement was broadly not actually right would know what you mean and agree on vibes. people who didn't could reasonably be misled, i think.
- olliestanley 1y agoWe definitely plan to maintain the project for as long as there is interest in it. If you have ideas for new tasks, we'd always welcome contributions!
- phh 1y agoThanks for the answer! As a toy project I implemented wikiracing with trl. I'll probably try to PR that to your gym. (can't say that I managed to improve score with it though)
- CuriouslyC 1y agoGemini 2.5 Pro's superiority is IMO largely driven by their long context support and training methodology. Compare Gemini as a beta reader for a 100k token book with GPT4.1 or Claude 4, and it becomes quite clear how much more effectively it can reason across its context than other comparable models. This also makes it much better for architecting new features into a system, since you can load a lot of the current system into the context and it'll conform to existing styles and architecture patterns more closely.
- jacob019 1y agoAgreed, 2.5 flash too. I analyze a large json document of metrics for pricing decisions. Typically around 200k, occtionallly up to 1M, Gemini 2.5 significantly outperforms for my task. It isn't 100%, but role playing gets close. I suppose that's a form of inference time compute.
- t55 1y agoFor a 100k token context window; all those models are comparable though gemini 2.5 pro shines for 200k+ tokens
- CuriouslyC 1y agoI can confirm from first hand experience that even at 100k they are most definitely not comparable for the task of beta reading.
- throwaway314155 1y agosplitting hairs much?
- ninakostoska 1y agoCool to see NVIDIA’s most recent reasoning model [1] already uses Reasoning Gymas a large part of their data mixture [1] https://arxiv.org/abs/2505.24864 https://arxiv.org/abs/2505.24864
- t55 1y ago> prolonged RL training can uncover novel reasoning strategies that are inaccessible to base models, even under extensive sampling does this mean that previous RL papers claiming the opposite were possibly bottlenecked by small datasets?
- yorwba 1y agoNo, they do not point to any specific examples of novel reasoning strategies that were uncovered, nor is their sampling that extensive (at most 256 samples vs the 2048 used in https://limit-of-rlvr.github.io/ https://limit-of-rlvr.github.io/ ).
- t55 1y agoso you think it's fake news? another example of a paper with strong claims without much evidence?
- yorwba 1y agoI think it's a case of not coming up with alternative explanations for the observed evidence and hence not designing experiments to distinguish between those explanations. Their results are consistent with novel reasoning strategies, but they're also consistent with more reliable execution of reasoning strategies that the base model can generate in principle, but rarely succeeds at due to a large number of steps. (If you have a model that can do each step independently with 99% success rate and getting the correct result requires 1000 steps, the chance of making it all the way to the end without a single error is only about 0.004%.)
- 1y ago
- jimmySixDOF 1y agoRL is proving to be a weird science lately : >Spurious Rewards: Rethinking Training Signals in RLVR ### *TL;DR* We show that you can do RLVR on Qwen2.5-Math models with *completely random or incorrect rewards*, and still get massive math benchmark gains. All of the following spurious rewards give 15-20+ points on MATH-500 when RLVR training Qwen2.5-Math-7B: - RLVR + format reward (reward responses with `\boxed{}`): *+16.4%* - RLVR + incorrect reward (only incorrect answers rewarded): *+24.6%* - RLVR + random reward: *+21.4%* - (as a reference) RLVR + ground-truth reward: + 28.8% How can these spurious rewards possibly work? Can we get similar gains on other models with broken rewards? >Learning to Reason without External Rewards Training large language models (LLMs) for complex reasoning via Reinforcement Learning with Verifiable Rewards (RLVR) is effective but limited by reliance on costly, domain-specific supervision. We explore Reinforcement Learning from Internal Feedback (RLIF), a framework that enables LLMs to learn from intrinsic signals without external rewards or labeled data. We propose Intuitor, an RLIF method that uses a model's own confidence, termed self-certainty, as its sole reward signal. Intuitor replaces external rewards in Group Relative Policy Optimization (GRPO) with self-certainty scores, enabling fully unsupervised learning. Experiments demonstrate that Intuitor matches GRPO's performance on mathematical benchmarks while achieving superior generalization to out-of-domain tasks like code generation, without requiring gold solutions or test cases. Our findings show that intrinsic model signals can drive effective learning across domains, offering a scalable alternative to RLVR for autonomous AI systems where verifiable rewards are unavailable. [2] [1] https://rethink-rlvr.notion.site/Spurious-Rewards-Rethinking-Training-Signals-in-RLVR-1f4df34dac1880948858f95aeb88872f https://rethink-rlvr.notion.site/Spurious-Rewards-Rethinking... [2] https://arxiv.org/abs/2505.19590 https://arxiv.org/abs/2505.19590
- t55 1y agoyeah, RLVR is still nascent and hence there's lots of noise. > How can these spurious rewards possibly work? Can we get similar gains on other models with broken rewards? it's because in those cases, RLVR merely elicits the reasoning strategies already contained in the model through pre-training this paper, which uses Reasoning gym, shows that you need to train for way longer than those papers you mentioned to actually uncover novel reasoning strategies: https://arxiv.org/abs/2505.24864 https://arxiv.org/abs/2505.24864
- sadboots 1y agoby the love of god, please stop overfitting on gsm8k
- i5heu 1y agoIt looks like your neural network is overfitted on seeing overfitt where is none. Prejudices is a form of overfitting IMHO
- t55 1y agoagree, the RG evals feel like a fresh breeze
- olliestanley 1y agoDifficult one. GSM8K and MATH evals (both reported in Reasoning Gym paper) are common in smaller model RL papers for a reason, which is that smaller models can get decent scores on them, unlike fresher & harder benchmarks. Part of the aim of RG is to be used as a difficulty-adjustable & non-repeating eval though so if people think it's a good benchmark, perhaps it will allow this status quo to shift!
- vkgvrfkvmrfkrf 1y ago[flagged]
- vkgvrfkvmrfkrf 1y ago[flagged]
- curtisszmania 1y ago[dead]