6 ms·
TinyZero: Reproduction of DeepSeek R1 Zero in countdown and multiplication tasks
- Tepix 2y agoUnrolled non-X link with the announcement: https://threadreaderapp.com/thread/1882839370505621655.html https://threadreaderapp.com/thread/1882839370505621655.html
- blackeyeblitzar 2y agoWhat does it mean to reproduce DeepSeek R1-Zero? Like they have a model of equivalent performance? Is there a simple explanation of this post for those who aren't machine learning experts? Also is the technique here related at all to the technique people think DeepSeek themselves used, where they apparently trained the model using OpenAI outputs?
- 3nthusia5t 2y agoCould you provide source for the training the model on OpenAI outputs? I can’t find any news about that.
- blackeyeblitzar 2y agoI don't have a source to share, but I saw this claim on social media a few times in the last couple days, where people said their conversation with the model revealed that it thought it was some other OpenAI model. I have no idea how such training can work using another model's output, but I saw comments claiming that this is why their training was so cheap.
- suraci 2y ago> What does it mean to reproduce DeepSeek R1-Zero? means it's reproducible
- coolThingsFirst 2y agoWesterners cant reproduce chinese geniuses
- evertedsphere 2y agoreproducing the alphazero-like "model learns to reason on its own without supervised fine-tuning" phenomenon that deepseek-r1-zero exhibited
- dvh 2y agoReminds me of old polish encyclopedia: horse - everybody knows what horse is https://en.wikipedia.org/wiki/Nowe_Ateny https://en.wikipedia.org/wiki/Nowe_Ateny
- coolThingsFirst 2y agoI think there are 2 levels in the brain. One is used for programming the other for language. Doing them in parallel fails for some reason. A lot of GH projects just don't have solid explanation - i don't know what they built.
- zamadatix 2y agoR1-Zero is trained differently than most reasoning models, such as the "normal" R1 model, in regards what steps are done in training. TinyZero applies the same approach (but only on a subset of use cases) on a much smaller model to show it can apply on much smaller models as well. The details of how it's trained different start to get into "machine learning expert" territory but you can get a decent high level via a casual read through of the DeepSeek link if you want to dive deeper.
- serialx 2y agoSo to my understanding, this work reproduces DeepSeek R1's reinforcement learning mechanism in a very small language model. The AI gets "rewards" (like points) for doing two things correctly: Accuracy : Getting the right answer. For example, math answers must be in a specific format (e.g., inside a box) so a computer can easily check them. For coding problems, test cases verify if the code works. Format : Using the <think> and <answer> tags properly. This forces the AI to organize its responses clearly. So in this case, the training program can extract the model's answer by parsing <answer> tag. We can eval the answer and evaluate if it's correct or not. If it's correct give reward, else: no reward. Create N such answers from a single question, create N reward array. This is enough for the RL algorithm to guide the model to be more smart.
- suraci 2y agoIt looks like the 'old-school' RL to me, which makes me wonder why it took so long to get here
- vixen99 2y agoNothing like acronyms to make me feel dumb and ill-informed.
- basementcat 2y agoReinforcement Learning https://en.m.wikipedia.org/wiki/Reinforcement_learning https://en.m.wikipedia.org/wiki/Reinforcement_learning
- danju 2y ago[dead]
- krackers 2y agoI've been trying to follow the literature on PPO/GRPO as applied to LLMs. From what I understand, since reward is only given once the entire COT sequence is sampled, traditional RL techniques would require some form of credit-assignment to distribute that reward amongst individual tokens – which is where the critic/value network comes in, right? Instead DeepSeek (with GRPO) seems to just omit that value function entirely and use only sparse rewards. How does this end up being more efficient, since I thought the sparse nature of rewards makes it harder to converge to the optimal policy?
- deleted 2y ago[deleted]
- nxobject 2y agoThe author notes in their Twitter announcement [a] that their model’s reasoning abilities are only validated within the domain directly within their the domain of their Countdown training material. They admit that the real test of this training method is whether it produces outputs that pass the sniff test in other subject domains, or even abstract reasoning. However, given that there are “standardized test style” abstract reasoning tests with relatively small corpora (eg. ZebraLogic [b] on the order of 1000 or so cases), I do think they missed an opportunity to… do _some_ small benchmark for abstract reasoning before announcement. [a] https://threadreaderapp.com/thread/1882839370505621655.html https://threadreaderapp.com/thread/1882839370505621655.html - thanks @Tepix [b] https://huggingface.co/blog/yuchenlin/zebra-logic https://huggingface.co/blog/yuchenlin/zebra-logic