3 ms·
But Deepseek took some huge and hugely expensive models that others had paid $Ms to train (well those who didn't already only the hardware mostly used investor
by willvarfar 2y ago
But Deepseek took some huge and hugely expensive models that others had paid $Ms to train (well those who didn't already only the hardware mostly used investor cloud credits, but still) and distilled them, rather than training from scratch?
- Tiberium 2y agoThey trained their own V3 and then trained R1 Zero from V3 purely as RL, it didn't completely work so then they took some CoT examples and trained R1 on RL + some SFT. You're thinking of the finetunes based on R1 outputs - those are not actually distills, and they're all far worse than the original R1. And yes, they're finetuned based on other models.
- ForOldHack 2y agoGarbage in, garbage out. More garbage out. Mmm... that garbage is looking pretty good. Can we have some more garbage?
- Tiberium 2y agoFinetunes of smaller models on R1 output are pretty bad, but the main R1 model is very, very far from being "garbage".