2 ms·
> DeepSeek never claimed to have trained the base models they used Isn't that exactly what they have done? Maybe you are confused with the distilled models? D
by supermatt 2y ago
> DeepSeek never claimed to have trained the base models they used
Isn't that exactly what they have done? Maybe you are confused with the distilled models?
DeepSeek-R1: "DeepSeek-R1-Zero & DeepSeek-R1 are trained based on DeepSeek-V3-Base"
DeepSeek-V3: "At an economical cost of only 2.664M H800 GPU hours, we complete the pre-training of DeepSeek-V3 on 14.8T tokens, producing the currently strongest open-source base model. The subsequent training stages after pre-training require only 0.1M GPU hours"
- chpatrick 2y agoAren't they based on qwen? I thought the clever bit with DeepSeek is a way to fine tune with reinforcement learning, not training a huge model from scratch.
- vineyardmike 2y agoNo… mostly. They developed their “zero” model, which (they claim) is a from-scratch model. Base models are typically not fine-tuned for particular applications (eg chat). They trained their Zero model into a Chat+Reasoning model, which is what is attracting news. They ALSO fine-tuned small Qwen models using their big models as a teacher (distillation technique).
- supermatt 2y agodeepseek-r1 (reasoning) is a post-trained (not simply finetuned) deepseek-r3 (base model) by deepseek. qwen is a completely separate 3rd party model by alibaba. deepseek distilled (finetuned on the outputs of) r1 into qwen. They also did the same with llama.