3 ms·

The first two are not the same thing. DeepSeek never claimed to have trained the base models they used. Now maybe a lot of people inferred that they did, but t

The first two are not the same thing. DeepSeek never claimed to have trained the base models they used. Now maybe a lot of people inferred that they did, but that's not what they claimed. Their breakthrough was more along the lines of, "given a model, we can train it to reason using a certain class of RL techniques". Which is, to my mind, more useful in any case. But yeah, if there are people out there thinking they can train base models from scratch for USD6 Million with no data, they're likely to be disagreeably surprised when they make the attempt.

deleted 2y ago
[deleted]
supermatt 2y ago
> DeepSeek never claimed to have trained the base models they used Isn't that exactly what they have done? Maybe you are confused with the distilled models? DeepSeek-R1: "DeepSeek-R1-Zero & DeepSeek-R1 are trained based on DeepSeek-V3-Base" DeepSeek-V3: "At an economical cost of only 2.664M H800 GPU hours, we complete the pre-training of DeepSeek-V3 on 14.8T tokens, producing the currently strongest open-source base model. The subsequent training stages after pre-training require only 0.1M GPU hours"
- chpatrick 2y ago
  Aren't they based on qwen? I thought the clever bit with DeepSeek is a way to fine tune with reinforcement learning, not training a huge model from scratch.
  - vineyardmike 2y ago
    No… mostly. They developed their “zero” model, which (they claim) is a from-scratch model. Base models are typically not fine-tuned for particular applications (eg chat). They trained their Zero model into a Chat+Reasoning model, which is what is attracting news. They ALSO fine-tuned small Qwen models using their big models as a teacher (distillation technique).
  - supermatt 2y ago
    deepseek-r1 (reasoning) is a post-trained (not simply finetuned) deepseek-r3 (base model) by deepseek. qwen is a completely separate 3rd party model by alibaba. deepseek distilled (finetuned on the outputs of) r1 into qwen. They also did the same with llama.
btown 2y ago
There's a lot of conflicting information here. There are two papers: https://arxiv.org/pdf/2412.19437v1 https://arxiv.org/pdf/2412.19437v1 for DeepSeek-V3-Base, which makes claims about 2.8M H800 GPU hours, and https://arxiv.org/pdf/2501.12948 https://arxiv.org/pdf/2501.12948 for DeepSeek-R1, which was based on V3-Base but doesn't seem to quote the cost of its RL steps. https://newsletter.languagemodels.co/p/the-illustrated-deepseek-r1 https://newsletter.languagemodels.co/p/the-illustrated-deeps... is a really fascinating overview of what those new steps were! The innovation here seems to be that the creation of a cutting-edge reasoning model could be made from a relatively inexpensive base model, regardless of how much RL is needed. Part of this seems to be a novel methodology for generating large volumes of chain-of-thought training data; it's left ambiguous exactly how the bulk of this data was generated, though, and it could have been a long process of manual curation. I'd also take the assertion that they started R1 with V3-Base in its release form with a grain of salt. We'll see whether people can reproduce similar results. Either way, there's a new chain-of-thought model with open weights, and that's an incredible achievement regardless of the context - and one that will be meaningful to future research.
- deleted 2y ago
  [deleted]

deleted 2y ago

[deleted]

supermatt 2y ago

> DeepSeek never claimed to have trained the base models they used Isn't that exactly what they have done? Maybe you are confused with the distilled models? DeepSeek-R1: "DeepSeek-R1-Zero & DeepSeek-R1 are trained based on DeepSeek-V3-Base" DeepSeek-V3: "At an economical cost of only 2.664M H800 GPU hours, we complete the pre-training of DeepSeek-V3 on 14.8T tokens, producing the currently strongest open-source base model. The subsequent training stages after pre-training require only 0.1M GPU hours"

chpatrick 2y ago
Aren't they based on qwen? I thought the clever bit with DeepSeek is a way to fine tune with reinforcement learning, not training a huge model from scratch.
- vineyardmike 2y ago
  No… mostly. They developed their “zero” model, which (they claim) is a from-scratch model. Base models are typically not fine-tuned for particular applications (eg chat). They trained their Zero model into a Chat+Reasoning model, which is what is attracting news. They ALSO fine-tuned small Qwen models using their big models as a teacher (distillation technique).
- supermatt 2y ago
  deepseek-r1 (reasoning) is a post-trained (not simply finetuned) deepseek-r3 (base model) by deepseek. qwen is a completely separate 3rd party model by alibaba. deepseek distilled (finetuned on the outputs of) r1 into qwen. They also did the same with llama.

btown 2y ago

There's a lot of conflicting information here. There are two papers: https://arxiv.org/pdf/2412.19437v1 https://arxiv.org/pdf/2412.19437v1 for DeepSeek-V3-Base, which makes claims about 2.8M H800 GPU hours, and https://arxiv.org/pdf/2501.12948 https://arxiv.org/pdf/2501.12948 for DeepSeek-R1, which was based on V3-Base but doesn't seem to quote the cost of its RL steps. https://newsletter.languagemodels.co/p/the-illustrated-deepseek-r1 https://newsletter.languagemodels.co/p/the-illustrated-deeps... is a really fascinating overview of what those new steps were! The innovation here seems to be that the creation of a cutting-edge reasoning model could be made from a relatively inexpensive base model, regardless of how much RL is needed. Part of this seems to be a novel methodology for generating large volumes of chain-of-thought training data; it's left ambiguous exactly how the bulk of this data was generated, though, and it could have been a long process of manual curation. I'd also take the assertion that they started R1 with V3-Base in its release form with a grain of salt. We'll see whether people can reproduce similar results. Either way, there's a new chain-of-thought model with open weights, and that's an incredible achievement regardless of the context - and one that will be meaningful to future research.

deleted 2y ago
[deleted]