3 ms·
DeepSeek V3 and the cost of frontier AI models
- Havoc 2y agoThere is also the minor matter of the Alex Wang claim that they have 50k H100s but can't talk about it for obvious reasons... Doesn't take away from the outcome, but the cost numbers may not be 100%
- suraci 2y agoWho's Alex Wang, and why are his words taken seriously? If DeepSeek has 50k H100, I'll assume Tencent, Alibaba and Huawei have 2x or even 3x H100...
- langitbiru 2y agoHe is the CEO of Scale AI. https://x.com/kimmonismus/status/1882824571281436713 https://x.com/kimmonismus/status/1882824571281436713
- Havoc 2y agoIt’s hard to tell but I think the idea of a fair few h100s floating around china seems very plausible to me. Controls around it don’t seem super tight esp in early days
- suraci 2y agoWell, thinking this way will likely lead to making assumptions: 1. The rule that more dollars = more GPUs = greater computing power = more advanced LLMs still holds true. 2. Strengthening restrictions on exporting high-end chips to China can effectively curb the development of Chinese AI
- raphaelj 2y agoDo we know which change(s) made DeepSeek V3 so much more efficient than other models? I went through the paper and I understood they made these improvements compared to "regular" MoE models: 1. Latent Multi-head Attention. If I understand correctly, they were able to do some caching on the attention computation. This one is still a little bit confusing to me; 2. new MoE architecture with one shared expert and a large number of small routed experts (256 total, but 8 in use in the same token inference). This was already used in DeepSeek v2; 3. Better load balancing of the training of experts. During training, they add bias or "bonus" value to experts that are less used to make them more likely to be selected in the future training steps; 4. They added a few smaller transformer layers to predict not only the first next token, but a few additional tokens. Their training error/loss function then uses all these predicted tokens, not only the first one. This is supposed to improve the transformer capabilities of predicting sequences of tokens. Note that they don't use this for inference, except for some latency optimisation by doing speculative execution on the 2nd token. 5. They are using FP8 instead of FP16 when it does not impact accuracy. My guess would be that 4) is the most impactful improvement. 1), 2), 3) and 5) could explain why their model train faster, but not how is performs greatly better than models with way more activated parameters (e.g. llama 3).
- sschueller 2y agoDeli Chen at DeepSeek said they would open source AGI. https://x.com/victor207755822/status/1882757279436718454?t=0yOReRVNh2OKRRhaDFkVrg&s=19 https://x.com/victor207755822/status/1882757279436718454?t=0...
- unraveller 2y ago"Pushing to help make it a reality" as in they want credit if someone else does it with this lot of LLMs. If they built something that conforms to everyone's definition of AGI I doubt they'd release that thing, unless someone else was about to.
- sschueller 2y agoAt least there is a non-zero chance with DeepSeek vs OpenAI which 100% will not open source AGI if such a thing is ever realized. Over at OpenAI it's profits over the good for humanity.