12 ms·
One thing I'd love to hear opinions on from someone with more free time to read these papers from DeepSeek is: am I right to feel like they're... publishing all
by erulabs 2y ago
One thing I'd love to hear opinions on from someone with more free time to read these papers from DeepSeek is: am I right to feel like they're... publishing all their secret sauce? The paper for R1 (1) seems to be pretty clear how they got such good results with so little horsepower (see: 'Group Relative Policy Optimization'). Is it not likely that Facebook, OpenAI, etc will just read these papers and implement the tricks? Am I missing something?
1. https://arxiv.org/abs/2501.12948 https://arxiv.org/abs/2501.12948
- skyyler 2y ago>am I right to feel like they're... publishing all their secret sauce? This would make perfect sense if the goal is to devalue existing players more than it is capture the market.
- nateburke 2y agoAnd trade against the devaluations...
- shakezooola 2y agoIndeed.
- raducu 2y agoDeepSeek probably can't compete with Open Ai in terms of scaling their data centers due to the bans, so why bother? If they did not open source it and instead just launched a payed (albeit much cheaper) closed model with similar performance to O1, would people trust them? I don't think DeepSeek has any malicious intent, but boy oh boy am I glad the USA boys get wrekt by this (though I also lose money on stocks). This is just poetic justice for the Orange Man's backwards 17th century policies.
- oefnak 2y agoYes, same here. As a European, I used to feel we (USA and Europe) were on the same side, the West, since they/you did save us from the baddies 70 years ago... But who's the baddies now? China is not waging war everywhere. Or threatening to steal Greenland... Or ruining our teenagers with social media.
- oefnak 2y agoOkay maybe they do too, with TikTok. But still.
- creato 2y ago> But who's the baddies now? Russia is currently invading Europe, to the tune of hundreds of thousands KIA. And Russia's invasion would be dead in the water without Chinese support.
- eunos 2y ago> And Russia's invasion would be dead in the water without Chinese support To paraphrase the Chinese rep on the UN. If China indeed supported Russia, then this war would have ended by now.
- creato 2y agoThere are obviously degrees of support. Sure, China could support Russia more than they are now, but they'd be risking their ability to trade with Europe and the US, so they don't.
- myko 2y agoAgree with the other commenter here re: degrees of support. If the US had fully supported Ukraine the war would also be over (unless China stepped in to help their nominal ally)
- eunos 2y agoVery lopsided support. US and allies offered heavy munitions, HIMARS, Jetfighters. What did China give to Russia aside of FPV drones, Holosun optics and Baofeng radios? I'd say if China also sent Type 99, PCL 181, PCL 191 and scores of AEWC drones then it's all over.
- _fizz_buzz_ 2y agoRussia is obviously has a very strong military, but economically they are dwarfs. And if you subtract natural resources they are basically irrelevant. China and the US are economically dominating everything.
- benterix 2y agoNot only that, I also enjoy their chain of thought being completely transparent for the user. I'm very curious what Altman is doing right now...
- whimsicalism 2y agopreparing for o3 release
- mi_lk 2y ago.. that'll be caught up in weeks
- kandesbunzler 2y agoI wonder if he still gets his billions for Stargate. I'm sure softbank is regretting that decision big time
- benterix 2y agoPersonally I'm very curious about the future of the Pro plan.
- sigmaisaletter 2y agoSoftBank is very used to regretting decisions. Very used to.
- NeutralCrane 2y agoI feel like SoftBank investing in Stargate should have been the number one red flag
- varelse 2y ago[dead]
- option 2y agono, they (like others) publish very little details about their training data.
- cosmojg 2y agoDeepSeek is a "side project" run by High-Flyer, a Chinese quantitative hedge fund. They have no interest in directly competing with LLM providers like OpenAI and Anthropic. If anything, they're likely trying to commoditize their complement in a way not all that dissimilar from Meta's approach.
- whimsicalism 2y agomore of a pivot, China started cracking down heavily on quants in 2024
- silisili 2y agoI'm curious about this. Two articles I've read all but said they basically failed as a quant and lost more than they gained. The wiki points out some losses, but some wins, so is unclear. Have they actually pivoted, or are they just messing around to see what sticks?
- sudosysgen 2y agoDidn't they crack down mostly on HFT? I haven't heard of a huge crackdown on low/medium frequency quants, and LLM research has low crossover with high freq. quant stuff
- whimsicalism 2y agoalmost all quant work is 'HFT'
- frankc 2y agoThis is wrong. There are plenty of quant strategies that aren't a race.
- whimsicalism 2y agomost quant money/employment is among market makers who are not holding for longer than a day and most trades probably complete within a few seconds. regardless, high-flyer is an HFT firm
- bilbo0s 2y agoThe secret sauce is the data. I wouldn't hold my breath on getting access to it.
- jgalt212 2y agoIndeed. Litigation exposure is just too great when releasing the training data.
- reissbaker 2y agoJust about anything useful in the secret sauce data can be distilled from the model by inspecting the logits; for example, they published distills using Llama 3.1 70b as a base, Qwen 32b, etc etc. There is no "secret" sauce. Only sauce. Additionally, R1-Zero shows that you don't even really need much secret sauce data, since they trained it with zero SFT data. Take an existing base model, do GRPO RL, and tada: you have a SOTA reasoning model. SFT data improves it, but the secret sauce isn't in the data.
- mv4 2y agofyi Yann LeCun, Chief AI Scientist at Meta, said: “To people who see the performance of DeepSeek and think: ‘China is surpassing the US in AI.’ You are reading this wrong. The correct reading is: ‘Open source models are surpassing proprietary ones.’ DeepSeek has profited from open research and open source (e.g., PyTorch and Llama from Meta). They came up with new ideas and built them on top of other people’s work. Because their work is published and open source, everyone can profit from it. That is the power of open research and open source.” [1] https://www.forbes.com/sites/luisromero/2025/01/27/chatgpt-deepseek-or-llama-metas-lecun-says-open-source-is-the-key/ https://www.forbes.com/sites/luisromero/2025/01/27/chatgpt-d...
- melenaboija 2y agoLol this is almost comical. As if anyone riding this wave and making billions is not sitting on top of thousands of papers and millions of lines of open source code. And as if releasing llama is one of the main reasons we got here in AI…
- pilooch 2y agoSure but it's good to recognize Meta never stopped publishing even after Openai and deepmind most notably stopped sharing the good sauce. From clip to dinov2 and llama series, it's a serious track to be remembered.
- deleted 2y ago[deleted]
- AnimeLife 2y agoBut there is a big difference, llama is still way behind chatgpt and one of the key reasons to open source it could have been to use open source community to catch up with chatgpt. Deepseek on contrary is already at par with chatgpt.
- llm_trw 2y agoLlama is worse than gpt4 because they are releasing models 1/50th to 1/5th the size. R1 is a 650b monster no one can run locally. This is like complaining an electric bike only goes up to 80km/h
- coliveira 2y agoThis is not coming from a big corporation. These people need to establish their authority, or nobody will believe what they're doing. So it makes sense that they publish their ideas and open source the result. Now they have the attention and can play with their cards.
- layer8 2y agoThis interview with DeepSeek founder and CEO Liang Wenfeng, also co-founder of the hedge fund backing DeepSeek, might shed some light on the question: https://www.chinatalk.media/p/deepseek-ceo-interview-with-chinas https://www.chinatalk.media/p/deepseek-ceo-interview-with-ch... Some relevant excerpts: “Because we believe the most important thing now is to participate in the global innovation wave. For many years, Chinese companies are used to others doing technological innovation, while we focused on application monetization — but this isn’t inevitable. In this wave, our starting point is not to take advantage of the opportunity to make a quick profit, but rather to reach the technical frontier and drive the development of the entire ecosystem.” “We believe that as the economy develops, China should gradually become a contributor instead of freeriding. In the past 30+ years of the IT wave, we basically didn’t participate in real technological innovation. We’re used to Moore’s Law falling out of the sky, lying at home waiting 18 months for better hardware and software to emerge. That’s how the Scaling Law is being treated. “But in fact, this is something that has been created through the tireless efforts of generations of Western-led tech communities. It’s just because we weren’t previously involved in this process that we’ve ignored its existence.” “We do not have financing plans in the short term. Money has never been the problem for us; bans on shipments of advanced chips are the problem.” “In the face of disruptive technologies, moats created by closed source are temporary. Even OpenAI’s closed source approach can’t prevent others from catching up. So we anchor our value in our team — our colleagues grow through this process, accumulate know-how, and form an organization and culture capable of innovation. That’s our moat. “Open source, publishing papers, in fact, do not cost us anything. For technical talent, having others follow your innovation gives a great sense of accomplishment. In fact, open source is more of a cultural behavior than a commercial one, and contributing to it earns us respect. There is also a cultural attraction for a company to do this.”
- tarkin2 2y agoImpressive, honestly. They're trying to become a mecca for innovation and research, trying to lead rather than follow, build a culture where innovation can spark future economic advantages, whereas OpenAI seem to more about monetisation currently, many of their researchers and scientists now departed. Under the aegis of a dictatorship they may be, but this encourages me more than anything OpenAI have said in a while.
- blackeyeblitzar 2y agoNo, they aren’t publishing all their secret sauce. For example, we have no idea how their baseline model was trained. They’ve not said anything about the data or code relating to this training. They have talked about some of the optimization techniques they’ve used in arriving at their final models that they released weights for, but their claims on cost seem suspicious because we don’t know what prior work they built on. I’ve seen many people sharing evidence that DeepSeek’s models seem to think they are OpenAI models, which supports the theory that DeepSeek first built a baseline trained off the outputs of other models. DeepSeek also likely has a much larger number of GPUs than what they’ve admitted, perhaps to avoid attention on their suppliers who may have violated sanctions.
- HarHarVeryFunny 2y agoThe number of GPUs they have (which may well be export-legal H800's as NVidia believe they are) goes hand in hand with the amount it cost to train (however you define that), and is something people trying to replicate their approach can verify (or not). It seems obvious that you need to have a model trained, or fine-tuned, on some reasoning data (with backtracking etc) such that reasoning behavior is part of it's repertoire, before you can use RL to hopefully get it to use such reasoning pursuant to whatever goals you are setting. I'd not be surprised if they used O1 outputs to bootstrap the model in this way, although O1's reasoning traces are a deliberate obfuscation of what it is really doing (an after-the-fact summary) so even if this is the case that should be borne in mind! OTOH, while reasoning data may be scarce in the wild, it's presumably not entirely unavailable, and/or DeepSeek may have created some themselves, so who knows what mix DeepSeek used for this initial bootstrapping stage. As you say, this aspect remains as "secret sauce". Of course once they've got their first stage model trained they then use that to generate data for the second/final stage.
- rlupi 2y agoDeepSeek is a company whose funds comes from a edge fund. If the edge fund has predicted the impact of all these releases correctly, they have likely made tons of money while at the same time advanced Chinese interests and prestige abroad. It seems a great move.
- Horffupolde 2y agoAre you French?
- dtquad 2y agoDeepSeek and their quant/algotrading parent company have years of experience in raw C/C++ CUDA programming and low-level CUDA optimization. That is one of the main reasons they could do model training and serve inference so effectively and cheaply. That hard-earned experience is not something they have shared publicly.
- blitzar 2y agoWe get free Ai from a hedge fund and $200/month Ai from a nonprofit.
- deleted 2y ago[deleted]
- _DeadFred_ 2y agoHow are you liking 2025?
- msoad 2y agoI hope the hedge fund shorted NVDA to make some good money along the way too hahaha!
- eunos 2y agoThis is a bit of conspiracy theory, but there's a theory that China's strategic goal isn't to have an all-encompassing Chinese OpenAI that can charge a shit ton of money. The goal is to prevent US companies from doing so.
- ascorbic 2y agoThat's not really a conspiracy theory. It's a pretty much the core of China's industrial strategy. Projects like "Made in China 2025" are explicitly about developing their own innovation base to avoid being reliant on foreign companies. AI is one of the areas where this was most urgent.
- lofaszvanitt 2y agoBut why? As a reply to the sanctions? You try to hamstring us, we'll burst your state protected NVDA?
- kelipso 2y agoBecause it would be economically destructive to everyone except those large companies if it were to happen.. and the everyone here includes China.
- 2-3-7-43-1807 2y agodo you think the chinese government will allow them to spill the secret sauce? the secret sauce to what is eventually going to be weapon? clear no.
- msoad 2y agoThey could make a ton of money shorting NVDA and releasing the paper. The most honest short position ever!
- Sparkyte 2y agoNo company with an open source solution shows their hand. I said the same thing about AI music stuff, bet you 1000% that record studios have been pounding away at AI generated music/lyricals for years now well before the likes of models we see today.
- keepamovin 2y agoThe strategic move by China to release a free, open, DeepSeek model that’s way cheaper to train and use, and that resulted in a $1 trillion market loss on the news, has a clear objective to destroy the US AI companies that had leapt ahead of China in commercial offerings.
- zx10rse 2y agoIt doesn't matter if they are not open sourcing their models they are already loosing the game, sure they are making money now.
- fedeb95 2y agothey're trying to show that a good part of the economic assumptions behind the AI boom are flawed. That is, dramatic increase in energy and chip demand.
- kazinator 2y agoAny perception that OpenAI are playing catch-up, and moreover by taking ideas, will have a great negative impact. Anyway, we may be past peak OpenAI at this juncture.