4 ms·
Except it’s not really a fair comparison, since DeepSeek is able to take advantage of a lot of the research pioneered by those companies with infinite budgets w
by wrasee 2y ago
Except it’s not really a fair comparison, since DeepSeek is able to take advantage of a lot of the research pioneered by those companies with infinite budgets who have been researching this stuff in some cases for decades now.
The key insight is that those building foundational models and original research are always first, and then models like DeepSeek always appear 6 to 12 months later. This latest move towards reasoning models is a perfect example.
Or perhaps DeepSeek is also doing all their own original research and it’s just coincidence they end up with something similar yet always a little bit behind.
- netdur 2y agoDidn't DeepSeek's CEO say that Llama is two generations behind, and that's why they didn't use their methods?
- matthewdgreen 2y agoThis is what many folks said about OpenAI when they appeared on the scene building on foundational work done at Google. But the real point here is not to assign arbitrary credit, it’s to ask how those big companies are going to recoup their infinite budgets when all they’re buying is a 6-12 month head start.
- wrasee 2y agoThis is true, and practically speaking it is how it is. My point was just not to pretend that it’s a fair comparison.
- mattlutze 2y agoFor-profit companies don't have to publish papers on the SOTA they product. In previous generations and other industries, it was common to keep some things locked away as company secrets. But Google, OpenAI and Meta have chosen to let their teams mostly publish their innovations, because they've decided either to be terribly altruistic or that there's a financial benefit in their researchers getting timely credit for their science. But that means then that anyone with access can read and adapt. They give up the moat for notariety. And it's a fine comparison to look at how others have leapfrogged. Anthropic is similarly young—just 3 and a bit years old—but no one is accusing them of riding other companies' coat tails in the success of their current frontier models. A final note that may not need saying is: it's also very difficult to make big tech small while maintaining capabilities. The engineering work they've done is impressive and a credit to the inginuity of their staff.
- miohtama 2y agoThese companies could not retain the best talent if they cannot publish:an individual researcher needs to get his name there "to get better."
- kridsdale1 2y agoExactly. This is why Apple is so far behind.
- wrasee 2y agoAnthropic was founded in part from OpenAI alumni, so to some extent it’s true for them too. And it’s still taken them over 3 years to get to this point.
- wrasee 2y agoAlso don’t forget that if you think some of the big names are playing fast and loose with copyright / personal data then DeepSeek is able to operate in a regulatory environment that has even less regard for such things, especially so for foreign copyright.
- rvnx 2y agoWhich is great for users. We all benefit from Libgen training, and generally copyright laws do not forbid reading copyrighted content, but to create derivative works, but in that case, at which point a work is derivative and at which point it is not ? On the paper all works is derivative from something else, even the copyrighted ones.
- wrasee 2y agoDisrespecting copyright and personal data is good for users? I guess I disagree. I would say that it’s likely great for the company’s users, but not so great for everyone else (and ultimately, humankind).
- byefruit 2y agoThis is pretty harsh on DeepSeek. There are some significant innovations behind behind v2 and v3 like multi-headed latent attention, their many MoE improvements and multi-token prediction.
- wrasee 2y agoI don’t think it’s that harsh. And I don’t also deny that they’re a capable competitor and will surely mix in their own innovations. But would they be where they are if they were not able to borrow heavily from what has come before?
- djtango 2y agoWe all stand on the shoulder of giants? Should every engineer rediscover the Turing machine and the Von Neumann architecture?
- wrasee 2y agoOf course not. But in this context the point was simply that it’s not exactly a fair comparison. I’m reminded how hard it is to reply to a comment and assume that people will still interpret that in the same context as the existing discussion. Never mind.
- dcow 2y agoDon’t get salty just because people aren't interested in your point. I for one, think it’s an entirely _fair_ comparison because culture is transitive. People are not ignoring the context of your point, they’re disagreeing with the utility of it. If I best you in a 100m sprint people don’t look at our training budgets and say oh well it wasn’t a fair competition you’ve been sponsored by Nike and training for years with specialized equipment and I just took notes and trained on my own and beat you. It’s quite silly in any normal context.
- wrasee 2y ago
- gizmo 2y agoFast following is still super hard. No AI startup in Europe can match DeepSeek for instance, and not for lack of trying.
- wrasee 2y agoMistral.
- rvnx 2y agoMistral is mostly a cheap copy of LLaMA
- wrasee 2y agoI would extend the same reasoning to Mistral as DeekSeek as to where they sit on the innovation pipeline. That doesn’t have to be a bad thing (when done fairly), only to remain mindful that it’s not a fair comparison (to go back to the original point).
- Der_Einzige 2y ago[flagged]
- deleted 2y ago[deleted]
- int_19h 2y agoIn what sense is Mistral a copy of LLaMA, specifically?
- rvnx 2y agohttps://x.com/arthurmensch/status/1752737462663684344?s=46 https://x.com/arthurmensch/status/1752737462663684344?s=46 This is one message of the founders of Mistral when they accidentally leaked one work-in-progress version that was a fine-tune of LLaMA, and there are few hints for that. Like: > What is the architectural difference between Mistral and Llama? HF Mistral seems the same as Llama except for sliding window attention. So even their “trained from scratch” models like 7B aren’t that impressive if they just pick the dataset and tweak a few parameter.
- techload 2y agoYou can learn more about DeepSeek and Liang Wenfeng here: https://www.chinatalk.media/p/deepseek-ceo-interview-with-chinas https://www.chinatalk.media/p/deepseek-ceo-interview-with-ch...
- qqqult 2y agogreat article, thank you
- versteegen 2y agoThis article is amazing. It explains not just why DeepSeek is so successful, but really indicates that innovators elsewhere will be too: that extensive opportunities exist for improving transformers. Yet few companies do (not just China, but everywhere): incredible amounts are spent just replicating someone else's work with a fear of trying anything substantially different.
- nowittyusername 2y agoThat was a really good article. I dig the CEO's attitude, i agree with everything he says and I am an American. From a Chinese perspective he must be talking an alien language so I salute him with trying to push past the bounds of acceptable hum drum. If the rest of China takes on this attitude the west will have serious competition.
- h8hawk 2y agoThat’s totally not true. https://epoch.ai/gradient-updates/how-has-deepseek-improved-the-transformer-architecture https://epoch.ai/gradient-updates/how-has-deepseek-improved-...